I felt this way when we started in 2007, even though I'd heard of geocaching back in 2001 or so and hadn't felt motivated to try it.
I missed the opportunity to seek caches in Kuwait, Afghanistan, Iraq, Qatar, and Oman, including the oldest cache in Iraq, GCGXJB, which I was literally standing 10' or 3 meters from at one point. These are not opportunities I really want to repeat, though, so that ship has sailed.
I've revisited and cached in all the countries I want to revisit that I'd been to before geocaching: Czechia, Ireland, Germany, Iceland, Italy, Hungary, and Malta. So look at it as a great excuse to go back. :D