If you assume there is no way to exceed the speed of light, and you assume it takes a resource rich world (like ours, circling a 3rd generation star) to achieve a high-tech culture that might go into space, then it makes sense that there wouldn't be that many space-faring cultures in our galaxy and that not many would be much further-developed than ours.
Even if you assume that there are multiple millions of planets in our galaxy suitable for life to develop and evolve, how many would produce intelligent life, and how many of those would develop to the point of space travel on the scale necessary. We haven't yet and the odds that we will are only so-so.
Add to those considerations the points already raised about why aliens would visit our remote planet, and there's very little reason to think we'd be scheduled for an early visit.
BTW, the idea that it won't be a big deal to figure out FTL travel is almost certainly wrong. It is unquestionably a characteristic of normal space-time. And while we can conjecture and calculate about leaving or altering normal space-time, as far as I know the only plausible non-wormhole approach anyone has ever thought of to go faster than light involves artificially inducing or altering the inflation factor in a non-trivial region of space and then "surfing" that wave of altered space-time to a destination. Something that could take the energy of multiple stars to accomplish for a small trip and could destroy boh the departure location and the destination.
Traveling through wormholes is a wonderful sci-fi device. But we don't know if there really are any such things, or whether there really could be any. And if there are or could be, we don't have a clue how they operate, how we could control them, whether we could survive passage through them, or even if they exit back into normal space.
One of the more plausible views of wormholes is that they are holes or rips in the "field" within which our universe propagates. Think of the universe as a big block of Swiss cheese. Even if you can figure out how to enter a "hole" and even though that theoretically frees you of the speed of light constraint, how do you remain intact inside the hole and how do you exit on the other side?