I suspect that many Americans don't have passports because it's easier and cheaper to travel within the continental United States, and you don't need identification to cross state borders.
I do have a passport but I've only been to England and Mexico. I would love to travel more and visit other countries, but right now, money is tight. Such is the life of a poor and busy graduate student. And it's hard on me too, because I hear my classmates and relatives chatter about exotic places they've been, and it's very embarrassing to say you can't go places because of finances.
But once I have the money, I plan to see the world.
I've been criticized by my husband's European uncle for not traveling more, but his uncle has always been wealthy so it was difficult to make him understand that money is a problem. It's
not because I don't want to travel more, it's because it's not a real possibility right now.
That said, though, I have visited a lot of places in the United States: Massachusetts, Nevada, New York, Louisiana, Washington DC, Pennsylvania, Oregon, Utah, etc. And sometimes going to a different state feels a bit like going to a different country--American cultures, lifestyles, accents, cuisine, etc. can vary from state to state (and even city to city), and it's very cool.
Joost, I live in California, and this summer my husband and I drove for SIX hours going upstate and we were
still in California at the end of our journey. If you live where I do, you can drive for endless hours and you'll just wind up in the desert, the ocean, or the middle of nowhere (like Fresno :-/ ). Things are spread so far apart here on the West Coast, so it's not like Europe.