I voted for "Abbey Road" too.
The first time I heard Abbey Road (about six months after Sgt. Pepper's), I was a young teenager and didn't know what to make of a lot of it. I Want You (She's So Heavy) sounded vaguely familiar to me but I was weirded out by the repetition, the guitar riff, the length.... and I didn't know what the hell to make of the side 2 suite. But I grew into it, and now I don't think there's a weak moment on the album. Side 2 is a masterpiece, a great way for the Beatles to go out.
I had the same experience you did. Abbey Road confused me when I was a lot younger, and I never had that feeling about a Beatles album before. I had just never heard anything like it since it wasn't a regular pop record. But I gave it many listens and finally realized it was a beautiful masterpiece. Since then it's been my favorite Beatles album.
I'm surprised Sgt. Pepper has ranked last so far. It's kind of funny. I love Sgt. Pepper and it's been lauded as the Beatles' magnum opus, but although it's wonderful, innovative, and very "listenable," it seems like many fans choose other albums as their favorite.
