Agree with you on 'When I'm Sixty-Four' and I'll go as far as saying that it is a necessary track there, that - along with 'She's Leaving Home' - provides a little glimpse of reality, a short wake-up call, a celebration of day-to-day little pleasures. The Beatles were not dumb guys, and the track was in no way put there by mistake, in my opinion. Works just as 'Sloop John B' on 'Pet Sounds' - the narrator's break of inner-self-analysis, in favor of an outer experience (that eventually turns out to have an emotional impact on him after all).
I get what you mean in your second paragraph and you expressed your thoughts very well. You mention the fact that few songs would've been special if not for the adventurous production. But I think, in some cases, this is a moot point, because they weren't. Ask yourself this: would John have attempted to write a song based on a circus poster if not knowing that he'll be able to (try to) create a somewhat circus-like atmosphere? They knew that George Martin was not about to shy away from trying anything in the studio and they kept that in mind while composing. Nobody expects 'Lucy In The Sky' to work, given a simpler arrangement, because that's not the point of it. Never was. Again, if you like the approach of RS and Revolver better, that's fine. But Sgt Pepper is a totally different story and it must not be downgraded based on '65/'66 criterias.