George Harrison worked on Only A Northern Song before he worked on Within You, Without You, but George Martin didn't care for the implication to Northern Songs Inc, and the song was therefore shelved (and used later as a "throwaway" song for the Yellow Submarine movie, which is such bullspit and such a diss to Harrison).
I always loved Only A Northern Song, and never cared for Within You, Without You, but that's beside the point of this thread. The OP mentioned that Within You was the only Harrison song on the album, therefore it couldn't be dropped, but what if Northern Song was there instead.
Anyway, and I know people will think I'm crazy, but I'd drop Day In The Life. I find the instrumentation annoying, how it goes to that high pitch more than once in the song. If I'm playing it in the car with older family members, they're tempted to tell me to change the CD because it is really unlistenable.
So that song is dropped. What next? Hmmm...drop She's Leaving Home. That song doesn't even fit in the album's concept, if you ask me. How biast George Martin is (or a complete ass-kisser) towards Paul vs George. She's Leaving Home should have been the throwaway, and kept Only A Northern Song.
So, with A Day In The Life and She's Leaving Home dropped, you can now add in Penny Lane and SFF.
I never did understand why just because they were released as singles prior to the album being released that they couldn't include them?