I think 'Smile' was going to be his masterpiece (so he thought) and then he heard Sgt. Peppers and it crushed him because that was the vision he was looking for.
I believe that the impact that Sgt. Pepper had on Brian has always been hugely overstated.
There are several reasons I can think of why Brian abandoned Smile:
- Some members of the band, especially Mike Love, hated most of the new material (according to Brian this was the main reason).
- Brian's dad Murry, who still treated him like a little kid, hated the new material as well.
- Brian was seriously mentally ill (he has later been diagnosed with schizophrenia and bipolar disorder) and was doing too much drugs at the time.
- Pet Sounds had (relatively) been a commercial failure. Smile was sceduled to be released in time for Christmas 1966 but it still wasn't finished in May 1967. Needless to say Capitol Records also wasn't very happy with the way things were going.
- Brian's lyricist for the project, Van Dyke Parks, had quit because he had a hard time dealing with Brian's drug use and behavior, and the criticism of especially Mike Love.
- The band had several legal problems at the time, most notably a dispute with Capitol over royalty payments and Carl Wilson's call-up for the draft.
In conclusion: I think Brian had more weight on his shoulders than a man in his mental state was able to bear. Sure it bummed him out that The Beatles had released their masterpiece before he'd finished his, but I don't think it was one of the main reasons why he shelved Smile. I think that's just a myth that Beatles fans and sensationalist journalists would like to believe. "The Beatles were so good that it literally drove their biggest rival insane!". Based on the dozen+ books (and countless articles and interviews) that I've read about Brian and The Beach Boys, I'm sure that what happened to Brian was bound to happen anyway. With or without Sgt. Pepper.