Click here to listen to an exclusive preview of Forever She'll Be My Surfer Girl

Anyone who saw the Beach Boys documentary An American Band in the early Eighties would have concluded that Brian Wilson – overweight, lank-haired, creatively stymied by bandmates and inner demons – was heading for only one heavenly arch, and it wasn’t the one outside Waterloo Station. Yet next week he’ll be performing his third set of shows at the Royal Festival Hall this decade, and, like his Smile performance in 2004, it’ll be a world premiere; That Lucky Old Sun is a suite of songs with a narrative that even the folks on the South Bank haven’t heard yet.

“The Festival Hall people called last year and asked me to write a short piece for the reopening of the hall,” he says from rehearsals in Los Angeles. “I was having a creative explosion, you might say, last summer. So I called [ Smile collaborator] Van Dyke Parks in to write some narratives for it, ones that went hand in hand with the lyrics. I hope people in London get some southern California sunshine from this piece.”

The title song that threads through the new work has been recorded by everyone from Ray Charles to Jerry Garcia to Johnny Cash. Originally it was a slave song, which may sound odd, but Wilson can undoubtedly relate. In the early Seventies, isolated from his band and beset with mental and pharmaceutical problems, Wilson was expected to continue laying golden eggs for the Beach Boys.

“I was playing my synthesizer last summer and I just ran across that song, That Lucky Old Sun,you know it? ‘Up in the mornin’, out on the job, work like the devil for my pay, but that lucky old sun has nothin’ to do but roll around heaven all day.’ And I thought I wanna learn the whole piece. So I bought the Louis Armstrong version of it.”

The writer Paul Williams encapsulated the feel of the Beach Boys’ music in three words – warmness, serenity, friendship. They have been called the most emotionally satisfying band of all time. Though he couldn’t surf (brother Dennis was the truest Beach Boy in this sense), Wilson had an incredible ear for melody and harmony. Harnessing family and school friends, he created a string of million-selling pop songs that celebrated youth, sun, girls and cars. The harmonies he borrowed from the Four Freshmen, the joyous, edge-of-the-seat production from Phil Spector.

Soon he was turning out three albums a year of the stuff, and the supply of hits – Surfin’ USA, I Get Around, Fun Fun Fun – seemed inexhaustible. The first cracks appeared in 1964, when he found himself crying on a plane, unable to fly. He was replaced on tour and, given time to mature in the studio, came up with Pet Sounds in 1966, still considered to be pop’s ultimate flowering. When every insider from brother Dennis to Leonard Bernstein claimed the follow-up, Smile, was going to be five times better, expectations went through the roof.

People expected more from Wilson than from anyone else in pop. Wilson didn’t mind being called a genius, but the weight of expectation, combined with his fondness for LSD, tipped him over the edge. First, he abandoned Smile. Then he pulled the Beach Boys out of the 1967 Monterey Pop festival, which turned out to be a quantum leap in rock’s development. In just over a year the Beach Boys went from a band whose every album had gone gold to one that could only shift 23,000 copies of the low-key Friends.

The only song that anyone has heard from That Lucky Old Sun is called Midnight’s Another Day. The piano-led chords are instantly recognisable as Wilson’s work. “I think there’s a bit of Gershwin in there,” he says, “but I think it’s a very mature, emotional song. It’s a low-key song. The darker periods of my life make me want to express myself; vocally. They make me want to sing.”

Even at 65, Brian Wilson seems naive, pop’s own Charlie Brown. One of the most poignant moments of the Smile shows at the Festival Hall wasn’t musical at all. As Wilson left the stage to a standing ovation his shirt caught on something; he tried to walk off but was held back. After what seemed an age he noticed what was wrong, but not before he’d appeared to 2,000 people as a living cartoon bear, bumbling and bemused.

He still seems a little lost. “I hope I can bring some southern California sunshine to you,” he tells me for the second time in 20 minutes. In spite of it all – and because of it all – you know he will.

Brian Wilson’s That Lucky Old Sun (a Narrative) is at the Royal Festival Hall, London SE1 (0871 6632500) from September 10-16, then touring (see www.brianwilson.com for details).

Explore Music

• CD Reviews

• Live Reviews

• Classical

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010