After 14 years missing in action, the cult 1960s musician Kevin Ayers turned up last summer – in the back of my car. Stranger still, he was playing Teddy Bear’s Picnic on a guitar destined for Pete Doherty. Ayers and I hadn’t met before, but the charmingly dishevelled former Soft Machine singer had hopped in for a lift between a mutual friend’s birthday dinner and a party in nearby Notting Hill. On arrival, he sat on the floor and struck up a song. Within minutes, a worse-for-wear crowd – which included the hip (rock writer Charles Shaar Murray, Grammy award-winning producer Peter Henderson) and the hooray (a bunch of posh barristers) – was crouched open-mouthed around him, bewitched by his sensual, sonorous vocals and pretty, intricate playing. Most hadn’t a clue who he was.

Almost exactly a year on, the day after his 63rd birthday, Ayers and I are in a quaint fisherman’s shack on a beach in Felixstowe, where he is supposed to be detoxing, but admits to having a hangover. The trip to London, it transpires, was his third attempt in more than a decade to seal a record deal. The songs were never the problem: Ayers may have been in self-imposed exile in the south of France since the mid1980s (or, as he puts it, “Living a cool, quiet life, cooking, drinking wine and swimming every morning”), but he never stopped composing quirky, quality folk-rock.

What he hates is the music industry. It was, he claims now, the reason he quit the psychedelic innovators Soft Machine after only one album, fresh from an American tour supporting fan Jimi Hendrix, and just as Pink Floyd were banking their first big cheques. It was why he spent the summer of love in St Tropez, mixing with Brigitte Bardot, Catherine Deneuve and a posse of French intellectuals while the Beatles, David Bowie and Brian Eno were blending his sounds into their songs; and why, in the early 1970s, he moved to Mallorca, vowing never to return to England.

Last summer, however, Ayers finally found a handful of industry outsiders he was willing to work with, and tomorrow he releases The Unfairground, his first studio album since 1992 and, more important, his best since his heyday. Over 10 taut tracks, his deep, rich, still defiantly middle-class English vocals grace a beguiling mix of jaunty folk, melodic midtempo rock, music-hall and soft psychedelia, while poetic lyrics look back over his life and reluctantly accept the arrival of old age. “I didn’t begin with a theme, God forbid,” bristles Ayers, a slow, cautious speaker who can be both painfully shy and playfully mischievous. “But it is very much a reflective album: lost love, lost feelings, lost sensibilities. Of course, I had to include some of my blood, sweat and tears – if you are going to be honest, it can’t be avoided – but it’s not an album of regret.”

Recorded in New York, Arizona, London and Glasgow, The Unfairground features contributions from a host of Ayers devotees, from the veterans Phil Manzanera of Roxy Music, his former Soft Machine colleague Robert Wyatt and members of Teenage Fanclub and Gorky’s Zygotic Mynci, to the fast-rising Liverpudlian Candie Payne and leftfield indie outfits the Ladybug Transistor, Neutral Milk Hotel and Architecture in Helsinki – the latter trio giving much of the album an unexpected contemporary edge. Henderson, producer and engineer for Paul McCartney and Supertramp’s Breakfast in America, signed on to oversee the recordings the morning after Ayers’s impromptu party performance.

Yet ask Ayers about his plethora of guest artists and he looks perplexed. “I have no idea who most of them are,” he admits. “Or why they would want to work with me. I’m not being flippant – I’m flattered to have them – but like most people in my life, they just turned up.”

Surely he had some say in who was to play on such personal songs? “No, I don’t make those decisions,” he snaps, exasperated. “I know it sounds crazy, but they just happen. I provide the songs. Everything else that goes on is beyond me. On this occasion, I was lucky, because so many incredible musicians came, but I had very little say in who they were or where we went. I’m not in charge. Who is? I dunno. I’m just a songwriter, that’s my function in life.”

In fact, it was Ayers’s new manager, Tim Shepard, who mobilised famous fans by e-mail. An American pop artist who created The Unfairground’s eye-catching cover, Shepard met Ayers in France three years ago and, as the pair became mates, encouraged him to record songs he had been taping on an answering machine in his kitchen. Effectively, Shepard has stepped into the shoes of Ayers’s former best friend and longtime guitarist, Ollie Halsall, who died of a heroin-related heart attack in 1992, immediately after which Ayers disappeared. “Kevin can be cripplingly shy,” Shepard says. “He tells a story about arriving in Berlin in the early 1970s. He met up with Iggy Pop and a newly famous David Bowie, and the three of them scored some coke. Kevin had a 500-Deutschmark note – his only cash, to last him a week – and he took it out and rolled it up. He used it, Iggy used it, then Bowie used it and pocketed it. Kevin says that’s why Bowie became a rock star and he didn’t. Bowie had the balls to nick his money, and he was too shy to even mention it.”

Brought up in Malaysia by his mother and stepfather, a district officer, between the ages of 6 and 12, Ayers was a troubled teenager, rebelling at the boarding school to which he was sent on returning to England. (Which school? “Colditz, Auschwitz ... can’t remember,” he replies.) At 17, a drugs bust he still claims was a setup saw a judge order him to leave London, and he went to live in Canterbury, where he formed Soft Machine, prime movers in the famous Canterbury scene. Even then, Ayers’s ability outstripped his self-belief. “Canterbury was full of misplaced, middle-class kids fresh from school, not knowing what to do with themselves,” Ayers recalls. “I had no desire to become a musician and no musical talent – not that I could discern, anyway. I joined bands because that’s what my friends did. It was less a career option than a lack of career options.”

Alongside Syd Barrett’s Pink Floyd, playing underground London venues like UFO and the Roundhouse, Soft Machine helped to invent British psychedelic rock, releasing their debut single, Love Makes Sweet Music/ Feelin’ Reelin’ Squeelin’, in February 1967, a month before Floyd put out Arnold Layne. They were also first to use a customised light show, although Pink Floyd were soon sharing their lighting technician.

Ayers went solo in 1969, releasing the classic album Joy of a Toy, but, thanks to his good looks – he was a high-cheekboned blond, famous for female conquests including Nico and the wives of John Cale and Rich-ard Branson – his major label pushed him as a pop star. He fled to Europe and hard drugs, though his 1988 album, Falling Up, was seen as the start of a comeback. Four years later, he vanished.

I ask him if he envies the huge commercial success of old contemporaries, and he laughs. “That’s the worst question I’ve ever been asked. Of course, but I didn’t have the ego.” But surely to share a stage with Hendrix requires ego? “No, darling,” he drawls. “Just a good bottle of wine. Now let me take you to the pub.”

The Unfairground is released on Sept 10

Explore Music

• CD Reviews

• Live Reviews

• Classical

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010