You would have thought that if something extraordinary — something really markedly unusual — happened to the biggest pop star in Britain, it would, at least, be commented on.

Apparently not. In 2002 Robbie Williams decided how he wanted to reinvent himself. Having split from his writing partner Guy Chambers, Williams had no shortage of surefire hitmakers queueing up to work with him. But he had other ideas. Five years earlier, his keyboard player Clare Worrall had played him a record by the Lilac Time — the cult folk combo fronted by Stephen Duffy. Chris Heath, the author of the recent Williams biography, Feel, was there for the moment when Robbie made an impromptu visit to Duffy’s tiny North London studio. On one side of the room, there’s Robbie Williams — recipient of a record-breaking £80 million deal; on the other, Stephen Duffy, holder of another record — the only recording artist in Britain to be dropped by five major record labels. Two weeks later, the two were working on what was to become Williams’s sixth No 1 single, Radio .

“He just sang it off the top of his head,” Duffy explains, youthfully beatnik at 44; his own Warhol portrait in the attic. “It was the first take. It was insane. As a fan of Blonde on Blonde period Dylan, to actually work with someone who is coming up with this stream of consciousness stuff — and doing it with a journalist and a bodyguard sitting next to him . . .”

When Radio was finished, both Duffy and Williams were fairly certain that people would notice what an odd, unexpected song it was. It is, after all, a stabbing, slightly lopsided piece of Burroughsian electro-pop, which appears to centre on the notion that some hidden, malevolent aspect of Williams’s own id is haunting him. By stealing his Oscars and, um, combing his ’fro.

“It’s the most radical single released by a major artist since, what?” Duffy asks. “And yet when it comes out, all anyone talks about is the fact that he’s going out with a probation officer from Newcastle.” He fills his glass with wine, adding drily: “It beggars belief.”

Of course, Duffy is no stranger to how an ingrained public image can outweigh contrary hard evidence. Aged 18, he left the unsigned Duran Duran to form an electro-pop duo called Tin Tin. The other guy left and Duffy briefly became a pop star — Stephen “Tin Tin” Duffy. Then, to the horror of his record company, he had a revelation (while sitting on Noël Coward’s grave in Jamaica, off his face on Ecstasy and listening to Nick Drake): it was time to move to the Malverns and form a pastoral pop-folk band, the Lilac Time (named after a line in Drake’s River Man). He spent the next few years releasing a succession of radiantly romantic albums; yet to the population at large — at least those sections of the populace who remember him — he continues to be the big-eyed Brummie art student who gave us Kiss Me.

It’s little wonder that in 1993 — in a collaboration that presaged his partnership with Williams — Duffy hooked up with a fellow Malvern resident, Nigel Kennedy, who was also stuck with an outdated public image, as a public school faux-punk. Yet Kennedy was keen to get off his face on mescaline and make psychedelic pop.

“I was smoking a lot of pot,” Duffy says, recalling their alliance. “I was out of my mind — and very keen to get the word ‘opalescent’ into a song.” Their resulting album, Music in Colours, is a lost, very English, absolutely incongruous gem. Written at the point where ecstatic abandon is about to turn to tears, Kennedy’s baroque, wah-wah violin lends reds and blues to Duffy’s pencil-sketch poetry.

If you missed it the first time, you’ll be pleased to know that it was reissued last month to a slew of critical superlatives. In the heyday of grunge, however, its reception couldn’t have been more different. And yet it was by no means a difficult album. On the carnal lit-pop of Natalie, an aggrieved Duffy sang: “Now not sleeping around’s enlightened/ What did we do wrong?” — while Holte End Hotel had Duffy doing for Aston Villa ’s football ground what Penny Lane did for the streets of the Beatles’ youth. The problem was that Duffy’s career moves — working with Kennedy, singing words such as “opalescent” — were deemed so unfathomable that the music press just decided to ignore him, and hope he went away. Which, as it happens, he did — he became so depressed at the album’s reception he ran away to Alaska for a year.

Consider that Duffy has spent the best part of 20 years having his career progressions ignored, and his current rapport with the similarly misunderstood Robbie Williams makes more sense. Indeed, their next collaboration, the single Misunderstood, could almost be their mutual anthem. Duffy, however, refuses to be morose about it.

“Last year, I was thirteenth in Mojo’s list of the unluckiest artists of all time. But I’ve never had to pack sandwiches for a living. Every penny I’ve made has been through music. Besides, most of the other people in that list were dead. So that’s quite lucky, isn’t it?”

In the New Year, Duffy flies to Los Angeles to work on Williams’s forthcoming album. With 50 songs already in the bag, both are determined to “go for it”. “It has to be the heavy s***, no messing around. He’ll be there on every song, playing, right in the middle. It’s going to be Plastic Ono Band at No 1.”

If it is, will anyone notice?

• Misunderstood is out now on Chrysalis

Explore Music

• CD Reviews

• Live Reviews

• Classical

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010