At first, it looked like an idyllic working holiday for Steve Coogan. A month’s filming in Hawaii in August, with his 11-year-old daughter and new girlfriend in tow, with tiptop accommodation in a luxury beach house in Maui. All this for Coogan’s most prestigious film role since he set up a part-time base in LA in 2001. In Tropic Thunder, a comedy directed by his buddy Ben Stiller, concerning a hapless attempt to make a movie about the Vietnam war, Coogan was starring alongside Nick Nolte and Robert Downey Jr. His ambition to move on from his defining comic persona here – the spoof provincial chat-show host Alan Partridge – and build a reputation as a more versatile thesp in Hollywood, seemed now to be within reach.

In the event, the Hawaii trip turned into a PR calamity with an unfortunate whiff of nemesis for a man seemingly unable to shake off the rowdy ghosts of his past. Three weeks into filming, a failed suicide bid by Coogan’s friend and co-star Owen Wilson back in LA put Wilson out of Tropic Thunder and into hospital with a drugs overdose. Worse, a welter of brutal tabloid allegations appeared in which Coogan was named by Courtney Love as the malign agent of Wilson’s breakdown. Subsequent revelations that Wilson had in fact gone off the rails after splitting up with his girlfriend, the actress Kate Hudson, came too late to stop some of Love’s mud from sticking.

Coogan is no stranger to embarrassing personal publicity. Ever since he was reported to the tabloids in 1994 for showering a topless model with £10 notes on a bed – a lark he later explained as “me being crassly ironic about what you do when you’re famous” – he has had to endure several uncomfortable red-top exposés. Like the one in April 2004 when he was shamed into admitting that he had taken cocaine in a hotel room with two women he met after a charity performance at the Albert Hall.

None of these past indiscretions had been life-threatening, nor had Coogan disputed them. The lurid headlines that splashed across British newspapers on August 29 last year, however, prompted him, for the first time in his career, to threaten immediate libel action against anybody reprinting them. “Owen and I have a completely apple-pie friendship,” Coogan tells me later. “We run together on Venice Beach. I have never had a drink or taken drugs with him.” But even as he publicly criticised Love’s charges as “unfounded, unhelpful and hurtful to all concerned”, Coogan must have been aware that he had helped to bring this misfortune upon himself.

Anybody who embarks on a relationship, however brief, with a loud-mouthed loose cannon like Courtney Love is asking for trouble, as Coogan had already discovered. Following their two-week liaison in LA in 2005, the vengeful Love let rip. The highly volatile rock star, actress and widow of Kurt Cobain stormed that Coogan was an inveterate substance-abuser and “a f***ing sex addict” with a thing for orgies. It was his womanising ways, Love said, that had inveigled her into dating a loser she identified with his famously inept comic character. “Given the A-list stars I’ve dated, it’s embarrassing. I mean, Alan Partridge!” An unfounded rumour circulated that Love was carrying Coogan’s love child.

At the time, Coogan was supposedly trying to get his life back on track. He had just come through a turbulent year that had seen the break-up of his 18-month marriage to Caroline Hickman after the Albert Hall caper – she cited his “unreasonable behaviour” in the divorce proceedings – and a spell in a rehab centre in Arizona. “I’ve taken a lot of drugs but I’m clean now,” he told an interviewer. His main priorities, he said, were advancing his career, particularly in America, and stemming the flow of Coogan-the-party-animal stories for the sake of his daughter, Clare. “I have a wonderful relationship with her and I want to protect that.”

The brouhaha surrounding the filming of Tropic Thunder last August was doubly unhelpful. Along with Clare, Coogan was in Hawaii with his first serious girlfriend since his wife walked out in 2004. China Chow, the 34-year-old British-born daughter of the famous Beverly Hills restaurateur Michael Chow, founder of the Mr Chow chain, is an actress and model who has been written up recently as the woman who might, finally, ease 42-year-old Coogan out of a bachelor lifestyle that has survived the birth of his only child and one marriage. When the couple attended the Sundance film festival together in Utah in March, the gossip columnists gushed they were “inseparable”, and speculated that marriage was imminent; which it isn’t.

Though Coogan now stays with Chow in her Hollywood apartment whenever he is in LA, they see much less of each other when he is living in his house in Brighton and working in Britain, as he will be for most of 2008. Chow has her own acting career back in LA: the week’s driving holiday in the Lake District that Coogan took her on in February required some diary-juggling from each of them.

Coogan’s upcoming schedule is beyond hectic; “completely mad”, he calls it. Two months in Manchester filming Sunshine, a new comedy series for the BBC written by his friend Craig Cash, one of the creators of The Royle Family. Then a month back in Brighton working on the script for his first stand-up tour of the UK since 1998, which takes up most of the autumn, and which Coogan intends to film for a separate project. In the middle of all that, he has to fit in a US promotional tour to publicise another movie, Hamlet 2. When he says “I really would like to settle down, but circumstances don’t allow that at the moment,” you can see his point. You can also see why his girlfriend may wonder what sort of a life beckons for her as Mrs C.

Interviewing Coogan has never been easy. “I’d rather have the media misinterpret me and have a part of me stay private,” he said by way of a “no comment” when quizzed about the Courtney Love affair. In line with his patchily maintained privacy policy, his cuttings file is a bipolar collection comprising, on the one hand, pieces in which he holds forth about specific projects – his latest film, comedy series or whatever – and on the other, tabloid tales of his late-night misdemeanours. Rarely are the two sides of Steve Coogan aired in the same article. Publicists who have worked with him describe him as “basically shy” and “a bit paranoid”. When you read the mixture of soft soap and sleaze that has been written about him over the past 15 years, both comments make sense.

After the Wilson incident, Coogan went to ground for six months before finally agreeing to meet. In the interim I heard from various people who have worked with him, who were unanimous on what a live wire he is: “A brain full of ideas that carry on developing even when he’s on camera,” said an actress who was in his most recent comedy series, Saxondale. “Steve has the observational skills of the best stand-up comedians and the acting skills to assume a character,” said a fellow comic who compared him to the late Peter Sellers.

Insights into the man behind the talent were harder to come by. He doesn’t seem to do close friendships. He tends to hang out with long-term business colleagues, like Henry Normal, a former insurance executive with whom he writes scripts and runs the Baby Cow production company. “You can be talking to him and he’ll be off somewhere, inventing worlds in his brain,” Normal said. Michael Winterbottom, who has directed Coogan in several films, said he thought Alan Partridge “was an aspect of who he really is, which he exaggerates”. A woman who went out with him for a while called him “commitment-phobic and impulsive”, recounting how, at short notice, he would ring up and send a car to her London flat to drive her down to his Brighton house for the night.

That amorous roustabout is not the Steve Coogan I meet at his Soho club in February for an exploratory chat – no recorder, no notes. He arrives late, flustered and profusely apologetic, and orders a glass of white wine. Though he looks recognisable enough – if more sensibly dressed than his comic creations tend to be, in a blue windcheater and cream chinos – Coogan passes largely unnoticed in this well-populated media lair, despite being by some distance the most famous person in the place. He says he hardly ever gets stopped on the street. Perhaps that’s because he is naturally diffident and has no aura. The earnest stare, which draws laughs when he deploys it in character on stage or screen, is the default expression of a guy who seems more nervous than most, and slightly uncomfortable in his own skin. As Winterbottom says, there is a hint of Partridge-like gaucheness about him.

Like many professional funnymen – John Cleese and Woody Allen spring to mind – Coogan is almost studiously unfunny in person. Rather than tell amusing stories, or put on the funny voices that are his forte, he prefers to ponder the nature of his craft. He quotes a remark by Henry Normal that comedy is “a mixture of music and mathematics”. He says he likes the “deep pathos” that humour can sometimes tap, but he also likes the broad knockabout stuff. “When I’m in front of 2,000 people making them all laugh at the same thing, I’m in a Zen state, the most comfortable place I know.” He is proud of the fact that his audience is made up of regular blokes – he calls them “the lager crowd” – and more serious-minded, chin-stroking types.

But he’s certainly not vain; rather, he’s scrupulously modest. He attributes his success to his ability “to hitch my bandwagon to other people’s talents”, pointing out that his Alan Partridge character was a team effort. Partridge was originally devised in collaboration with the radio comedian Armando Iannucci and the playwright Patrick Marber, and later developed with the writer Peter Baynham, who went on to work with Sacha Baron Cohen on Borat. Coogan’s most recent creation, the roadie turned rat-catcher Tommy Saxondale, was made up with a previously unknown Scottish writer, Neil Maclennan, who sent him an unsolicited script he liked. He needs to write with other people, he says, “because I have to have a dialogue. I need somebody to listen to me, basically. Without that, I can’t tell what’s really funny”.

So when he says he “can’t stand having smoke blown up my arse” and doesn’t want this article to be “a whitewash”, you want to believe him – even though he’s noticeably guarded about what he talks about outside his work. For a lot of our hour-long conversation, he discusses Brighton and its posher sister town, Hove, to which he relocated from London in 1998 and where he still lives in a six-bedroom Victorian house with a swimming pool in the garden. On his own. He says the house “is too big for me” and that he bought it in the hope that “things would pan out differently. I thought it would become a family home, and that hasn’t happened”.

He is most talkative about his daughter Clare’s schooling. She lives, most of the time, with her mother, Anna Cole, a solicitor. Coogan is at pains to stress how well he gets on now with his ex-girlfriend – with whom he split up around the time Clare was born – and is equally keen not to reveal anything about her. His ex-wife, Caroline, is another no-go area: “I don’t want to talk about my relationships. It’s my choice to have a public profile, but other people haven’t made that choice. I have to respect that.”

Six weeks later, at a vegetarian restaurant in Brighton, he is in a more buoyant and expansive mood for the interview proper. He has recently returned from Utah, where Hamlet 2 – a low-budget independent production starring Coogan as an American high-school drama teacher staging a bonkers sequel to the Shakespearian classic – was the big hit of the Sundance film festival. It was bought for $10m by Focus Features, making it the second biggest deal in Sundance’s history, and will be heavily promoted ahead of its cinema release in August. Coogan thinks it’s his best shot yet at making a breakthrough in America, which he describes, in his self-effacing way, as “an experiment. There’s a question mark over whether I can forge a meaningful career there. I’m still unknown. For somebody to pay that much means they’re investing in my potential”.

) ) ) ) )

The first people to recognise Steve Coogan’s potential were his brothers and sisters. He grew up in north Manchester, the fifth of seven children – five boys, two girls – in a devoutly Catholic household. His socialist parents also engaged in short-term fostering, which swelled the numbers in the Coogan nest to the point where a visiting policeman once mistook the house for a community centre. “It was a very stable, solid background,” Coogan says. “And it was character-forming. I had to vie for attention.”

His main ploy was to mimic the voices of characters in the TV shows that the family would watch together every night. In the days before video, nine-year-old Steven found he could hold the attention of the entire household by replaying, with precocious accuracy, key scenes from favourite BBC comedies such as Fawlty Towers and Dad’s Army. “Get Steve to do it” became a regular family refrain.

It was a rare talent, but it didn’t get Coogan into any of the London drama colleges he applied to after secondary school, a memory that still rankles. “I used to dream about this other cosmopolitan world in London, where sophisticated jet-set people would drink cocktails and drive Aston Martins. I felt like a northern reject.” That feeling was amplified by the people he met at his Rada interview. “They all had names like Sebastian and Julian, and wore long scarves and Byronic hairstyles. They seemed to have a built-in confidence, and I didn’t have that.” Coogan’s perception of himself as an outsider – borderline chippy – stayed with him as his career took off. While still a drama student at Manchester Poly, he landed a plum job doing voiceovers for Spitting Image, the TV puppet show on which he impersonated John Major and Neil Kinnock. It gave him his first break on the stand-up circuit, but didn’t gain him any cred on the metropolitan comedy scene. “They thought I was just a mimic, for God’s sake, not a proper comedian.”

Audiences disagreed. Coogan won a Perrier comedy award at Edinburgh in 1992 for a one-man show in which he appeared as a lager-swilling yob, Paul Calf, a yuppie called Gavin Gannet and other buffoons. Soon afterwards he was hired by the BBC for a new radio comedy series, On the Hour, a spoof current-affairs programme fronted by Chris Morris. Coogan played a haplessly ignorant sports reporter with a hilarious obsession about injuries “in the groinal area”. Alan Partridge went on to appear in On the Hour’s television sequel, The Day Today, before leaving the commentary box and taking flight as a naff chat-show host from Norwich, the star of Knowing Me, Knowing You. At this point, Coogan knew that he was “cutting edge. I wouldn’t have swapped places with anyone”.

By the mid-1990s he was living the dream he had had as a Mancunian teenager: resident in London, driving expensive sports cars, consuming cocktails, you name it. Which the tabloids often did. A neighbour of Coogan’s at the time remembers a steady throughput of lady visitors to his house off the Holloway Road. He admits that the dream went sour eventually, that he was “living in a sea of madness. I was like a kid in a candy shop, but at the end of the day, you have to wake up with yourself”. His wake-up call came after his girlfriend Anna Cole upped sticks to Brighton in 1996, taking their daughter with her. A year or so later – as a shower of Bafta awards in 1998 enshrined Partridge as one of Britain’s all-time comedic favourites – Coogan followed, and has never regretted it. He says he loves the “relaxed, post-hippie attitude of the people here” and enjoys the healthier lifestyle. He now runs along the seafront every morning, jogging down to the house where his friends Zoe Ball and Norman “Fatboy Slim” Cook live.

Though he still retains a flat in Notting Hill, he never stays there, preferring to do a day’s commute by train whenever he needs to visit the Baby Cow office. London is now back to where it was in Coogan’s world: an alien place that he doesn’t quite fit into.

To judge from the work he has done these past 10 years, you might conclude that he has put his rambunctious past behind him. He has become less of a gag man and more of a character actor – playing the 18th-century eccentric Tristram Shandy in A Cock and Bull Story, and the eponymous diarist in The Private Life of Samuel Pepys. It was his fine portrayal of the late Tony Wilson in 24 Hour Party People that opened the door for him in America.

The return of Alan Partridge in 2003, and the introduction of Tommy Saxondale in 2006, both suggested he was less interested in belly laughs than knowing smiles. He agrees, saying he’s trying to capture “universal truths that make people laugh in a different way, by shining a little light on what it means to be human”.

Unfortunately, this hankering after wisdom hasn’t always extended into his personal life, as witness the circumstances of his failed marriage and spell in rehab in 2004. The presumption at the time was that he had a serious drug problem and an uncontrollable libido. “Let’s say I had certain issues that needed to be addressed,” he says, adding after a long pause: “And I… er… dealt with them.” Does he still take drugs? “No, I don’t. Clarity of mind is a very precious thing to me. If you have an addictive personality, which I think I probably do, you have to channel it in a positive and productive way.” Which is why he has recently created such a heavy work schedule, to keep himself out of trouble? Long pause. “I am a workaholic. And that is another ‘holic’, a form of addiction. And I do have to be careful about that.”

Before I leave him, Coogan takes me back to his house in Hove, where some photographs for this article are being taken in the garden, which backs onto the cricket ground where the Sussex county side plays. You can see what he means by his earlier remark that he “rather rattles around” in here. Spread over five floors, with the office where he writes at the top and a large basement that he might turn into a home cinema, it has the size and slightly anonymous feel of a smart guesthouse. Signs of occupancy are scarce: a pile of magazines in the range kitchen, a packet of fags on the dining table, a grey Porsche 911 in the garage. There’s one photograph of Clare on the kitchen mantelpiece. Nothing denoting the new girlfriend. As he points out the 17th-century map of Asia on the wall in the hallway, and sits down at the piano, unprompted, to play Bach’s Air on the G String, you get the sense that he would like to be seen as cultivated, and settled. You also get the sense that gestures in that direction are possibly all he has time for.

Steve Coogan is a work-in-progress, and he knows it. He says he can’t decide whether to keep this house or move to the country, maybe even to rural Ireland, where his grandparents came from and where he loves to go walking with his brothers. He also says that he can’t wait to get back on the road this autumn with his stand-up show, Steve Coogan Is Alan Partridge and Other Less Successful Characters.

There’s something touchingly naive in his wavering, as there is in his diffidence, not to mention that earnest stare, which deepens considerably when I ask him if he thinks he might have grown up too fast. The longest pause of the afternoon ensues. “I didn’t really have an adolescence: I sort of put that off till later,” he says. “And I sort of wish now I hadn’t.”

Explore Film

• Film Reviews

• DVD

• Oscars

• Bollywood

Editor's choice

• The End: the worst movie endings ever
• Introducing The Times Proms Podcast
• Heath Ledger tipped for Oscar