Jonathan Coe takes the news that he is not on this year’s Man Booker longlist rather well, I think. Sitting in a pub in deepest Kerry just two hours after the announcement, he simply shrugs and says how sorry he feels for his publishers: “Three of my novels have been eligible now, and, so far, nothing doing.” Not a big deal, he assures me.

Coe is a famously nice, self-effacing man – wry, gently melancholic, quietly cerebral, a successful literary novelist with almost no enemies (something of a feat in bitchy literary London). But, although he has always insisted he doesn’t believe in literary prizes (a comment that came back to haunt him when he won the Samuel Johnson prize for Like a Fiery Elephant), one wonders how much the Man Booker no-show really means to him. Because, whether Coe likes it or not, his eighth novel, The Rain Before It Falls, does represent a substantial risk for the 46-year-old, Birmingham-born author.

Like Martin Amis and Joseph Heller before him, Coe is a writer burdened by the success of one novel. For Amis, it was Money; for Heller, it was Catch22. For Coe, the literary lodestone has been his fourth novel, the 1994 hit What a Carve Up!. An international bestseller, this exuberant slice of social and political burlesque so effectively and joyfully skewered the Thatcher era that many fans have been clamouring for more of the same ever since. The House of Sleep (1997) dished up similar comic bravado alongside more philosophical fare, and both the semi-autobiographical The Rotters’ Club (2001) and its 2004 sequel, The Closed Circle, offered generous helpings of social and political comment to go with the more personal story lines (the former on the 1970s; the latter on the Blair years). But a persistent niggle for Coe gourmands is that those are all just tasty side dishes, when what they really want is a substantial second slab of the main course.

Rather magnificently – and The Rain Before It Falls is a hugely affecting novel, full of poignant set pieces – the new book simply fails to oblige. Indeed, it dispenses with many trademark Coe tics altogether. Gone are the humour (I counted just one joke, which Coe almost took out in the edit); the easy, chatty style; the cut-and-paste interventions; the pop-culture allusions; and the satire. Instead, Coe has foregrounded a more melancholy sensibility, one he insists has always been there. “In books such as What a Carve Up!, there are lots of different tones jostling for supremacy,” he says. “One is the tone I’ve adopted. This time, though, I’ve just isolated it and concentrated on it.There’s really no sociopolitical dimension to this story at all.” Instead, the book focuses movingly on ideas of regret and loss, and the marks left by one generation on those that follow. Chief spear-carrier for these notions is the main narrator, Rosamond, an elderly, childless spinster, whose funeral takes place within the first few pages. Visiting her Shropshire home afterwards, Rosamond’s niece Gill discovers disturbing mementos – an empty bottle of diazepam, and a package of tapes and 20 carefully arranged photos that her aunt wants delivered

to a blind girl called Imogen, whom Gill last saw as a child at a family party 20 years before. Failing to locate Imogen, Gill finds herself listening to Rosamond’s taped account of her lifelong, bittersweet friendship with the girl’s grandmother, Beatrix. As Rosamond proceeds, photo by photo, through her own life, both the source of her long-felt affection for the girl and the fraught circumstances of Imogen’s blindness become apparent.

The book has been turning over in Coe’s mind for more than 20 years (most of his novels have elephant-like gestation periods: The Rotters’ Club was conceived at school). “Novels for me often start with one image,” he says. “For this one, it was meeting a little seven-year-old blonde blind girl at a family wedding in the late 1980s. This girl was moving around outside in the garden – nobody I met really seemed to know who she was, but she had an extraordinary quality of stillness and centredness. I never saw her again, but always wanted to write a novel about her.”

Why such an image should have stayed with Coe for so long becomes clearer when you look at his career as a whole. If there is one giant, tungsten-steel thread running through his work, it is not to do with politics, but his own bittersweet childhood. For Coe, the past’s pull appears irresistible – his books teem with characters yearning for lost innocence or bemoaning some teenage blow. That is particularly true of The Rain Before It Falls, in which Rosamond’s juvenile fixation with Beatrix, and Beatrix’s unhappy experiences with her own parents, warp and dominate both their lives.

One might imagine that the source of Coe’s fixation was some boyhood anguish, but he admits that his early years, spent with his teacher mother and electrical-engineer father in the suburbs of Birmingham, were almost superabundantly joyful: “A lot of novelists write out of trauma. I don’t. My childhood was almost unnaturally untraumatic. In fact, that sense of loss and regret in my books may come, paradoxically, from the fact that I had a very happy time and I’m constantly trying to get back to it.”

You find childhood almost everywhere in Coe’s writing – in the Shropshire landscape, for instance, that so dominates The Rain Before It Falls (his grandparents lived on a farm very like the one featured), and in the photos that so powerfully drive the narrative (many of them draw on family snaps: he wrote using a split screen, photos on the right, text on the left).

Coe’s fictional attachment to the Midlands – he calls himself a provincial novelist, despite having lived in London for 21 years – also has its roots in his youth. “My feeling is that you can only write with real authority about a place you feel a strong emotional attachment to,” he admits. One can even trace Coe’s very career as a writer back to his early days: “I’ve always remembered being nostalgic, from a very young age. And the point at which I felt my childhood slipping away, at 16 or 17, was the point at which I started writing seriously.” At Cambridge – a traumatic experience for a Midlands direct-grant boy – he locked himself away in his room, writing.

In terms of wider literary inspirations (Coe went on to do a PhD at Warwick on Henry Fielding), The Rain Before It Falls owes a considerable amount to two rather surprising writers. One is BS Johnson, a modernist novelist of the 1960s and 1970s, who was the subject of Coe’s biography Like a Fiery Elephant, and whose tortured credo that “telling stories is telling lies” haunted Coe during the writing of the new book (he admits, in fact, that his novels have been getting darker ever since the Johnson biography). The other, happier influence is Rosamond Lehmann, whom Coe first read in his mid-twenties, and whom he considers one of the 20th century’s key writers. Everything, from his staccato narrative style to the names Rosamond and Beatrix (Lehmann’s sister) and even the plot (a take on her 1944 novel The Ballad and the Source), he says he owes to her.

He may also owe to her, indirectly, his extraordinary facility with female voices.

Few male British writers come anywhere near Coe’s ease with women characters, which he puts down to his family – both his young children are girls – and to strong female friendships. “I would have preferred to be a woman, I think. I don’t feel strongly enough to undergo surgery or anything, but, for me, being able to write in the voice of the other gender is first base for a novelist.”

As to the future, Coe is planning another family novel, about fathers and sons. “I have a hunger to read about men by men – to have maleness analysed and explained. Nick Hornby and Tim Lott are doing it to a certain extent, but that’s not why you would read Martin Amis or Julian Barnes.” After that, he intends to return to Rosamond’s family.

And does that mean the end of the knockabout political novels? “I can’t imagine myself returning to the tone of What a Carve Up!,” he says. “The themes in it seem quite small now, compared with what the world’s facing.

“How do you write humorously about terrorism or global warming?”

Explore Books

• Book Extracts

• Book Reviews

• Books Group

• Audio Books

Times Recommends

• The 100 Best Books of the Decade
• The Times picks the 50 best paperbacks of 2009
• The conversation: James Ellroy