This is a story of envy, backbiting and revenge that details how an Irish novelist ranked as an outsider overtook the hot favourites to win Britain’s highest literary honour, the Man Booker prize for fiction.

When John Banville, author of the lyrical novel The Sea, was proclaimed the winner on Monday, a shocked hush fell on the glittering gathering at the awards ceremony in London’s Guildhall. Ice began to form on their upper slopes.

Then the invective began. “Yesterday the Man Booker judges made possibly the worst, certainly the most perverse, and perhaps the most indefensible choice in the 36-year history of the contest,” fulminated Boyd Tonkin, The Independent’s literary editor. Banville’s prose, he wrote, “exhibits all the chilly perfection of a waxwork model” and the result was “a travesty of a judging process”.

Privately, many figures in the literary world were also foaming at the mouth. One characterised the 59-year-old Dubliner’s work as “empty, vapid, cold, humourless, self-indulgent, snooty and pretentious”. Another confessed he was “flabbergasted” at the result.

The distinct lack of acclamation left Banville perplexed. “If they give me the bloody prize, why can’t they say nice things about me?” he complained.

So far, so normal: the Booker has a hallowed tradition of producing winners that induce apoplexy. But the reasons for anger went deeper this time. On the surface the trigger was Banville’s triumph over the best field of fiction writers in years. It had been a foregone conclusion that if the £50,000 prize was not awarded to Julian Barnes, the bookies’ hottest tip, for his acclaimed Arthur & George, it would go to Kazuo Ishiguro for Never Let Me Go, Zadie Smith for her third novel On Beauty, Ali Smith’s The Accidental, or Sebastian Barry’s A Long, Long Way.

There was another subtext, however. Some have never forgiven Banville for his demolition job on Ian McEwan’s latest novel Saturday in The New York Review of Books in May. The book had been greeted with rave reviews elsewhere and was originally the bookies’ favourite for the Booker prize, but after Banville’s mauling it never made it to the shortlist.

In a 3,000-word review Banville took a stiletto to McEwan’s “dismayingly bad book” and “self-satisfied and . . . ridiculous novel”. The book’s set pieces were “hinged together with the subtlety of a child’s erector set”, he wrote.

This savaging, it was suggested, might be revenge for the fact that Banville’s novel The Untouchable had failed to make the Booker shortlist when McEwan won with his novel Amsterdam in 1998. Last week Banville dismissed the notion as ridiculous.

Whatever the reason, support for McEwan leached away. “The tide seemed to turn,” says Peter Kemp, The Sunday Times’s fiction editor. “I can’t imagine it was just because of Banville’s article, but people who had rated the book highly suddenly found all sorts of things wrong with it.”

Banville’s own contender, The Sea, suffered a rough passage when published. It is the story of a bereaved Irish alcoholic, Max Morden, who returns to a holiday resort of his childhood and stirs up memories of a family he fell in love with.

The Guardian’s critic Finn Fordham described the author as “a ventriloquist” who brought to life monologues by “fleshed out dummies”. David Grylls, in The Sunday Times, was irritated by such epithets as “velutinous” and “cinereal”, and scathing about Banville’s style of “puffed up grandeur”. The Sea, he concluded, “is a crashing disappointment”.

Banville veers between sounding peeved and proud of his small sales, while railing against commercialism. And herein lies the ultimate revenge theory in last week’s drama. It concerns the ambiguous role of the chairman of the Booker judges. It was John Sutherland who leapt to McEwan’s defence after Banville’s harsh review of Saturday, taking the critic to task in a published letter.

So what explains Sutherland’s crucial vote for Banville when the Booker panel was split? Even the Irishman was astonished: “When I read Sutherland’s letter I thought, well, I can kiss the Booker goodbye.”

Sutherland claimed he was persuaded by the forceful rhetoric of Banville’s protagonists. But was his motive more subtle? “Banville keeps going on about the evils of commercialism,” says one cynic. “Sutherland decided to give him a taste of it.” If true, the strategy is working: sales of The Sea have tripled and Banville has been signed up for a series of historical “thrillers”. His moral torment has begun.