There is an old Zen fable, in which a man says to his yoga teacher: “I have tried to break my habit of going to wine shops and brothels, but I can’t do it. I am in chains to my nature. You can’t expect a man in chains to do anything.” Some days later, the teacher meets him going to the town one evening. He is smartly dressed and walking briskly in anticipation. The teacher says: “You don’t look like a man in chains.”

I was reminded of this when reading Chris Martin’s latest, and characteristically disingenuous, comments about the new Coldplay album, Viva la Vida or Death and All His Friends. As with previous album campaigns, Martin professed reluctance, humility and “what, me?” disbelief about the millions of copies of Viva la Vida that he is about to sell to a mainstream global audience that obviously cares not a jot for his oft-avowed unease about being so successful. The old “Coldplay contra mundum” canard has been trotted out once more. “We’re about to be fed to the lions again,” Martin said to one interviewer, as if Coldplay face unique, almost insurmountable odds in their bid to be heard, loved - and bought.

Critical disdain doesn’t seem to dent their appeal: their last album, 2005’s pleasant but lyrically trite and ultimately insipid X&Y, sold an estimated 8.3m copies, despite The New York Times describing Coldplay then as “the most insufferable band of the decade”. That’s surely a bit harsh, but perhaps we should take Martin at his word and appraise Viva la Vida on the basis that it is the work of a man (he is the band’s principal songwriter) who is deeply uncomfortable with the mainstream, with accessibility - although, confusingly, he has proved remarkably adept at writing radio-friendly hits such as The Scientist and Clocks, yet still can’t write a chorus to save his life - and cleaves instead to the profound, the opaque, the experimental.

The involvement of everyone’s favourite studio sage, Brian Eno - who shares production duties with Markus Dravs (Arcade Fire) - pointed to a bolder, more challenging album. Lo and behold, a multitude of 64-track bells and whistles, in the form of North African strings, crunching R&B beats, backing vocals recorded in Spanish churches and song structures that suddenly yank you off in quite unexpected directions, are chucked at the music. Rock as a genre has been here before and will come here again: when you’ve conquered the world’s charts and stadiums, what next? Do you get louder, dancier, weirder, safer? Well, no - in most cases, you hit a wall and, having done so, come back with more of the same under different wrapping.

Viva la Vida’s sonic squiggles and fancies act as a sort of evil twin to Martin’s innate musical politesse. Yet, for all that his producers seek to lure him into darker corners, the wide-eyed, zealous and populist pastor in Martin keeps reasserting himself. “Just because I’m losing,” he avers sloganistically on track three, “doesn’t mean I’m lost.” Later, on the album’s low point, Lovers in Japan, he mews, with clunking repetition: “Lovers keep on marching on / Lovers, until the race is won”. Even his self-doubt seems suspect: “Time is so short and I’m sure there must be something more,” he sings on 42, one of several tracks that contemplates, to not especially enlightening effect, the meaning of life and death.

There are some truly beautiful moments. Musically, Reign of Love, 42 and the end of Violet Hill all have a hymnal quality, evident in much of Martin’s work, that could, allowed more room, have led to a fascinating and genuinely contemplative and subdued record.

But the showman in Martin cannot, it seems, be suppressed.

Like Tony Blair, with whom he shares a penchant for soundbites, calculating candour, woolly interfaith-speak, self-deprecation and excruciating blokeish humour, Martin feeds off affirmation and the imagined resistance of fantasy enemies. Blair stormed to three election landslides. Coldplay have released three huge albums, and they, at least, will triumph again with their fourth. Will Martin run a mile from this, aghast, defeated by the love of the masses? Or will he walk, chains and all, towards it - briskly, in anticipation?

Viva la Vida or Death and All His Friends is released on Parlophone on Thursday

Explore Music

• CD Reviews

• Live Reviews

• Classical

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010