To Iraqi parliamentarians meeting in the comfort of the green zone in August 2004, it seemed like a sensible idea: dispatch a delegation of political and religious leaders to the holy city of Najaf to get Muqtada al-Sadr to dissolve his militia and vacate the Imam Ali shrine. It was to be a final opportunity to avert a bloody military operation, to convince al-Sadr that, after several failed attempts to put him out of business, the Americans and their hand-picked Iraqi government were going to finish the job this time. If he did not disband his fighters, warned American officials, he would be killed or captured.

Foolishly thinking that this might be the end to the al-Sadr menace, I went along, too.Once we entered the gold-domed compound, the most sacred place in the world for Shi'ite Muslims, it became clear that the green zone assumptions were dead wrong. Although thousands of his fighters were in and around the shrine, al-Sadr was nowhere near there. He had no desire to meet the delegation. And no intention of surrendering.

The next day, as the American military intensified its attack against the Sadrists, I wondered if al-Sadr was a madman or a clever strategist, a man who could manipulate just about everyone in Iraq. The American military would go on to kill hundreds of his Mahdi Army militiamen but not al-Sadr. Fearful that the shrine would be destroyed, Shi'ite elders brokered a cease-fire. The Americans pulled out of Najaf; in turn, the Sadrists gave up their weapons, a few carts of rusty arms. Al-Sadr lived to fight another day.

No Iraqi has negotiated the chaotic landscape of post-Saddam Iraq as deftly as al-Sadr. Written off by the Americans and many of his fellow Iraqis as an empty turban, an unstable young man who hails from a line of highly respected clerics but lacks religious education, a rabble-rouser more interested in picking fights than promoting harmony, al-Sadr has become the most powerful person in Iraq today. He commands a broad base of support in Baghdad's Shi'ite neighborhoods and across much of central and southern Iraq. His militia can hold its own against Iraq's new army and even the much better equipped American military. In Shi'ite areas, he can turn violence on or off with an order.

Patrick Cockburn was among the first journalists to grasp al-Sadr's influence. In this fascinating biography, he neatly punctures the myth that al-Sadr is a crazy gangster who stumbled into leading thousands of disaffected young Shi'ites in armed rebellion. Instead, he draws upon his own reporting and often overlooked aspects of Iraqi history to present a compelling, detail-packed tale of how al-Sadr outflanked everyone from Saddam Hussein to the Grand Ayatollah Ali al-Sistani.

Cockburn spends more than half the book recounting the modern history of Iraq's Shi'ites to make one overriding point: that the first but by no means sole reason for al-Sadr's influence is his lineage. His cousin and father-in-law, the Grand Ayatollah Muhammad Baqir al-Sadr, was one of the most senior and respected leaders of Iraq's Shi'ite community. Baqir chafed against the largely secular, Sunni-led tyranny of the Ba'athists and sought to overthrow the regime by endorsing terror attacks against government figures. Saddam eventually ordered him and his sister to be arrested. Sadrists believe Baqir was forced to witness the rape of his sister before he was executed.

Al-Sadr's father continued building the Sadrist opposition movement until he, too, fell foul of the Ba'athists. In 1999, he and two of his sons were gunned down by Saddam's henchmen at a Najaf roundabout. It fell to al-Sadr, then aged 25, to assume leadership of the family and, by extension, of the most significant internal opposition movement. But in the first indication of his ability to pull back from the brink, he opted not to confront the government directly, lest he suffer the same fate.

Cockburn's emphasis on history, though at times tedious, offers some additional lessons, most notably the Sadrists' thirst for revenge spawned by Ba'athist repression, and the animus toward other Shi'ite opposition groups because of their failure to aid the Sadrists in their moments of need. It helps to explain why the Mahdi Army has brutally terrorised the Sunni community, and why al-Sadr, to this day, remains so wary of collaborating with other Shi'ite political leaders.

As American troops converged on Baghdad, Cockburn notes that al-Sadr “moved more quickly than anyone else to organise his supporters”. Within a few days, they had seized control of most schools, mosques and government buildings in Sadr City, the sprawling Shi'ite slum that is home to more than 6m people. I remember visiting the hospital there a few weeks after Saddam was overthrown. The place was under the control of a 31-year-old electronics technician who had not been to medical school. His only claim to power was a one-page edict from al-Sadr's office.

In those early days of the American occupation, al-Sadr did not regard himself as a renegade. He wanted a seat in Ambassador Paul Bremer's governing council, but he was blackballed by other Shi'ite leaders. It was the first - and perhaps best - opportunity to bring him into the tent. But he would make the most of the snub, fashioning himself into the most prominent Shi'ite critic of the occupation and picking up thousands more followers along the way.

“One of the grossest of US errors in Iraq was to try to marginalise him and his movement,” writes Cockburn. “Had he been part of the political process from the beginning then the chances of creating a peaceful, prosperous Iraq would have been greater. In any real accommodation between Shia and Sunni, the Sadrists must play a central part.”

As the months wore on, al-Sadr would overplay his hand at times (he once sought to set up his own shadow government), but he managed to build a following that was enough to make Iraqi leaders fearful of the consequences of killing or capturing him. When Bremer sought to finish him off in April 2004 by provoking a fight, Sadrists fought back so fiercely that the Americans had to call in brigades of reinforcements. Although thousands of his militiamen were killed, al-Sadr survived because Iraqi politicians did not want to assault his redoubt. The Americans, Cockburn writes, “achieved the exact opposite of what they wanted and had elevated Muqtada into a major player”.

This book is essential reading for anyone seeking to understand Iraq today. The recent fighting in Basra and Sadr City has once again demonstrated al-Sadr's cunning and power. American military commanders boasted last year that their troop surge had led al-Sadr to flee to Iran and that his militia was splintering. But he appears to have made another shrewd move. Instead of fighting it out with the Americans, he opted to cut a deal with the prime minister, Nouri al-Maliki, to stand down the Mahdi Army - for the time being. Nobody knows for sure what al-Sadr's plans are, but it is clear that he and his militia are not a spent force.

Muqtada al-Sadr and the Fall of Iraq by Patrick Cockburn
Faber £16.99 pp272

Times Recommends

• The Sunday Times 100 best holiday reads
• What will we be reading this summer: the booksellers predict
• Literary festivals: a survivor’s guide