An American patrol ambushed by the Taliban in Uruzgan province on May 19, 2006 found its foes growing in number as the firefight continued. Local farmers working in nearby fields rushed home to get their weapons and join in. In Afghanistan, young men like doing that sort of thing.

The episode exemplifies David Kilcullen's thesis: that many Muslims who take up arms against the West in Iraq, Afghanistan and indeed Europe are not committed ideologues. Instead, they are young men alienated variously by foreign intrusion, corrupt government, local factionalism and grievances, bitterness about globalisation; or simply enthused by a belief in the dignity of combat.

At the heart of this significant book is the author's declaration that terrorism cannot be addressed by military means alone; that for American or British soldiers merely to kill insurgents is meaningless. He urges policies based upon securing and succouring populations, not on enemy body counts.

The “accidental guerrilla”, he says, is often created through a cycle of infection, contagion, intervention and rejection. By this, he means that allied responses to perceived Islamic militant threats frequently recruit new insurgents for Al-Qaeda, as happened in Iraq.

Kilcullen is an influential man. A former Australian army officer, he became a key adviser to General David Petraeus, and then Condoleezza Rice's principal counter-terrorist strategist at the State Department. His book synthesises lessons that America has learnt by bitter experience, and that, hopefully, will continue to influence its policies in the Obama era.

He was hostile to the 2003 Iraq invasion and never subscribed to Bush's idiotic “global war on terror”, recognising that all insurgencies are different. America, he says, failed to see the many rival and mutually hostile strands in militant Islam, and made the mistake of taking on all of them. He deplores the loss of American moral authority at Guantanamo Bay and Abu Ghraib prison. In future, “American power must be matched by American virtue”.

No sane enemy will again fight a military superpower on conventional terms: “The efforts of insurgents and terrorists since 9/11...have already put an end to the much-bruited military superiority of the US, showing the way to all future adversaries and leaving western powers with fabulously capable and appallingly expensive militaries that are precisely adapted to exactly the wrong kind of war.” The cash significance of asymmetric warfare is that Al-Qaeda staged the twin towers attacks for less than $500,000, while Iraq costs America $400m a day.

Kilcullen calls for a dramatic rebalancing of the forces deployed against terrorists and guerrillas. Today, there are 1.68m Americans in uniform, compared with only 6,000 State Department foreign service officers and 2,000 USAID personnel. He argues that any future foreign interventions should focus upon forging close relationships with host societies from the bottom up, and deliver much more visible development benefits.

Instead of dramatic short-term combat commitments, continuous engagement with threatened societies and a broader containment approach are needed. Kilcullen believes that we must keep trying in Afghanistan, because the evolution of a “greater Pashtunistan”, straddling Pakistan and possibly with control of its nuclear weapons, is unacceptable.

So the West has to brace itself to stay the course for years ahead. Success will come from building local relationships with the tribes and wooing all but irreconcilable militants into mainstream society. He quotes the Vietcong veteran who said proudly that his movement had turned a mouse of insurrection into an elephant. The author - and Petraeus - are preoccupied with the challenge of how to turn an elephant back into a mouse.

It is futile to dispatch western forces abroad merely to cull Al-Qaeda fighters: that lesson has been painfully learned. Success in Iraq, Afghanistan and, indeed, Pakistan can come only from helping the people to build societies that they themselves believe in. It is a shocking manifestation of allied policy failure that today many Afghans perceive Taliban local justice as “cruel but fair”, while the alternative peddled by President Hamid Karzai's Kabul government is deemed remote and corrupt.

The big question about the policies that Kilcullen proposes, already implemented by Petraeus in Iraq and now being adopted in Afghanistan, is whether they come too late. The author reckons Iraq has a chance of viability. As for Afghanistan, he shares the view of almost all thoughtful soldiers and diplomats, that western objectives are cursed by divided strategies and command.

Kilcullen perceives dialogue with Iran as indispensable, recognising the legitimacy of its “aspirations to play a regional role”. But surprisingly he makes no mention of the difficulties created by almost unqualified American support for Israeli oppression of the Palestinians. Maybe that is a minefield too deep for even the most enlightened Washington strategy guru.

In the face of future foreign threats, Kilcullen favours low-key military responses, and high-energy civil ones. Some American strategists go further than this. They suggest that the lesson of Afghanistan and Iraq is that it is counter-productive to challenge Al-Qaeda in its own spawning beds, in the midst of cultures alien to the West; that counter-terrorism must henceforward emphasise domestic responses.

Kilcullen's book suffers lapses into American military academic-speak. This is a language as difficult as Pashtun - which, deplorably, scarcely one of the Britons or Americans serving in Afghanistan has bothered to learn. But almost everything the author says makes sense. His work reflects wisdom purchased by eight years of western military and political folly.

The Accidental Guerrilla by David Kilcullen
Hurst £20 pp376

Times Recommends

• The 100 Best Books of the Decade
• The Times picks the 50 best paperbacks of 2009
• The conversation: James Ellroy