The British take pride in themselves as a martial race. We also think we are rather clever at the Great Game of intelligence. The Admiralty’s Room 40 cracked the German navy’s codes in the first world war. Bletchley Park contributed decisively to victory in the second world war. Whole libraries are filled with tales of the derring-do of British agents abroad.

But how good are the British at protecting the nation and its secrets from foreign enemies? Since 1909, when the Secret Service Bureau was established by two men operating from one room in Victoria, this task has been the responsibility of MI5, as it was christened in 1916. To mark the centenary, the service took the unprecedented step of inviting Christopher Andrew, Cambridge historian and doyen of intelligence chroniclers, to write an authorised history. The outcome is weighty, measured and compelling.

The first 300 pages, which take us to 1945, cover familiar ground. From the books of Nigel West and others, we know the flamboyant personalities who ran the service in peacetime. They were overwhelmingly successful in countering German ­intelligence efforts in Britain, and during the second world war ran the brilliant Double X operation, which used “turned” Nazi agents to feed a plethora of false information to Berlin.

Most of the book, however, addresses the post-war experience. The cold-war years were dominated by the fact that British intelligence and diplomacy were deeply penetrated by the ”Cambridge Five”. The worst aspect of their treachery, Andrew vividly shows, is that even after they were exposed (Donald Maclean and Guy Burgess fled east in 1951, Kim Philby followed in 1963, Anthony Blunt and John Cairncross confessed the following year) MI5 and the Secret Intelligence Service (SIS) remained racked by fears of other “moles” in their ranks. Roger Hollis, MI5’s director-general, told maverick officer Peter Wright that he exaggerated the powers of the KGB: “They’re not 10ft tall, you know, Peter!” This merely fed Wright’s fantasies that Hollis himself was a Soviet agent.

Andrew demolishes the allegations of “Spycatcher” Wright and other conspiracy theorists, and I am sure he is correct. But for years, British intelligence relationships with the Americans were poisoned by fears about traitors. Jim Callaghan, as foreign secretary, was appalled that Washington might get to hear — as of course it did — that Hollis and his deputy were suspect. The author says that Wright “arguably did as much damage to the Service as Blunt’s treachery”. One problem was that US “Venona” decrypts of 1940-48 Soviet signal traffic, which exposed many American spies, mentioned 24 British traitors by their codenames. Only 12 were unmasked. The mystery of the identities of the remainder haunted Whitehall for the rest of the cold war.

The book concedes that Soviet intelligence-gathering was fiendishly effective. But Andrew says that Moscow never used its information as well as it might, because its analytical capacity was so poor. The KGB could tell the politburo only what it wanted to hear. Even such devoted servants of Stalin as Philby were disgusted by the insensitivity of the Soviets’ agent-handling. The Russians certainly made fools of the British (and, to a lesser degree, American) intelligence services. The betrayal of atomic secrets hastened the creation of the Soviet bomb. But it is doubtful whether thereafter the West’s security lapses much changed the course of history. Their main consequence was to cripple the self-confidence of the SIS and M15.

Some of the most sensational passages in Andrew’s work address the role of the Security Service in successive governments’ struggles with militant trade unionists and left-wing groups between the 1950s and 1980s. The Labour party was tormented by the presence of overt and covert communists in its ranks. But when opposition leader Hugh Gaitskell wished to consult M15 without reference to the Tory government, so ignorant was he of its workings that he had to get George Brown to ask journalist Chapman Pincher for a phone number.

MI5 provided vital bugging evidence of communist ballot-rigging in the electricians’ trade union in the late 1950s, which empowered the ETU’s right-wingers to cleanse the stable. The service later monitored miners’ leaders Arthur Scargill and Mick McGahey — the latter routinely consulting the Communist party about industrial strategy. They tracked the dealings of the Transport and General Workers’ Union’s Jack Jones with the Communist Party of Great Britain — and his receipt of cash from foreign communist coffers.

But Andrew convincingly shows that the service rejected as improper many requests by both Tory and Labour governments for surveillance of political foes. Again and again, MI5 told Margaret Thatcher that alleged subversives were merely radicals exercising their ­democratic rights. Joan Ruddock of the Campaign for Nuclear Disarmament (CND) had some dealings with a Russian diplomat, but the service reported that she was probably unaware he was a KGB agent. Andrew concedes that the opening of an MI5 file on Bruce Kent, CND’s chairman, as a possible anarchist, “now appears distinctly dubious”. But most of the detail given here about the service’s involvement in surveillance of the left suggests its officers had a more balanced view about sinister threats than did ministers.

It is an alarming reflection on the workings of democracy that Harold Wilson continued as 1974-76 prime minister despite compelling evidence that he was going mad. The tale of his ravings about an MI5 plot to undo him makes amazing reading. “During his last few months in office, Wilson appears rarely to have said anything in the lavatory without first turning on all the taps and gesturing at imaginary bugs in the ceiling.” Wilson’s trafficking with communist regimes while in opposition had caused MI5 to open a file to him under the pseudonym of Worthington. Given the number of Labour MPS who professed enthusiasm for the Soviet Union at the time, this seems a reasonable precaution for the service to have taken.

The account of M15’s Northern Ireland activities has much more to say about organisational issues than operational ones, for understandable security reasons. I am sceptical of Andrew’s assertions about close co-operation between the service and the RUC, whose Protestant officers frequently indicated that their agenda and that of London were by no means identical. As the tale enters modern times, focus shifts dramatically. From its inception, M15 was mostly a counterespionage body. In 1974, 4.5% of its effort was committed to Ulster, 3% to counterterrorism, 52% to chasing spies, 28% to anti-subversion. Today, just 3.5% of its budget goes on counter-espionage. Its principal function, taken over from Special Branch in 1992, is as lead arm in fighting terrorism.

The intelligence services play a far more important role than the armed forces in protecting Britain from its most dangerous enemies. Andrew suggests that they bring to the task a healthy scepticism. At a recent staff Christmas revue, President Bush’s “War on Terror” was mocked as the “War on Terry”.

With this book, the author has done a formidably good job for both the service and the public interest. He does not flinch from acknowledging the mavericks and outright lunatics who find their way into MI5. But most of its officers are bright, sensible, dedicated people, performing a vital role. Much is left unsaid about modern political and operational issues. Andrew might have reflected more on the relationship between the service, ministers and the public. There are also moments when he makes the case for MI5 too enthusiastically. But part of the justification for commissioning this work was to explain and demythologise our spooks. I regret the gaps, but find it hard to disbelieve much he asserts or denies. Conspiracists may be disappointed by the benignity of some of his conclusions, but his narrative offers a feast for students of intelligence and politics.

The Defence of the Realm by Christopher Andrew
Allen Lane £30 pp976

Times Recommends

• The 100 Best Books of the Decade
• The Times picks the 50 best paperbacks of 2009
• The conversation: James Ellroy