A more accurate title for Ferdinand Mount's memoir would be Rich and Famous People I Have Known. His mother was Lady Julia Pakenham, daughter of Brigadier-General Lord Longford, and she had a wide circle of friends whose distinction her son is eager to bring to our notice. She was at school, he tells us, with Unity Mitford (Bobo), and at Oxford with the Earl of Arran (Boofy) and Isaiah Berlin (Shaya). She went skiing with the owner of the great Elizabethan mansion Doddington Hall. Her sister Pansy married the painter Henry Lamb, a friend of Augustus John; her sister Violet married the novelist Anthony Powell. The extensive Pakenham family has continued to provide Mount with celebrity relatives over the years, among them David Cameron, son of his cousin Mary. Mount's father, Robin, an amateur steeplechase jockey, though less well-connected, was a close friend of the Hon David Tennant, founder of the Gargoyle Club, where he caroused with sundry art-world notables, among them Dylan Thomas and Philip Toynbee. Mount's parents settled in the Wiltshire village of Chitterne, and belonged to what he calls “a raffish subdivision of the upper class” known as “Hobohemia”, centring on Longleat. At Christmas parties the Marquess of Bath was Santa Claus, and little Mount, in bed with asthma, would be visited by the likes of George Orwell's widow Sonia and film star Robert Newton.

The Mounts were “relatively poor”, though not so poor that either of them needed to get a job. Indeed, their son recalls, almost nobody in their circle worked. His own studious attention to rank and genealogy began at prep school. Asthma often consigned him to the sanatorium, and his regular fellow patients included the son of Lord Willoughby de Broke and the future Lord Onslow. He also had the honour of being Prince Michael of Kent's dormitory prefect. He won a scholarship to Eton, where his teachers included Giles St Aubyn, who was the son of a lord and owned an island off the Welsh coast. At Oxford he joined the Gridiron Club, after being assured that its members were “all public-school men”. He went to films (called “scopes” in Gridiron slang), and enjoyed the lavish lunches given by Lord Rowallan's “enchanting” son Bobby. In summer he bathed in a stretch of the river Test with a friend whose godfather owned it, and who boasted that it was “the most expensive fishing in Europe”. He quickly found that his chosen subject, Politics, Philosophy and Economics, was beyond his grasp, and changed to modern languages. That, too, disappointed him, however, for he formed the impression that most of the French and German literary classics dwelt on sexual passion and had “a rank odour about them like fox urine”.

His Oxford, now half a century in the past, was unreformed. There were no co-educational colleges; state-school students were in a minority; social class and family were more important than brains in gaining entry. It had been like this for years. His father, though never quite able to manage joined-up writing, had attended Magdalen College, and gained a fourth-class degree. Mount looks back on these times as a golden age. There was a familiarity between dons and students, he claims, that has now vanished. In his day, undergraduates could expect to have lunch with Maurice Bowra, dine with Warden Sparrow at All Souls, and be welcomed by Lord David Cecil at his Sunday-morning sherry parties. He seems truly to believe that these (decidedly mixed) blessings were open to all, and that being an Old Etonian at Christ Church with Pakenham connections had nothing to do with it. It is this kind of naivety that makes his appointment as the head of Mrs Thatcher's policy unit, responsible for advising her on such problems as sink estates and failing schools, surprising. But he owed it, he explains, to the Pakenham network and “the oxygen of influence”. Hugh Fraser, who married his cousin Antonia, put in a word and got him into the Conservative party research department.

Prior to that he had drifted, living in Chelsea and doing odd jobs, among them leader writer for the Daily Sketch, which he describes as “a tabloid for housemaids”. In the evening he would put on a dinner jacket and go to debutante dances. It was a period when high society had brought off “the most brilliant reconstruction of its prewar heyday”. Conspicuous expenditure was in vogue, and Mount reports excitedly on its sumptuous triumphs. There were casualties. A friend, Tara Browne, the son of “the tearaway heiress Oonagh Guinness”, and the “Golden Child of the Sixties”, was killed driving his Lotus Elan at 100mph through a red traffic light. Wandering through today's Chelsea with its standardised superstores, Mount looks back wistfully at these high old times.

His gossip-column mode is insulting, since it presupposes that the reader will be as thrilled by titles and celebrities as he is. Only when he sloughs it off does his book come to life. A sad story lies behind the glitter. When he was 15 his mother took him to Italy. She tired of sightseeing more quickly than he did, and one afternoon he found himself saying, jokingly, “The trouble with you is that you have no soul.” A few moments later he realised she was in tears, and he did not know what to do or say. Soon after they returned to England she died of breast cancer. She had never been a beauty, having a squint and a tendency to plumpness, and Mount's father had been unfaithful and taken to drink. He had his own disappointments. His father had died without making provision for him, and his elder brother, Mount's uncle Bill, inherited Wasing, the family house in Berkshire, with its 4,000 acres. When Mount's father drank himself into debt, humiliating interviews with lawyers had to be undergone before some infinitesimal fraction of the family wealth was disbursed to bail him out. Pathetically, he would always scramble to the carriage window on his way to London to get a glimpse of Wasing's pedimented Adam facade as the train sped past.

Mount's admiration for Thatcher seems to stem from her unlikeness to his parents. They were idle and upper class, she loathed sloth and privilege. Tory high-ups despised her “suburban” respectability, but that was a quality Mount had not experienced, and he recognised it as a strength. She became, to a degree, a surrogate mother, and his account of her mannerisms and off-duty moments is by far the most vivid part of his book. But his storytelling skills show up elsewhere. He is unusually attentive to sounds. On the Italian holiday he visits the aesthete Harold Acton, the original of Anthony Blanche in Brideshead Revisited, and brilliantly describes his over-exquisite pronunciation, which seems to epitomise a kind of inner softness. In Ireland, staying at Desmond Guinness's Leixlip Castle, he watches a fencing match between the elderly Sir Oswald Mosley and the Irish fencing champion, and the whole bizarre scene is brought alive by the squeaking of Mosley's surgical boot on the wooden floor. Moments like these are oases in the wilderness of name-dropping.

Cold Cream: My Early Life and Other Mistakes by Ferdinand Mount
Bloomsbury £20 pp370

Times Recommends

• The 100 Best Books of the Decade
• The Times picks the 50 best paperbacks of 2009
• The conversation: James Ellroy