Clothes have always played a leading part in Linda Grant’s fiction and memoirs, from her first novel, The Cast Iron Shore, in which a character warns, “You are what you wear”. Even Remind Me Who I Am Again, her moving account of her mother’s dementia, starts off with a list of that lady’s gorgeous adornments: “Crocodile shoes and mink stoles, an eternity ring encrusted with diamonds, handbags in burnished patent leather . . .”

Naturally, then, Grant’s relish for a well-stocked wardrobe informs her new novel from the outset. Its title, The Clothes on Their Backs, is a poignant echo of the refugee’s lament, “I came here with only the clothes on my back”, perhaps among the worst deprivations Grant can imagine. Ervin and Berta Kovaks manage to reach London from Budapest before the start of the second world war, when Hungary’s Jews were rounded up, many of them, including Ervin’s father, never to return. The Kovaks settle cautiously into a mansion flat in Benson Court on the Marylebone Road, their lives crepuscular and pleasureless, offering their daughter Vivien only studious isolation and keeping their family history an unspoken secret. “No one bothers us,” is Ervin’s relieved boast, understandable in a man for whom, once, in another country, a knock on the door would have had horrific meaning.

Vivien’s outstanding memory of her enclosed childhood is the visit in 1963, when she was 10, of Ervin’s brother, Sandor, whose existence has never been mentioned. Her uncle presents a vision at odds with her shabby, morose parents. He is wearing “an electric-blue mohair suit, black hand-stitched suede shoes, his wrist flashing with a fancy watch attached to a diamond bracelet”. And he has another fascinating accessory: a sassily dressed black girl.

An enraged Ervin bars his brother from Benson Court, but finds his existence impossible to ignore when, soon afterwards, Sandor is put on trial for profiteering and subsequently sent to prison. The man with the fancy watch is, according to press reports, London’s notorious King of Crime. Grant writes in an acknowledgment that the character of Sandor was inspired by Peter Rachman, the infamous Notting Hill slum landlord, but her fictional crime king is far more sympathetic than his real-life counterpart. His greatest crime is to have lost his moral compass and, as Vivien discovers on slyly investigating him after his release from prison, it’s hardly surprising. Sandor, Budapest boulevardier and pimp, had been marched off to a wartime forced-labour camp, wearing one of his natty suits, which, by the end of his captivity, “no longer resembled clothes, but a kind of fungus excreted by his skin”. As a dubious man of property in London, he may have treated his tenants shamefully, but he himself has experienced much worse.

Vivien, whose own interest in appearances verges on the obses-sional, is drawn to her uncle’s dark glamour and ticklish ambiguity. While her parents glumly chew fish fingers in front of a television quiz show, Sandor takes her to tea at Harrods, to gorge on cakes from the “special tray”, wheedled out of a waitress bewitched by Sandor’s charm.

London is as much a part of the narrative as Grant’s unsettled characters, and, as 1977 approaches, the nightmarish changes in the city are jitteringly moved into the foreground. Vivien, going through a punkish phase, is attacked by skin-heads “because I was a filthy Jewish bitch who loved money they said”. In Benson Court, “fear ran down the walls”, as the National Front logo “as blunt and uncomplicated as a fist” makes its appearance on the streets. Sandor owes his final undoing to a provocative item of clothing, a leather jacket with a swastika on the back, worn by a weirdly gormless boy.

We are what we wear because clothes reveal our personalities but, as Grant makes clear as she guides us through a dizzying ethical maze, they also conceal them. We leave Vivien, now middle-aged and recently widowed for the second time, swinging a bag containing a svelte new dress – a symbol of hope for the future if anything is – and we understand that, in this meticulously textured and complex novel, beneath Grant’s surface dressing, what she is talking about is more than skin deep.

The Clothes on Their Backs by Linda Grant
Virago £17.99 pp293

Times Recommends

• Superfreakonomics by Steven D Levitt and Stephen J Dubner
• The Times picks the 50 best paperbacks of 2009
• Vincent van Gogh’s letters show the soul of the artist