Julia Blackburn's memoir opens with a painting in which a mother, father and daughter are defined by their lack of connection. A tall, odd man watches as a blank-faced young woman walks out of frame; behind them both, twisting away from an empty birdcage, a girl opens her mouth in silent scream. This is the “tangled fairy-tale triangle” of Blackburn's gripping story.

Blackburn's father, Thomas, was unstable: a poet addicted to barbiturates, alcohol and whatever pills he could put his hands on (even contraceptives if they were lying around), he grew increasingly enraged and violent before walking out on his wife and child. Still more threatening to Blackburn was her mother, Rosalie de Meric, a sexually predatory painter endlessly looking for “Mr Right” among a stream of lodgers recruited to fill the space left by her absent husband. Rosalie had not a maternal bone in her body.

Surrounded by threat and blame, Julia is tossed on the waves of her parents' troubles, emotionally and psychologically abandoned. As a lone witness, bouts of screaming are the only control she can exert, and useless sessions with a Freudian shrink are her reward. Tension is the air she breathes and chaotic dysfunction the imprisoning norm. Propositioned by the lodgers, used as a shield and finally cast in the role of sexual rival by Rosalie, she emerges as an awkward young adult with little sense of self or self-determination. Almost inevitably, she winds up as the mistress of her mother's long-term lover; seeking only safety, she fulfils all that Rosalie had ever expected of her.

What sets Blackburn's memoir apart from the usual run of hard-luck stories is her extraordinary ability to sit on the edges of her own drama, to notice the texture, cadence and scent of these lives and to capture the experience with a painterly precision. Words are the threads that hold her life together: her father's poems and those he taught her by heart as he watched her bathe in the evenings; the sting and lash of her mother's uncontrollable tongue; the bits and pieces of “wisdom” Julia scratches onto her bedside cupboard as she grows up; and the diaries and letters that her mother keeps, stuffed into a tight black briefcase.

As a teenager, Julia, too, retreats into an obsessional recording of her days, journals that now give her access to the detail of the past. Along with the prolific black-and-white photographs included in the narrative, the effect is one of voyeurism, as if we were turning the pages of a family notebook, dissecting the faces and the texts for what they promise to reveal, for the honesty or dissembling, the forthright or inscrutable.

Blackburn, a fine novelist and biographer, has an eye for the significance of tiny actions, for the flash of a red flannel nightie, the glimpse of a gold tooth at breakfast, or the hazy view through the bubbled glass on the bathroom door. And as if to reinforce the edgy instability of her surroundings her tale often swings from despair to hilarity: her father sets out to gas himself but, finding a chicken in the oven, eats it all up instead; conversation lurches from William Blake to the friability of cream crackers; she wears an innocent blue felt hat to school, but her blazer pocket conceals a packet of Woodbines.

The pace and clarity of Blackburn's writing all constantly throw into relief the really troubling undertow of desolation in her memoir. Hovering behind everything there is a hunger - for drink, sex, drugs, artistic expression, freedom, love or stability, according to the characters involved - so that it comes as a shock to realise that real food is almost absent from its pages, though booze is ever-present. A sense of displacement stalks the reader: despite learning the colour of the lino, we never quite know where this home is. It is simply somewhere on the edge of things, somewhere from which to escape.

Although this all sounds rather bleak, the book is leavened from the start by glimpses of the ultimate resolution of Blackburn's relationship with Rosalie. Each chapter ends with an extract from the author's notes or from faxes sent to Herman, one of the lodgers with whom she had an early affair and who - much, much later - she will marry. Written during the last month of Rosalie's life when, diagnosed with leukaemia, she comes to live with Blackburn, these chart the simultaneous decline of an ageing woman and the gentle rapprochement between mother and daughter. Embraced by her daughter's kindness, determined to make a valiant death, Rosalie mellows. Feeling loved without ambivalence for the first time, Blackburn sees the cold hatred she has felt for her mother begin to dissolve.

The Three of Us is an unnerving book about manipulation and loss - “the muddle and sadness and fight of it all” - and about the complicated burdens families inflict on one another down through the generations. As a literary memoir of a lost childhood, it is remarkable as much for its candour as its craftsmanship.

The Three of Us: A Memoir by Julia Blackburn
Cape £16.99 pp313

Times Recommends

• The Sunday Times 100 best holiday reads
• What will we be reading this summer: the booksellers predict
• Literary festivals: a survivor’s guide