Lucinda Williams is living her career in reverse. Most songwriters are prolific in their youth and sluggish as the years roll by. Not Williams, whose new album, West, is her third this century — while in her previous fortysomething years, she managed only four. At 54, she has Southern manners but a rock’n’roll past, a penchant for destructive relationships and a drink too far. She does lust, loss and sheer, unadulterated passion better than any songwriter alive. Her record-company boss once told me of his conundrum: nobody wishes bad things to happen to their artist, but only when Williams is miserable does she write truly great songs.

Yet times and circumstances change, and now she’s in love. Her life, once a constant, shifting battle, has a structure. She is fragile and skinny under the rock-chick wardrobe; the tattoo of a warped guitar neck on her arm contrasts with her distressed hair and glasses. She doesn’t handle success well, going on about how the photographers distracted her at her recent Shepherds Bush Empire show (although it was probably more to do with knowing that Bruce Springsteen was coming on for the encore). Accolades make her uneasy. In 2002, Time magazine declared her America’s best songwriter. “They had to pick someone that year,” she insists.

She’s a worrier, a perfectionist. If a song isn’t right, she’ll stop mid-performance and start again. Until six years ago, she would run all her songs past her father, the poet Miller Williams. He was her mentor, her personal creative-writing class and, though she no longer feels the need to show him her work, she still idolises him. Not that it was all perfect, growing up an academic brat, moving from one Southern university town to another, with stints in Santiago, Chile, and Mexico City. She watched her parents’ marriage disintegrate while trying to reconcile traditional family ritual with a liberal political bent. Her grandfather was a Methodist minister, a conscientious objector in the first world war and active in the Southern Tenant Farmers’ Union. Her godfather, George Haley, is the brother of the black author Alex “Roots” Haley. Williams waskicked out of high school for refusing to recite the pledge of allegiance; the family hired a civil-rights lawyer to get her back in.

Her father always wanted her to have a profession to fall back on, but she wasn’t having that. She quit college after a year — the truth was, she was lost to music long before. Aged 12, she wanted to be Joan Baez or Mary Travers. Then one of her father’s students lent her Highway 61 Revisited. “I started playing guitar at the height of the folk boom,” she says. “The 1960s just exploded around me. My dad was a big country-music fan, and jazz and blues. I knew all the great roots-rock bands, Cream, the Doors. I spent all my time listening to records and playing my guitar. I set these really high standards for myself early on.”

It was a long haul. At 19, she moved to Austin, working tables, playing in bars and recording a couple of albums for the Folkways label. In the mid1980s, she headed for Los Angeles and the A&Rsoft-shoe shuffle. She recounts how some record companies liked her songs, some her voice; others dissected or shredded her songwriting and told her she was out of step musically. Those criticisms still rankle. “I blazed on, started getting a little attention, but nobody knew how to market me,” she says. “The alternative-country thing hadn’t started yet.”

The turning point came when she signed for Rough Trade in 1988. Two of her songs, Changed the Locks and Passionate Kisses, were covered by Tom Petty and Mary Chapin Carpenter respectively, and there she was, a Grammy-winner with a record deal and money in the bank. From success came financial ruin. “I never had money before, so I went crazy,” she says. “I had all these credit cards, so I would buy things, go out with my friends and pick up the tab every time. I bought a house. Today, I can barely pay my taxes. I rent a modest house in North Hollywood for $3,500 per month, not expensive by LA standards, and I drive a Chevy Silver-ado pickup I bought in 2000.”

It took her six years to make what many consider her genre-bridging masterpiece, Car Wheels on a Gravel Road. She agonised over lyrics for days, going without sleep, pushing herself to the limit. Critics loved Car Wheels; it sold respectably; she toured with Dylan; she picked up another Grammy and a reputation for being difficult (which she denies). Her next two albums came, for her, thick and fast: Essence (2001), whose title track drips with lust, and World Without Tears (2003), where Those Three Days reeks with severed passion, loneliness and rage.

West is altogether calmer. Williams has not lost her edge, but there is an almost wistful tone: “I miss you... plus I no longer wish to rip your heart out.” Two of the songs, Mama You Sweet and Fancy Funeral, are about her mother, who died three years ago: “Losing a parent made me feel vulnerable, aware of my own mortality.” Then there’s Come On, Williams’s You’re So Vain, a brilliant, raucous put-down that is fast becoming a women’s anthem. “I wrote it as a takeoff on cock rock — those guys who wear tight pants that show everything. Unfortunately, when you get with them after the show, most of them are so f***ed up, they can’t perform.” She gives a naughty-girl giggle. “It is based on somebody, but I’m not going to say who.”

She once announced that she could not write in a happy, stable relationship. “That’s not true any more,” she says. “When I was younger, I used to think I had to suffer for my songs.” The reason she has mellowed is called Tom, and he works in the business. “I’ve always been a late bloomer, and the same can be said in the world of love,” she says softly. “I’ve only now met the right person for me. He’s my soul mate, my best friend. In the past, it was almost as if I had to be by myself and be independent, or be with someone and lose part of myself. Now I don’t have to make a choice. Though if I had to, I would choose my art. I have to be able to write or I’ll die.”

West is out now on Lost Highway

Explore Music

• CD Reviews

• Live Reviews

Editor's choice

• Here comes summer: our cultural must-sees
• Friends reunited: Sex and the City
• New country for old men