Has Hollywood’s golden generation of young auteurs lost its sparkle – and its bottle? In her 2005 book Rebels on the Backlot, Sharon Waxman wrote about the generation of film-makers who came of age in the 1990s and reshaped American cinema – David Fincher, Quentin Tarantino, David O Russell, Paul Thomas Anderson, Steven Soderbergh and Spike Jonze. Her thesis was that in films such as Se7en, Pulp Fiction and Magnolia, they “created a new cinematic language, recast audience expectations, and surprised us – and one another”. These films, Waxman wrote, “played with structure, wreaked havoc with traditional narrative form, fiddled with the film stock, and ushered in the whiplash editing style true to a generation of videogame children”.

The key year for the new generation was 1999, when, as the web magazine Salon put it, “the indie wave crashed onto the beach of American culture with tremendous force”. Magnolia, Three Kings, Being John Malkovich and Fight Club were released that year, as were American Beauty, Election, The Matrix and The Virgin Suicides.

Nearly a decade later, however, it’s clear that most of the directors Waxman was extolling have lost their way and have not produced anything like the important work done by the previous great generation of American auteurs in the 1970s: Francis Ford Coppola, Martin Scorsese, Hal Ashby and Robert Altman, directors to whom they were, Waxman said, “nothing if not self-conscious heirs”.

Their loss of connection to previously adoring audiences has been glaringly obvious in recent weeks, with the release of long-awaited films by Fincher and Tarantino. Fincher’s Zodiac, with Jake Gyllenhaal, Mark Ruffalo and Robert Downey Jr, tells the true story of the hunt for the so-called Zodiac serial killer, who terrorised San Francisco and the Bay Area in the 1960s and 1970s. (It will be the closing-night film at Cannes, and opens in the UK next week.) Although the film was touted as the most personal work from Fincher since Fight Club, and is his first movie since 2002’s Panic Room, it has taken just $32m so far, nowhere near enough to recoup its $75m-plus budget. Although Zodiac is likely to fare better in Europe, even Fincher admitted it would have a problem reaching an audience expecting a David Fincher serial-killer movie to offer at least a smidgen of the excitement of Se7en.

“It’s just people talking,” he said, “and it’s hard to make an audience realise they have to be paying attention.” Fincher seems to have fallen victim to the hubris and obsessional indulgences that afflict directors who enjoy too much adulation. He refused to listen to studio executives who wanted him to make the film more accessible, and infuriated the actors – Downey said he wanted to “garrotte” him – with as many as 100 takes for some shots, as he fretted about the tiniest details. He ended up filming for 115 days and shooting enough footage for two films. Such pettifogging criticisms don’t bother Fincher. “I think perfectionism has gotten a bad rap,” he said.

Tarantino’s latest, Grindhouse, a double-bill homage to 1970s exploitation movies, has also been a disappointment since it opened a month ago in America, taking just $20m. The producers, Harvey and Bob Weinstein, will have to eat a substantial loss. Again, hubris seems to be responsible for Tarantino’s almost complete alienation from the Gen X audience that he was once so instinctively in tune with. Tarantino and Robert Rodriguez, director of the other half of the bill, were supposed to make two 60-minute movies, costing a total of $40m, to be released last December. But with nobody apparently able to control the directors, the films ballooned out of control, ending up at 85 minutes each and, the trade paper Variety estimated, costing almost $100m.

It didn’t help, according to Variety, that Rodriguez fell in love with his leading lady, Rose McGowan, during the shoot. He has denied that the romance caused the break-up of his 16-year marriage and closed down the production for a month. The Weinsteins tried desperately to convince Tarantino and Rodriguez to let them release the films separately, but the directors refused, believing only they knew what the audience wanted. The films will be released separately in the UK later this year.

The more these directors have been indulged, the worse their behaviour has grown. David O Russell hasn’t released a film since the disastrous, incomprehensible I.Huckabees (2004), which made less than $13m in America. But he has been the subject of a viral video, shot on the set of the movie, that has probably been watched on YouTube by more people than saw the film. The clip shows Russell launching into an astonishingly vicious tirade at Lily Tomlin, screaming unbelievable obscenities at her, rushing out of the room, storming back in, kicking over a rubbish bin and furiously sweeping papers off the desk where an aghast Tomlin is sitting.

Leaked by someone on the film, the video has cemented Russell’s reputation for being close to insane on his sets. He has a propensity for grabbing his stars’ genitalia when he is excited, and even got into a fist fight with George Clooney on the set of Three Kings, when the actor felt he was being abusive to crew members. “Why don’t you just worry about your f***ed-up act?” Russell screamed at Clooney when he intervened. “You’re being a d***. You want to hit me? You want to hit me? Come on, pussy, hit me.”

Of the other “rebel” directors profiled by Waxman, both Anderson and Jonze have had trouble with their creative mojo. Anderson’s most recent movie was the disappointing Punch-Drunk Love, in 2002, which made just $18m in the USA. His next film, There Will Be Blood, about the turn-of-the-century oil business, won’t be released until the end of the year. That’s a five-year gap for a film-maker who should be at full throttle.

Jonze’s most recent film, Adaptation, was released in 2002. His adaptation of Maurice Sendak’s children’s book Where the Wild Things Are won’t appear until next year. That’s a six-year gap, for a director who is only 37. Beyond the lack of productivity and the box-office failures, there is another thing that distinguishes these directors from the auteurs of the 1970s whom they once imagined they might emulate. With few exceptions, they aren’t making films about the great issues of the day. Where is this generation’s Apocalypse Now, Coming Home or The Deer Hunter? One of the reasons Fincher’s Zodiac feels so unimportant is that it has nothing at all to say about America today. Tarantino and Rodriguez’s Grindhouse films comment on nothing more than their own hermetically sealed world of movies.

Waxman believes there is a reason they are so out of touch with the audience and with the issues that should concern us all, and them. “The self-indulgent American culture that shaped these film-makers has left them ill equipped to take on the weightier questions facing society in the new millennium,” she says. “Perhaps Tarantino, child of the video culture, feels at a loss when faced with the war in Iraq and global terrorism.”

Iraq? Who directed that?

Explore Film

• Film Reviews

• Trailers

• DVD

• London Film Festival

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010