Woody Allen is bemused by his own success. A year ago, his movie-making career appeared to have sputtered out, with Cassandra’s Dream taking less than $1m at the box office in America. Now his latest, Vicky Cristina Barcelona, is being talked up as an Oscar contender and looks as if it could become one of his most successful films. “I never know why people come to my films,” he says. “Or why they don’t.”

It’s the morning after Vicky Cristina Barcelona opened to a rapturous reception at the San Sebastian film festival. Despite a late finish, an early start and turning 73 on December 1, Allen is fresh and feisty, pretty much the same as when we last met, 12 years ago. “I did Match Point and everybody loved it. I did Cassandra’s Dream and people did not come to see it. If you could calculate why, it would be great, but I can’t figure it out. Nor can the studios, whose time is spent trying to work out what the public wants, giving cards to people, having discussion groups — then the directors and the studios change the movies to fit what the audience wants. It is a ridiculous way of making films and it doesn’t work that well.

“In my movies, there’s not a lot at stake, though. It probably costs more to do the focus group than it would be worth. One of my films will cost $15m to make and would really have to be a catastrophe to lose more than $7m.”

With Allen, it has always been hit and miss; he insists he doesn’t know why and that he doesn’t care what the critics say. “I’ve not read a word about myself in maybe 35 years,” he says. “I found out very early on that it didn’t alter my life in any appreciable way if a film of mine had lines around the block. I still went home at night alone and had to order food from the Chinese restaurant. The first time I was not well received, I thought, ‘Oh, God’, but nobody came to my house and assaulted me, and nobody said, ‘You will never work again.’ I have always moved on to the next thing.”

At tonight’s Golden Globe awards in LA, Vicky Cristina Barcelona has been nominated as best film comedy and the performances of Rebecca Hall, Javier Bardem and Penelope Cruz have been shortlisted for prizes. Come the Academy Awards, Allen may be looking at adding to his four Oscars (Annie Hall won best director, best picture and best original screenplay; Hannah and Her Sisters won best original screenplay) and a further 19 nominations. Four of his actors have won five Oscars, with a further 10 nominated.

The film’s reception as a comedy is also somewhat bemusing to him. “I was gonna do a film in Barcelona, and I had an idea about these two young women who get involved with an artist and his crazy ex-wife,” Allen says. “London or Berlin wouldn’t have worked because they’re drier, a little more intellectual. Spain’s got an exotic, Mediterranean feel. So I just wrote it as a straight love story — and it’s gotten many more laughs than I anticipated. I’m always surprised at the amount of laughs my films get. When I try for laughs, it becomes work, and I don’t get them. When I screened Match Point at Cannes, I’d never seen it with an audience before — there were a lot of laughs in it and I was shocked. It’s the same with this picture.”

Allen is being disingenuous. Vicky Cristina Barcelona has been a hit because it works the way Annie Hall or Hannah and Her Sisters worked. It’s romantic, funny and bittersweet, touching universal truths in a light, affectionate way. Unlike its predecessors, it is incredibly sensual and vibrant. Allen always hires good actors, but with Barcelona as a backdrop, they sparkle. You believe that uptight Vicky (Hall) and carefree but aimless Cristina (Scarlett Johansson) are best friends who can fall in love with the same painter, Juan Antonio (Bardem), who in turn is in thrall to Maria Elena, his passionate but crazy former wife (Cruz). Timing helps, too. In a world increasingly beset with financial woes, people want to see films that make them feel good.

Vicky Cristina Barcelona may be life- affirming, but that’s not its creator’s world-view. “I think life is a grim business. You’re speaking to one of the most gloomy pessimists around,” he says, with no trace of a smile. “I find it a meaningless, chaotic, bleak, suffering, terrible, nightmarish tragedy with a few delightful moments, oases of laughter and joy. If there was no mortality and no illness, it would fundamentally be a wonderful experience, with some down moments. But it isn’t that.”

To protect himself from spending too much time pondering “man’s predicament in the universe”, Allen is surrounded by a tight-knit group. His sister Letty Aronson, co-producer Helen Robin, editor Alisa Lepselter and casting director Juliet Taylor have worked with him for years and know how to keep him content.

“The distraction of work is helpful,” he says. “If there was nothing for me to do in the day, I’d be sitting consumed with this kind of depressive gloom over me, whereas if I’m working, I get lost in imaginary problems. If the movie fails, that’s not the worst thing in the world. If you have patients in an institution and you give them needlepoint or things to do, the very act of engaging in some physical therapy with a goal helps them. That’s why I do it. I could afford to retire, but I don’t know what I’d do. I’d walk the streets or sit home or watch television or start to have grim thoughts. This way, I get up and say to Soon-Yi, ‘We gotta get on a plane and go to San Sebastian and then Barcelona. Did you remember to pack this?’ ”

He has been with Soon-Yi Previn, the adopted daughter of his former partner, Mia Farrow, and 34 years his junior, since 1992. It is the longest-lasting relationship in his life; they married in 1997 and have two adopted daughters, Bechet and Manzie. He won’t discuss their relationship, saying only: “The love I feel for people is more important than the art I create. If you have an unhappy love life, it is much more painful than putting out a movie nobody likes. I am very happy right now.”

Despite his nerdish on-screen persona, Allen has always been a sportsman and hates the way age has curtailed his physical activities. “People always think I’m puny. I’m not,” he insists. “When I was a kid, I wanted to become a professional baseball player, I was a medal-winning athlete and I boxed. I don’t play competitive sports like basketball or tennis any more. I get up in the morning, I do the treadmill, and every other day I do weights and stretching. It’s better than just turning into an old man with osteoporosis and walking down the street with a question-mark posture. It’s like clarinet practice. If I go a day without practising, I’m so consumed with guilt that it’s not worth it. I feel completely energetic and youthful in general, but that doesn’t mean anything. This could be my last conversation . . . you never know.”

Explore Film

• Film Reviews

• Trailers

• DVD

• London Film Festival

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010