This week, I have been helping to judge the inaugural Jarman Award, to be given to an artist film-maker “who is to our times what Derek Jarman was to his”. That's a tall order. As Isaac Julien's new film Derek, and a retrospective of Jarman's work at the Serpentine (curated by Julien) make clear, the film-maker, who died of Aids in 1994 at the age of 52, was a one-off. In his paintings and films, he mixed high art, sex, flippancy, invective, politics and desire. He did it with barely any money. He courted outrage (which he loved) and his legacy is potent.

It seems an absolute scandal, then, that it has taken 14 years to mount a major retrospective of his films. The director is one of the few original thinkers that British cinema has produced. Actually British cinema had very little to do with him, which is probably why he faded so rapidly into the ether.

Jarman broke all the school rules. He was spanked by Lord Putnam and Mary Whitehouse on a regular basis. He was playing with Super-8 art-house films in the late 1960s and 1970s before anyone had the slightest clue what art-house meant. He was shamelessly fond of highbrow Shakespeare and lowbrow froth. And even more shamelessly queer. Who else could possibly have made Sebastiane (1976), the first homosexual feature film that actually enjoyed itself - in Latin? Certainly not Andy Warhol.

Jarman's biggest films - from the punk rock anthem Jubilee in 1977, to the achingly personal Blue in 1993 - set the gold standards of cinema experiment for an entire generation.

The Establishment cleverly absorbed what Jarman had to offer, and thanked him by trying to marginalise his greatest hits. On August 17, 2002, eight years after Jarman's death, Tilda Swinton, his most famous and glamorous muse, published an epistle entitled Letter to an Angel. What would Dear Derek make of our brave new world of film-making? she ponders.

Her letter has pushed the most extraordinary buttons. Swinton's loyalty to Jarman - and a style of film-making that we underfund at our peril - has sparked a campaign that is growing exponentially. The stunning centrepiece of the Serpentine exhibition is a 12-hour interview with Jarman, conducted by his great friend and producer Colin McCabe. In Derek, Julien has hard-boiled that last will and testament down to a mere 76 minutes. It was shot at Jarman's fisherman's cottage on Dungeness beach, after a death scare in March 1990. It's a story about a wonderful individual and a document about life in England from the 1950s to the 1990s. McCabe has implanted vintage splinters of Jarman's Super-8s, and recruited Swinton to do a voiceover. It's an ingenious piece of chutzpah.

Swinton's thoughts rattle around the soundtrack as she stomps moodily across London Bridge, cursing life, the film universe, and everything in between. She starred in seven of Jarman's best films, including Caravaggio in 1986, Edward II in 1991 and Blue in 1993, and is one of those rare British actresses who can regularly call the shots in Hollywood if needs be. And she is absolutely adored at festivals.

“I wouldn't say that I've become mainstream,” Swinton says. “The significant point is that the major studios have been employing people like Andrew Adamson, Spike Jonze, David Fincher, and Francis Lawrence [top-end mavericks] who have thought of me when putting together their $200 million features. Time was when no one would have known who I was.”

Jarman has always been her guiding light, she says. “The first time I met him [as a young Cambridge graduate in 1985], he opened the door of his flat in Phoenix House with a camera up to his eye, and never stopped shooting.

“Derek made film-makers out of all of us who worked with him. Our work came out of the pre-industrial atmosphere of an art context, not the segregated professionalism of industrial film-making. This is the atmosphere I carry with me everywhere, like an amniotic sack. It is the only way I know to work.

“Roles are never the criterion. The most important question is my dialogue with the film-maker. That's what I'm in it for. If the conversation is compelling, the role will follow. Making a deal with the role alone strikes me as a lonely and redundant business.”

Swinton's public lament for this lost, almost romantic style of independent movies has helped to make this a truly exciting retrospective, and an overdue rereckoning of an unpredictable auteur. The Serpentine Gallery has scraped together long-lost Jarman artworks and delicate Super-8 films, preserved by the producer and cinematographer, James Mackay.

Julien has assembled these films into a “constellation” piece. His own movie, Derek, will be showing on a loop. It is an outstanding documentary watch, for those who appreciate life as much as art. There is a viewing of a rare feather-bed installation (not a Tracey Emin) that has been seen only once before, in Glasgow. And a blizzard of lectures, films and talks all over London. I know one or two conservative souls will be turning in their graves, but frankly this is a wonderful thing, if only because the main figures involved have lived and loved Jarman.

Why did Jarman make such an impression, I ask McCabe. He presents a list.

“1. Sebastiane was not a very good film at all. But the impact of its release was sensational. There were queues around the block. It was the first film of its kind. It was supposed to be a joke. Boys speaking Latin to each other? Derek called it his “gay lib film”. It was the first genuinely ‘out' film which just celebrated being gay.

“2. He was incredibly brave and experimental, and he worked across a range of art forms, which is rare.

“3. Derek was the most charismatic person I ever knew.

“4. He was the first famous person in Britain to admit he was HIV-positive in 1986.”

McCabe says that Jarman was prophetic. “In retrospect, one sees how much he saw. Derek was uncannily aware of the way Thatcherism incubated new Labour; and the fake patriotism of the Falklands war, and boomtown Britain. Films like Edward II and The Last of England were very prescient. He was obsessed by the way his parents' generation dismantled the empire and created a welfare state. He was aware how self-confidence had evaporated and what you get in its place.”

The 22 entries to the award set up in his name are daring and vigorous. The shortlist of four will be announced on Friday, and the winner on April 1. The prize is £20,000 plus a commission for one of the Three Minute Wonders series on More 4. The three runners-up will receive £1,000 each. The entries include Andrew Kötting's absurd and brilliant journey around his dead father (whose image he paints on a balloon) and a lovely film called Magnetic Movie about how sunshine works.

One of my fellow judges, the director Nic Roeg, thought it odd that there was so little sex, given that this was an award set up in Jarman's spirit. I asked him why the kind of work Jarman made was so hard to find nowadays. “It's corporate sponsorship, dear boy. You have to be able to sell a damn film before you can make it.”

As judges, we are looking for a film-maker whose work most enshrines the spirit of Jarman's own work: collaborative, irreverent, not afraid to embody high art and visual daring. And, for the first time in a long time, far from the multiplex, I am excited - this is film as it is meant to be: breaking borders, transgressive, challenging. Jarman, I am sure, would be pretty delighted too.

Derek Jarman Curated by Isaac Julien, opens on Saturday and runs until April 13 (www.serpentinegallery.org, 020-7402 6075). The Jarman season on More4 continues with Sebastiane, tonight, 11.30pm, and Caravaggio, Friday, 11pm

Editor's choice

• The top 10 fan versus band scuffles
• Mamma Mia! The first review
• Times critics' 20 best films for kids