On Wednesday, the winner of the Arthur C. Clarke Award for the best science fiction novel of the year will be announced at the opening of the Sci-Fi-London Film Festival.

This annual award was established in 1986 with a grant from Clarke himself. The prize money - rising this year to £2,008 to coincide with the calendar - is still provided by Rocket Publishing, the Clarke family business.

The first presentation ignited a controversy that continues to bubble away over the definition of “science fiction”, when it went to Margaret Atwood for The Handmaid's Tale. It would be difficult to argue that a better novel was published that year, but as Atwood refused to accept that her dystopian vision was SF (which she said was all robots and monsters), and her publisher, Cape, did not want their star author tarred with the ghetto brush, fans felt insulted. The Ragged Astronauts by Bob Shaw, revealed as the “runner-up”, would undoubtedly have been a more popular choice, as the Northern Irish writer was a popular fixture at conventions, and wrote bouncy, entertaining space adventures.

Some thought an award bearing the Clarke name should go to books that concentrated on space travel and predictions about future technology, but as the members of the first judging panel made clear, they meant to interpret science fiction in a wider sense, to “push at the speculative edges of the genre”, as Tom Hunter, administrator for the award, said this year. Sir Arthur never gave an opinion on the matter, or tried to influence the award in any way.

For the next few years the winners were drawn from the genre pool, until, in 1993, there was more controversy when the winner was a mainstream author, Marge Piercy, for Body of Glass. (Full disclosure: one of my novels was shortlisted.)

The book most thought should win was Red Mars by Kim Stanley Robinson, a rich and highly plausible novel about the human settlement of Mars, which Clarke called “the best novel on the colonization of Mars which has ever been written”, and which went on to win the Hugo Award.

The first of a trilogy, Red Mars stands as one of the major science fiction novels of the 1990s, whereas Body of Glass, although well-intentioned, broke no new ground, and has what the critic John Clute called a “second-hand staleness”. Clute, one of the most respected critics in the field, still feels the decision was wrong, and he was one of¡ the judges who chose The Handmaid's Tale. People will always disagree with a selection of “the best”, but since the panel of judges - which changes every year - is composed of writers, critics and fans, unpaid volunteers who share a passion for science fiction, they can hardly be accused of taking their task lightly.

The decisions of judges, who must reread and argue over their selections, only occasionally coincide with the popular vote. In 20 years, four Clarke winners have also won the British Science Fiction Association Award, but that won't happen this year - the BSFA Award for Best Novel has already been won by Ian McDonald's Brasyl, the most glaring omission from this year's shortlist, which is otherwise a very good, and, for only the second time, completely British, selection:

Matthew De Abaitua's The Red Men (Snow Books) is an accomplished, quirky first novel, set in London and the North, about the creation of artificial life, mingling science with the occult. A strong contender.

Stephen Baxter's The H-Bomb Girl (Faber), set in Liverpool at the time of the Cuban Crisis in 1962, combines alternate histories, time travel and nuclear war with teen rebels and the Beatles. Fun, but written for kids.

Sarah Hall's The Carhullan Army (Faber) is a raw, compelling, beautifully written vision of female rebellion against an oppressive near-future society, and has more in common with Orwell's 1984 - or The Handmaid's Tale - than genre SF.

Steven Hall's The Raw Shark Texts (Canongate), a first novel tipped for cult status, is an exhilarating, original excursion into story via meta-fiction, philosophy and intellectual games.

Ken MacLeod's The Execution Channel (Orbit) is a gripping, astonishing techno-thriller that tackles big ideas with style and conviction. This is the fourth time that MacLeod has been up for the award, and he would be a popular choice.

Richard Morgan's Black Man (Gollancz) is yang to Sarah Hall's yin, being a big, action-packed adventure all about masculinity and violence. It is Morgan's second nomination.

Times Recommends

• The Sunday Times 100 best holiday reads
• What will we be reading this summer: the booksellers predict
• Literary festivals: a survivor’s guide