We've all tried making a list of our Desert Island Discs, and if you are like me, you find that your eight favourites change so rapidly that next week's list might be quite different. The Radio 4 programme With Great Pleasure lets you do the same sort of thing with prose and poetry, but again, because it lasts for only 40 minutes, you can't get much in, and the pieces tend to be what are in your mind at the time rather than things you really couldn't do without.

Choosing 40 books, however, is a different matter. That is not as much as 400 or 4,000, to be sure, but it's enough to let you establish a sort of settled personal canon ... or is it? Because it would be quite easy to choose 40 poets, as I very soon realised. Or 40 19th-century novels. Or 40 books about science. Or 40 books about great painters, lavishly illustrated. Damn! I wanted them all! What should be my principle here?

Well, it had to be variety, of course. I also thought I should avoid too many obvious classics. Was there much point in recommending Middlemarch or Hamlet? I thought that people could be trusted to find their way to those without my help. Another constraint was that the books had to be in print, which ruled out any of the 16 novels of the, to my mind, inexplicably forgotten writer Macdonald Harris, an American who died in 1993, and whose The Balloonist, at least, should be available.

Finally, I looked through the list that Sebastian Faulks had picked for his Writer's Table. I enjoyed his list, but there was only one point where we overlapped: Penelope Fitzgerald's The Blue Flower, which would have been one of my first choices otherwise.

The hardest area to select from was poetry. How could I leave out Coleridge? Wordsworth? Keats? Or Donne, Marvell, Herbert? Or - and so on. In a list of 4,000, of course, they would be there automatically. But then they'd be there in everyone's 4,000, and I felt I had to be more decisive; so in went Wallace Stevens, who is the poet that I've been turning to most frequently in recent years, and Elizabeth Bishop, whom I love: it's as simple as that. Then there was Rilke, who for me is the most interesting and important poet of the past century - by which I mean he says the most interesting and important things: but which translation? Luckily, there was a very good new translation of the Duino Elegies by Martyn Crucefix, which (and it was almost the clincher) came with a parallel German text. Good for Enitharmon Press: in it went. The biggest gap as far as poetry is concerned is where the great ballads would be - Sir Patrick Spens, The Wife of Usher's Well, and dozens of others; a good anthology would be the first thing to go in, if the 40 were 4,000, or even 400.

I've said elsewhere that children's books belong in the general conversation about books and not in some separate little nursery. So I applied the same rule here as I did with the rest: leave out the obvious classics. Alice didn't make it, but a less well-known Arthur Ransome than Swallows and Amazons did, and so did a book that still makes me laugh after 50 years, Norman Lindsay's The Magic Pudding. Just outside my list, but pressing hard, is Philippa Pearce's wonderful Tom's Midnight Garden.

One interesting thing was the growing realisation that my list was less about literature than about story, or about something else that wasn't fine writing, but was more important. Some of these books are badly written, but unforgettable: Lovecraft's stories, or David Lindsay's extraordinary A Voyage to Arcturus. Some are here because their subject matter is inexhaustibly intriguing: Richard Dawkins on evolution, William James on The Varieties of Religious Experience, Robert Burton's eccentric and monumental treatise on melancholy. Some are here because they're plain funny: Myles na Gopaleen, Molesworth. One, at least, is beyond classification altogether: Fernando Pessoa's The Book of Disquiet, the insomniac's perfect companion.

Would I make a different list if I did it again later? There would be some changes - Barry Lopez's Arctic Dreams would be among the first to step inside - but there's a core of permanence here too.

The book I look at most, though, isn't here at all, and that's the dictionary. But which dictionary? The great Oxford, of course, but Chambers is the one I keep at my right hand, and then there's Fowler, and Brewer, and Partridge's Dictionary of Slang and ... Choosing 40 reference books would be no trouble at all. But I'll stick with this list for now; some of them are fine fat books, too. They won't wear out in a hurry.

© 2008 Philip Pullman

Philip Pullman's Writer's Table will launch in selected Waterstone's stores on Thursday September 4. Find out more at www.waterstones.com/writerstable

Explore Books

• Book Extracts

• Book Reviews

• Books Group

• Audio Books

Times Recommends

• The 100 Best Books of the Decade
• The Times picks the 50 best paperbacks of 2009
• The conversation: James Ellroy