The winner of Britain’s biggest arts prize – the Gulbenkian – is announced on Thursday evening. Worth £100,000, it is awarded to the country’s best museum or gallery. It doesn’t attract as many headlines as, say, the Booker Prize or the Orange: it’s only about musty old museums, after all.

Think again. I’ve been a Gulbenkian judge this year and last (two judges of the seven-strong team carry on from one year to the next) and can safely say that museums are anything but uncontroversial. You’d think I was joking if I told you we’d had judges storming off down corridors after drawn-out battles over the way museums label their objects or the captions under paintings on gallery walls: but actually, I’m not stretching the truth all that much.

So what’s the controversy? Well, a lot of it centres on what you might call the dumbing down of museums and art galleries. The question at the heart of the issue is this: is it OK to give visitors pointers as to what they might think and feel about a picture or an exhibit – or should they simply be presented with objects and paintings and left to make up their own minds about what they feel about them?

Time and again as we deliberated around our table at Gulbenkian HQ this was the issue to which we returned. Some of the judges felt patronised – ridiculed, even – by oversimplified labels: at one of the museums on our shortlist, the Kelvingrove Museum and Gallery in Glasgow, we were asked to speculate on whether a Cézanne still life might have been painted because it was a rainy day and the artist couldn’t get out; at the Weston Park Museum in Sheffield, we were invited to play a sound-track of café chat and music to help us appreciate a portrait of a French waiter.

Banal, said some of my fellow judges. But is it really? Both the Kelvingrove and Weston Park are extremely well-visited, much loved institutions (the Kelvingrove has had three million visitors through its doors since it reopened last year after a £27.9 million refurbishment, and Weston Park had an astonishing 55,000 visitors in the first fortnight alone after its reopening last October. In the year before it closed, it had 80,000 visitors; in the first six months since it reopened, it has had 250,000).

Also, the Gulbenkian judging process invites public comments, and I have to say that few of the comments I saw mentioned feeling patronised.

The bottom line here is: what is a prize such as the Gulbenkian actually looking for? In my view, we’re seeking out a museum that pulls you in and whets your appetite for something that perhaps you’ve maybe never been remotely interested in before (the amount I know about textile history could be written on my knicker label, for example: but I found myself being wowed by ancient fabric samples at the tiny Warner Textile Archive in Braintree, Essex, which was on our longlist).

Alas, not all the longlisted entries – this year or last – came up to scratch: at least one turned out to be more mouth than trousers. Last year one of the longlisted entries turned out to be a boy-toy dream, with oodles of expensive computer equipment that did extremely clever stuff but told you next to nothing about anything (although it would probably impress the kind of bloke who gets excited by a trip to Dixon’s on a Saturday afternoon).

The commonsense truth is that, if you know little about a subject, you need some basic information: if you’re an aficionado, you need less. But the aficionados can’t expect to hold back appreciation of their art by denying the rest of us mere mortals a bit of a peep into their world: that’s elitism, and the arts world could do with a lot less of it.

From a longlist of ten, we whittled entries down to a shortlist of four. Two of the four – Kelvingrove and Weston Park – are municipal, multidisciplinary collections with everything from artistic masterpieces to stuffed animals to reconstructed houses and shops to computer games. The other two – Kew Palace in southwest London and Pallant House Gallery in Chichester – are refined collections spotlighting specifics: modern British art in the case of Pallant House, and the private life of George III and his family in the case of Kew.

Specifics v generalities. An in-depth insight into one thing up against all sorts of everything, all-human-and-animal-life-is-here. The debate was hot. The stakes were high. The decision may well surprise a lot of people.

The other Gulbenkian judges are: novelist and broadcaster Francine Stock who chairs the panel, museums advisor Tristram Besterman, Richard Calvocoressi, director of the Scottish National Gallery of Modern Art, Dr Mark Miodownik, head of the Materials Research Project at King’s College, London, and TV presenters Dan Snow and Mohini Sule

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010