I went to prison a couple of weeks ago. I may be blond and wear short skirts but it was not because I was caught driving while banned down Sunset Boulevard. I went to meet some artists, offenders who attend art classes in Wormwood Scrubs. I went to get a feeling for the people and environment, as I have been helping the Institute of Contemporary Arts and the Koestler Trust put together a show of prison art that opens this week.

The Koestler Trust is a charity founded by the writer Arthur Koestler in 1962 to encourage and promote art and literature made inside prisons and secure hospitals in the UK. Every year it receives thousands of artworks and pieces of writing entered into its award scheme. Winning prisoners receive small cash prizes.

Usually the visual art is piled into an annual exhibition at Kensington Town Hall. This year, though, the trust was approached by Mark Sladen at the ICA, offering not only a high-profile contemporary-art venue but also a contemporary-art viewpoint.

I have always had a love of outsider art – art made by people who have no formal training, who work outside the professional sphere just for the love of making or a need to express. So I did not hesitate when Sladen asked if I would like to help to curate the show, Insider Art, along with Zelda Cheadle, a photography consultant, and the academic Dr Mike Phillips.

At first I felt overwhelmed. Three thousand works in all media covering every surface of the trust’s large three-storey house. My second thought was, wow, some of this is really bad but in such a good way, and some is really good in a really good way. I found it remarkable that such a tiny section of our population could produce such a lot of art, much of it showing real intuitive talent. In his book What Good are the Arts? John Carey came to the conclusion that looking at art won’t make you a better person but making it does seem to have beneficial effects. I like to think that the men and women who made some of the beautifully sensitive and refined artworks found solace and growth in the making of them.

The trust tends to reward traditional skill and there are some wonderful examples in this show, such as View from the Titanic Restaurant, by Anon of Parkhurst, full of elegant detail. We curators, old artworld lags, were bowled over by the expressiveness, inventive mark-making and plain weirdness of works such as Sebastien Wilbur’s Cutting Life and Power Station, by Anon of Rampton, surely a reincarnation of the cult painter Philip Guston.

My concern was how, in taking this art out of its context and framing it with all the paraphernalia of contemporary art, we would sanitise it and rob it of its power. I also wondered if a lot of that power was purely because we know that this art is by offenders. I can project lurid backstories of ill fortune and depression and drugs busts, but I think the majority show natural flair that would stand out even if this was a degree show.

Despite being culled down to a couple of hundred works the show is still very busy and dense, but we found it hard to leave out our many favourites. I did not always share in the enthusiasms of my fellow curators, and some pieces we were obliged to include as they had been given special awards. Inevitably some of the works we chose because they amused or appalled us. Most Disturbing Award must go to Anon of Warren Hill Young Offenders Institution for his two drawings called Ghost and Raped Queen, pencil copies of old master portraits but with what looks like real blood oozing from their orifices.

Were we being patronising? Were we being the artists and treating the art like a found object? I hope not, for spending time with this art was a joyful experience. The tension between subtle exhibits such as The Dapper Woman, by Anon of HMP Wakefield, and Dusk, by Michael Whittaker, and the humorous kitsch of the parent and child cardboard Daleks or the large blue portrait of Madonna makes for a very lively show.

The painting that made me laugh out loud is called The Price of Chips. It shows a huge knife with blood-splattered price tag informing us that this grim weapon was reduced in the sale from £11.99 to £9.99. We actually got to meet the young artist, Aaron French, in his art class on our visit to Wormwood Scrubs and, like most of the inmate artists I met, he seemed a placid and funny guy. When we were chatting I was dying to ask, what have you done to end up here? But I was too polite or too scared to ask.

We walked around one of the classic Victorian wings and it felt like wandering through the school corridors when everyone else is in lessons. I was reminded that it costs more to keep someone in prison than to send them to Eton. A damning statistic but I doubt if old boys from Eton could put together such a fine show.

Insider Art is at the ICA, London SW1 (020-7930 3647), tomorrow until Sept 9

Editor's choice

• U2 at Nou Camp Stadium, Barcelona
• The 50 Best Movie Villains
• The 15 golden rules of theatre etiquette