In 1771 Johann Zoffany started to paint his now famous group portrait of Royal Academicians. It is a delightfully idiosyncratic document. The spectator has only to glance at the array of bewigged and frock-coated individuals represented — the grand Sir Joshua Reynolds with his sword and his silver ear-trumpet; the sturdy Francis Hayman resting up a gouty foot; the exotic Mr Chitqua in his native Cantonese costume; the foppish Richard Cosway striking his selfconscious pose — to suspect that the story of this most august of artistic institutions might be rather more colourful than a reading of Sidney Hutchison’s substantial (but dauntingly specialist) official history might suggest. You can almost hear the hum of the background gossip as you look.

This is the gossip that James Fenton now dishes up. This polymathic former professor of poetry at Oxford and frequent contributor of critical essays to the New York Review of Books was commissioned by the Royal Academy to write a history that would offer “an outsider’s unofficial view”. School of Genius is the slightly eccentric, eminently readable and richly illustrated result.

To outsiders the academy can seem an arcane and outmoded institution. Founded in 1768 under the patronage of George III, it operates like an elite gentleman’s club. Governed by its self-selecting members, it is a bit, said one academician, like a lunatic asylum that is run by the inmates. Its Summer Exhibition may be as much a part of the Season as the Gentleman v Players match at Lord’s, but it is critically scorned by the contemporary establishment. Most high-profile young artists are not elected — or else refuse — to become members. They prefer to leave the old buffers to stew in their own prejudices.

Of course, every now and then some controversial episode of the academy’s continuing soap opera attracts public attention. Scandals blow up like sudden squalls in the pages of the press. We have just witnessed one. Its outcome was the resignation of two secretaries and one president, the jailing of a bursar, the removal of a curator and, to top it all, the remarkable survival of Norman Rosenthal, the colourful exhibitions secretary (and — for almost 30 years — the public face of the academy), who at the height of all the arguments cavorted about at a party dressed up as a newt.

If this sounds a little fanciful, read Fenton’s book. His delightful anecdotal history puts such antics in proportion. But then, what else would you expect of an institution founded expressly for the purpose of furthering the careers of artists, traditionally among the most temperamental, egocentric, opinionated and passionate sector of society?

Fenton disapproves of some of the academy’s more foolish decisions (to sell off its most precious assets, including a Leonardo cartoon, for example, to stave off financial problems). And he exposes its pretensions, not least in the title of his book, School of Genius. If the RA was a club for artistic prodigies, it was only during its earlier history: when Reynolds was president (even though, insulted during a pitched battle over who would fill the post of professor of perspective, he resigned and had to be urged to rejoin by the King); when Turner shone; or Constable was elected, albeit belatedly.

Centuries later, under the catastrophic presidency of the fox-hunting Alfred Munnings, who, in a state of rip-roaring post-prandial inebriation, launched an attack on modern art that was (accidentally on purpose) broadcast live on the BBC, the academy’s reputation plummeted. As Fenton makes plain: “If there had been something within the bastion more strikingly worth defending, the effect of this might not have been so bad. But the artist-presidents who maintained this doomed redoubt were very far from being articulate proponents of any alternative set of values . . . they were like a needle stuck in its groove.”

This clear-sightedness allows Fenton to infiltrate the academy undaunted by opportunistic loyalties or false reverence. The reader, broaching the grand bastion of Burlington House, is taken on a behind-the-scenes tour by a guide who delights in all the idiosyncracies. Here are William Blake’s withering annotations of Reynold’s lofty Discourses. “This Man was Hired to Depress Art,” he ranted. Here are Gainsborough’s neurotic instructions for the hang (when the Academy didn’t comply he retired in a sulk, never to submit again). Here is Richard Redgrave mocking Turner: “What a riddle he was . . . who ever understood his jokes? Yet he chuckled over them himself until others joined him out of fellowship . . .”

Here, alongside such landmark scandals as the Epstein affair (which severed academicians from the avant-garde pretty irreparably), are stories of the trifling squabbles, the flung insults, the petty rivalries and private alliances, the bizarre foibles and battles of irascible old men.

Here are the snippets of gossip (Benjamin West’s mother was banned from places of worship on grounds of fornication, but went into labour at her agitation on hearing a Quaker preacher); the mad theories (Benjamin Haydon became convinced in the course of dissecting a lion that a “Negro was the link between animal and man”); and the quirks (Nollekens smuggled silk stockings into the country inside his hollowed plaster busts).

This entertainingly lively account takes some fascinating tacks. It looks at the role of women, for instance, represented in Zophany’s group portrait only by paintings on the wall, banned from attending life classes on grounds of indecency. It introduces us to the models, from “Smugglerius”, the bronze cast of a flayed criminal cut fresh from the gallows, through a black man almost suffocated while encased in a plaster mould, to the children who sat for Reynolds — “Good God!” exclaims Northcote, “he used to fill his painting room with such malkins, you would have been afraid to come near them.”

By contrast, the antics of the modern-day academicians that have so preoccupied the press over the past year or so start to pale. Maybe this is why Fenton devotes little more than a couple of pages to the history of the past few years.

The notorious Sensation show may have led to several resignations, but so did the Epstein affair. The ostracised James Barry’s dark hints about money disposed of in a “mysterious & secretive manner, in Pensions for themselves” puts the recent financial shenanigans in context. Even the glass Norman Foster staircase that leads up to the new Sackler Wing seems in some way a shadow of Chambers’s notorious Somerset House “stare-case”, its ascent so steep that men craned to peep up women’s skirts.

The current exhibitions secretary, Norman Rosenthal, may once, allegedly, have spat at a journalist. But then Nollekens, believing that to keep clay fresh you should shower it with water sprayed from the mouth, famously did this, without warning, in the presence of the King. And is dressing up as a newt so bizarre when you consider Cosway, who in Zoffany’s group portrait poses, resplendently overdressed, with the tip of his cane resting provocatively upon the pubic mound of a female plaster cast? Fenton doesn’t attempt any grand conclusions. He doesn’t even go so far as to explain the purpose of the academy in our modern-day milieu, or its present preoccupation with celebs (Saffron Aldridge, Anya Hindmarch, Jo Malone and Liz Hurley are expected to appear at the high-profile Royal Academy of Arts summer party in June).

The most important point, Fenton writes, is that “it has recovered a sense of the value of what it possesses”. And perhaps the most important thing it possesses is its history. Long may it live into its anecdotage.

School of Genius: A History of the Royal Academy of Arts by James Fenton is published by the Royal Academy, £35