Until September it looked as if Jay Jopling had perfected a rare balancing act: he was half of Brit Art's Golden Couple, a confidant of some of the most famous people in the country and yet still somehow unknowable. Even then, after the end of his 11-year marriage to the artist Sam Taylor-Wood, the art dealer who sold the Young British Artists to the world still kept carefully below the radar.

Although he is arguably the only figure in the London art world with the ambition, chutzpah and financial pragmatism to rival his biggest earner, Damien Hirst, Jopling has not given a single major interview in eight years. At the opening night of Taylor-Wood's awkwardly scheduled solo show in his flagship gallery in October it was a different recently single man, the couple's friend Guy Ritchie, who attracted all the attention, turning up with Robert Downey Jr and Jude Law and looking thrilled to be rid of Madonna.

So it was something of a surprise when photos were splashed all over the tabloids last weekend of Jopling, 45, cavorting in the Caribbean with the singer Lily Allen, 22 years his junior and the daughter of another of his many famous friends, the actor Keith Allen. Headlines such as “£100m tycoon wins Lily's art” and “Singer snogs gallery boss” are simply not Jopling's style and have focused unwelcome attention on him at just the moment when his decade-long domination of the London art scene is facing a fierce challenge on three fronts: from the global financial crisis, a sudden lack of confidence in the market for contemporary art and uncertainty over his relationship with Hirst.

“The big issue now is not who Jay's next girlfriend is,” says Sir Norman Rosenthal, the former exhibitions secretary of the Royal Academy who put the YBAs on the international map with the brilliant Sensation show in 1997. “It is how all these players, and the art world in general, cope with the new financial world. In a funny way, in contemporary art terms I don't think we've quite come out of the 1990s yet, but I think we are about to. Jay has been the archetypal art dealer-player of the 1990s in London. It's a new challenge for him and for every aspect of the cultural world.”

Rosenthal, like most people who have watched the workaholic, ultra-sociable Jopling at close quarters, believes that he will be one of the ones to emerge with his reputation and influence more or less intact. Two early strategic coups set the course for Jopling's career, he believes. First Jopling backed a hunch, spotting the potential of an extraordinarily confident young artist from Leeds. Jopling met Hirst in a pub in the late 1980s, and then found the money to order the shark for the formaldehyde that became a defining work of Brit Art when it was shown at Sensation years later. “At the time is wasn't that clear that the crucial figure [for Brit Art] was Damien. Jay was one of the first to single him out as a superstar.”

Then in 1993 he persuaded Christie's to let him have rent-free a tiny gallery space on Duke Street, the oldest art-dealing thoroughfare in the world. The original White Cube mounted often shocking shows by the likes of Hirst, Tracey Emin, Sarah Lucas, Gavin Turk, Marc Quinn, Chuck Close and Doris Salcedo at the centre of what was then a very small London art world. Jopling was up and running.

Right from the time he arrived in London in the mid-1980s it was clear that Jopling was not so much a breath of fresh air for the art world as a blast of galeforce iconoclasm.

Born in 1963, he came from a privileged background at odds with most of the artists he would make famous.

While Emin endured a famously torrid upbringing in Margate and Taylor-Wood struggled as a “free school dinners kid” from a broken home, Jopling was the son of Michael (now Lord) Jopling, a Yorkshire landowner who was Chief Whip and Minister of Agriculture in Margaret Thatcher's Government. He was sent to boarding school in Scarborough at the age of 7 and then on to Eton, where he persuaded Bridget Riley to create a cover for a school magazine. A contemporary remembers him as “gregarious and offbeat in a mild sense. He was pretty good at everything and one of those people that everyone remembers being there. He was a personality at school.”

At 14 he bought his first work of art from the Anthony d'Offay Gallery, a limited edition Gilbert and George book. Many years later he would become their dealer too.

He studied art history at Edinburgh University and in his final year succeeded with his most brazen stunt yet. Flushed with the spirit of Band Aid, he persuaded major artists including Jean-Michel Basquiat, Keith Haring and Anthony Caro to donate their work to an auction for the Save the Children Fund, raising more than $250,000.

Not long after that he moved to London and started dealing art from the front room of his small terrace house in Brixton. Maureen Paley, one of the first gallery owners to colonise the East End, remembers visiting the twentysomething would-be art impresario in the house. “It was beautifully, beautifully appointed. He had really exquisite art around, from the Judd in the living room to the Coca-Cola that he offered you from the fridge in a perfect old-fashioned cola bottle. He had a sense of how to do things and that's something you either have or you don't. He stood out.”

In 2000 Jopling opened the White Cube, a much larger gallery in Hoxton, at the centre of the newly fashionable East End. In 2006 White Cube Mason's Yard, another new gallery, costing £12 million and on the site of an electricity substation, took his empire back west to St James's again.

Artists love him. Marc Quinn was Jopling's first artist. They started working together in 1988. "He understands art, people and business. It's his ability to understand all three levels that makes him a great dealer. I would picture him in movement, he has this energy about him and he will always do whatever it takes. Wtih the blood head [Self, the 1991 work that made Quinn's name] I remember driving through London with him at 90 mph to get it from the place where we'd frozen it to the gallery and in Basel [for the art fair] before he'd made it we didn't have anywhere to stay so we found an empty house with the door open and slept there. Jay 's very creative. He will find a way to make things work in difficult times."

Emin, who has been with Jopling for 15 years said: "We are the same age and we've kind of grown up together. Gallerists have always had this austere distant thing but he isn't like that. He's your friend and he's really good fun."

He can be cutting, “a master of the chilling putdown”, as one regular at his parties put it, but friends and rivals alike agree that Jopling's deadly charm and “dandyish charisma”, allied to a formidable nose for the right new artist, have been key in his rise.

Paley thinks that it was his energy and his larger-than-life personality that set him apart. “He was very dashing, from the English dandy tradition. His initial charisma attracted people to him and he was able to pursue his activities with enormous energy. Those activities changed the art scene. There was a lot less flying in the face of tradition before him. A few things were starting to introduce the new and the young but he was able to focus that and figurehead it more than anyone else.”

Charles Dupplin, an art expert at the specialist insurer Hiscox, emphasises Jopling's extraordinary focus. “He's a bit like Charles Saatchi in that art is his life, his religion and his business all in one.”

When Jopling first emerged from the pack of dealers working with the YBAs, it was his willingness to go out all night with his hell-raising artists that stuck out. “Jay was always one of them,” one close friend remembers. “He loved parties, he would be at the Groucho all the time and I think that was part of his success.”

As a father of two he has calmed down now. Unusually tall, he is always immaculately turned out in a snappy suit and distinctive Harry Palmer glasses, a peripheral, watchful presence at White Cube functions. At night though, there are still glimpses of the old hedonism. “You have to see him at a big opening,” says one observer of the art world. “He's a huge flirt but he can pull off bad behaviour because he makes everyone feel really special.”

With Taylor-Wood he lived at the centre of a celebrity-spangled social whirl. Their spectacular art-stuffed Georgian house in Marylebone hosted part of Kate Moss's 30th birthday party celebrations. Daniel Craig, Elton John, David Walliams and the Pet Shop Boys all attended the christening of their second child, Jessie.

Friends say that the split last year has been “very civilised” but will not talk about what is (or isn't) going on with Allen.

The darker clouds on the horizon for Jopling now are financial. Last year the boom in contemporary art that he had surfed and done much to drive since the early 1990s came to an abrupt halt. Auction results reached a plateau and then nosedived. The last hurrah came at Sotheby's on September 15, the very day that the collapse of Lehman Brothers seemed to usher in a new Dark Age for the City.

That night Jopling watched Hirst sell £111 million worth of freshly minted spot paintings, cabinets, dead butterfly collages and pickled animals direct to collectors at auction. Jopling, like Larry Gagosian, the American he aspires to succeed as the world's pre-eminent contemporary art dealer, was cut out of the process altogether. White Cube was reduced to chasing work by its own artist to shore up the value of its Hirst holdings, bidding on 20 of the 56 lots and securing at least four.

It was an electrifying occasion, the art world's wealthiest collectors fiddling while Rome burned, but it also marked the high-water mark of Hirst's bankability.

This is not good news for Jopling. In September The Art Newspaper reported that White Cube was sitting on a “Hirst mountain” of more than 200 unsold works. The gallery denied the report. In addition it is still not clear whether Jopling was one of the investors who, it is claimed, bought into For the Love of God, Hirst's notorious £50million diamond skull, in 2007. Some estimates suggest that he may own as much as a third of it.

However, nobody in the art world will bet against him riding out the hard times ahead. According to Dupplin, White Cube is relatively cheap to run and Jopling has the nous to stay on top.

“He may have to go through some pretty painful years right now but a very, very skilled dealer like Jay will be there when the art market returns with gusto.”

Rosenthal agrees. “He's had an extraordinary run of wonderful exhibitions and been an enriching figure in the London cultural world. My own feeling is that there will be victims, [of the recession in the art world] but the big players will survive after a moment of reflection.”

Editor's choice

• U2 at Nou Camp Stadium, Barcelona
• The 50 Best Movie Villains
• The 15 golden rules of theatre etiquette