Naim Attallah is showing me photographs of all the beautiful women – and famous men – that he has met. Their framed faces cover every inch of the four walls of his elegant office. “And here’s the goddess Nigella Lawson. Look, look here’s the lovely Sabrina Guinness . . . this is Lady Cosima Fry. Here’s Koo Stark and Paula Yates and Patsy Kensit and . . .” He smiles like a proud father showing snapshots of his children.

For me these photos are a trip down memory lane. I saw at first hand this world and these faces from the 1980s. It’s a decade that Naim chronicles in his new book of memoirs, Fulfilment and Betrayal: 1975-1995.

In the 1980s Naim Attallah was a legend in his own launch time. He was known as the flamboyant Palestinian publisher – owner of Quartet Books, CEO of the luxury goods firm Asprey, film producer, perfumer, patron of the arts and pretty posh girls. And let’s not forget party-giver and party-goer extraordinaire.

In 1984 I got my first column writing for a magazine he owned called the Literary Review. This gave me access to every publishing party in London. I’d hit four or five a night and if that sounds like hell, believe me I was in heaven. The world of book publishing had been a stuffy and snooty place – and then along came Naim. Out went tweed jackets; in came babes with brains.

Attallah published books and threw parties like nobody else. For a book called Chastity in Focus we got a room full of models in their knickers, stockings and suspender belts – which was far better than sharing canapés with A S Byatt. At other parties his girls would be in rubber. Sloanes rubbed shoulders with punks, highbrows rubbed god-knows-what with nobrows, and everyone had a good time.

Naim’s chat-up line never seemed to fail with women: “Dal-leeng, you’re pretty. Come work for me.” What was the relationship between the middle-aged and married Naim and all these young glamorous girls? Mutual flattery. He made these bright young things feel they were brilliant, even when they were mediocre. “They’re talented, they’re tantalising, they’re the toast of a generation. Who are they? They are the Eighties girls,” went the puff prose of a book of photographs Naim published.

They flattered and flirted with him and made him feel young and fascinating. The amazing thing is that it still goes on today. His new book has dozens of letters from former female employees all testifying to what a joy it was working for him. But not all of his employees remember him so fondly. “He was always going on about women’s bodies, their curves, their breasts. He gave me the creeps,” one girl told me.

In the 1980s I admired Naim Attallah. He reminded me of my dad, Jay Landesman, who had his own small publishing company in Soho, next door to a sex parlour. My dad was the poor man’s Naim – a snappy dresser, brilliant at self-promotion and absolutely sex mad. He didn’t have a bevy of beauties – just one at time. I worked at the family business part-time, but I wished I could work for Naim Attallah. Cocktails with Nigella Lawson at Claridge’s would have been a lot more fun.

Still, I always hoped I’d get the call, until he approached my then wife Julie Burchill and asked for an interview for his forthcoming book about prominent women. He thought Julie would be flattered to be part of such esteemed company as Ger-maine Greer, Tina Brown, Joan Bakewell, Margaret Drabble and Clare Short. She replied, “No, I won’t be in your book. One, because I don’t like Arabs in general. Two, because I don’t like Palestinians in particular. And three, because I particularly don’t like you. Now sling your hook.”

When I suggested to Julie that perhaps a simple “no, thanks” might be a more appropriate response, she called me “a weak, gutless, self-loathing Jew”.

Anyway, I still admired Naim as a publisher. He had real flair and most of all he had guts. He was the patron saint of lost causes – promoting quality writers that nobody wanted to publish and magazines (like the Literary Review) that few wanted to buy. He lost more than £5m on causes he believed in – and has no regrets.

It’s true he did publish a lot of tacky rubbish – books of naked celebrities, a silly book on breasts. But there were great works as well: Dennis Potter’s Pennies from Heaven, Julian Barnes’s first novel, Metroland, and Ryszard Kapuscinski’s The Emperor.

The 1980s passed and suddenly we were all asking: what happened to Attallah? In 1995 he lost his job at Asprey’s. Suddenly the party was over. “I just disappeared,” Naim tells me. “After leaving Asprey I owed the bank £4m. It was a nightmare.”

Suddenly all his rich and powerful friends stopped calling. “When I was riding high and chief executive of a huge conglomerate I was invited to every party,” says Naim, “but as soon as you’re out of it, the invitations don’t come as often.”

He was dragged back into public life when in November 2004 a book called Ghosting by Jennie Erdal was published, claiming that for 15 years she had been doing most of the writing that appeared under his name. This included two novels, numerous articles and even a love letter to Naim’s wife Maria. The great benefactor of literature was revealed as a literary fraud.

Attallah was hurt by Erdal’s revelations, but the thing that bothered him the most was the implication that he couldn’t write without help from someone else. One of his most cherished dreams is to prove to the world that he is a talented man of letters. Ever since then he has been trying to write his way back to credibility. Fulfilment and Betrayal is the third volume of memoirs and the one he believes will prove his talent.

Unfortunately, the book is a landmark in vanity publishing. Written by Naim and published by him, it spends nearly 800 pages chronicling in minute detail every book, party, lunch, business decision, critical review and famous person associated with him for 20 years.

“They said it was vanity publishing when Quartet published my book on women. But that was one of our bestsellers and we made lots of money,” says Naim. “So I laughed all the way to the bank.”

I haven’t the heart to point out the obvious fact that reading interviews with 800 eminent women is different from reading 800 pages on one Naim.

One of the big topics of the book is sex. The back jacket features a naked girl and a Willie Rushton illustration of him with his meat and two veg on display. “Did you see this? My wife thought it was disgusting,” he says. So did my editor.

When it comes to sex, Naim acts like a naughty schoolboy. He tells me how at chic London restaurants the highlight of the meal was his practice of slipping off a girl’s knickers and putting them in his pocket. With relish he asks me, “Did you read in my book about the girl who asked me to f*** her mother?”

When I ask him how he had the superhuman strength to surround himself with so many attractive women and not to give in to temptation, he is stumped. “That’s a good question,” he says, and lapses into silence, before continuing, “Of course I was attracted to women . . . God only knows why I didn’t take the plunge. Maybe it had something to do with family values. But I’ve always had straightforward relationships with people and never been dishonest.”

My, what an admirable man of character. But then one of his former employees puts me straight: “He boasted to me of sleeping with various women. I know of one case where Naim pressed himself on a woman who worked for him. After her marriage she thought Naim would not cross the line, but he was unhappy with that. She told me in tears it was awful and in the end he sent her to Coventry.” It’s tempting to dismiss such talk as petty gossip. But the woman I spoke to actually still admires Naim and has contributed one of the fan letters found in his new book. The woman she talks about still admits to a great affection for Attallah. He is a man of contradictions.

Explore Books

• Book Extracts

• Book Reviews

• Books Group

• Audio Books

Times Recommends

• Superfreakonomics by Steven D Levitt and Stephen J Dubner
• The Times picks the 50 best paperbacks of 2009
• Vincent van Gogh’s letters show the soul of the artist