John Milton was accused by William Blake of being, in Paradise Lost, of the Devil’s party without knowing it. This is not a charge you could level against Jake Arnott. Sympathy for the Devil is his speciality.

In Arnott’s spectacular debut novel, The Long Firm, a gangland sociopath with more than a touch of the Krays about him was the most likeable figure in the book. In the follow-up, He Kills Coppers, Arnott took us to within an inch of forgiving a ringer for the notorious police killer Harry Roberts.

Now, in his fifth novel, The Devil’s Paintbrush, we are persuaded to rethink the reputation of the occultist Aleister Crowley, a figure so repulsive that even his mother called him The Beast. Readers may even consider that Crowley does a favour to the book’s other protagonist, Sir Hector Macdonald — a military hero of the British Empire now practically forgotten — by pushing him into a spiral of drug-fuelled debauchery during a chance (and mainly fictional) encounter in Paris in 1903.

“I think being damned is quite an important step one can take,” Arnott says calmly over nothing more hallucinogenic than coffee in a bar in North London. “You then realise you do not have to be good any more. It certainly destroys Hector, the need to be good, as it destroys so many people. The necessity to be good can take away a lot of natural things about people, particularly their sexuality.”

Arnott, whose relationships have mainly been with men, surprised himself a few years back by falling in love with the lesbian novelist Stephanie Theobald. Although I gather that at 47 he has not entirely retired from the bad boys’ club.

Does he not regard himself as a good person? “No, I don’t. It would nice occasionally to be able to do a good thing and I don’t necessarily think of myself as a bad person, but I think the problem with embodying virtue or being forced to embody virtue is that it is a terrible burden and it can actually lead to very bad things.”

Written in two distinct styles — the adventure-yarn voice of John Buchan and the crazy aestheticism of an English Joris-Karl Huysmans — The Devil’s Paintbrush does not adjudicate on whether Hector Macdonald’s military career furthered an evil or virtuous Empire. It is clear, however, that Macdonald, like Gordon of Khartoum, Lawrence of Arabia, Baden-Powell and (more than likely) Kitchener, was a homosexual at a time when the Empire was in angry and possibly cynical denial about the variety of its citizens’ sexual preferences.

“I think,” Arnott says, “that there was an instinctive feeling that this repression was a way of powering expansion. Crowley makes this comparison early on in the book with steam power, which the Victorians exploited. Steam power is all about repression.”

In addition, the book seems to argue that when in the 20th century lids started flying off, what was repressed found expression not in the harmless diabolism of Crowley but in an orgy of mechanised violence, whose emblem was the Devil’s paintbrush itself, the machinegun that turned the century’s battlefields red.

The prophecy of automated extermination is given in the book to Macdonald, a Presbyterian crofter’s son who distinguished himself in Afghanistan, Sudan and against the Boers, but returned to England disgraced after allegations of homosexuality in Ceylon. It is on his way back to face a court martial that Macdonald runs into Crowley (which in real life he did, although only for lunch). History may record Macdonald as a war hero, but in the novel it is Crowley, at terms with his bisexuality, who is more like the hero. During their night together, Macdonald is taken to a point where he acknowledges his suppressed make-up. Arnott sympathises with both his characters but confesses to liking Crowley, whom he had assumed was “a rather ridiculous, flamboyant charlatan”, much more than he thought. “I see him as this strange prefiguring of the modern age.”

By virtue of being set a hundred years ago, The Devil’s Paintbrush definitively refutes the caricature that Arnott writes modern “gangster lit”. The Long Firm had the mixed fortune of surfacing at about the same time as Guy Ritchie’s Lock, Stock and Two Smoking Barrels.

The success of The Long Firm — ten years ago but fresh in many minds from a 2004 television adaptation — came when he was 38 after a career that was nothing if not chequered. Brought up in middle-class Buckinghamshire as the third of four children, he spotted a vacancy for a bad sheep in the family. At 16, very unhappy at a grammar school he found unintellectual, he left for London, rejecting on the way down his mother’s Roman Catholicism and the work ethic of his father, a management consultant.

The struggle to work out his sexuality meant, he jokes, that he got his midlife crises in early. An early job was as a hospital mortuary technician. This, alongside the Aids crisis of the early Eighties and a debilitating bout of hepatitis B he suffered between 1981 and 1982, may account for his works’ intense preoccupation with death.

After a decade spent in squats fighting a half-hearted war against Thatcherism, he accepted an acting job with a theatre in Leeds. The gig lasted six months, but he stayed in Leeds to work for seven years as a social services care assistant. After penning his unpublishable “squat novel”, he won a six-figure book deal for The Long Firm. The nearest equivalents to a talent content to hide for decades under a bush are probably Ricky Gervais and Susan Boyle.

At the time, papers hinted that the saturnine good looks of this former model and film extra won him an advance beyond his worth. Not so, but life as a literary novelist is precarious and his subsequent novels have not, despite good notices and a cover endorsement from David Bowie, sold as well as the first.

The reader of the new novel, based on the lives of two largely discredited real men, will wonder if Arnott has really bought Crowley’s vision of enlightenment through drugs, occultism and sex. Taking the alleged liberators in order, I ask if he shares Crowley’s belief that drugs can open doors of perception. “I think so. I think they can. My own experience of psychedelic drugs, to use that strange term — acid and magic mushrooms — is that you do sometimes suddenly shoot off and your consciousness seems to go somewhere else.”

But does it go anywhere useful? “That is a very good question. I suspect it doesn’t in terms of a rational connection with our surroundings but in terms of narrative it is tremendous.” Had he taken a little trip or two to refresh his memory? “I didn’t need to. It was fresh in my mind.”

What about mysticism? Did his research lead him to think there is anything to it? “The occult makes the kind of sense that real life does not make. It is like The Holy Blood and the Holy Grail and The Protocols of Zion. These things do make perfect sense. Real life doesn’t. But the thing about religion and belief is that in some way we do have to regulate them. You cannot completely do without them and, because there is an element of mental illness linked to all of these things, people do get carried away if left on their own. The problem is denying them all.”

And what of Crowley’s notion that “sex magic” was the key to elevated spiritual knowledge? “I think this is one of the great disappointments of the 20th century. Sex still has this tremendous hold on us but one of the things about the 20th century right up into the Sixties was this feeling that any moment now there will be this incredible explosion and revolution and we will suddenly be in this idyllic world. And it didn’t happen. We realise now that it is not that big a deal.”

Arnott is a persuasive writer, so agile at crawling into the minds of others, and to hear him giggle camply over the theories his characters take so seriously is slightly disorientating. When I question the importance of a heterosexual experience to Crowley near the end of the book, it does not seem to occur to Arnott that I may be hinting at his own conversion to heterosexuality. So I ask him if meeting Theobald four years ago made him feel that he had at last reached the safe shore of monogamy.

“Quite possibly,” he laughs, “but in terms of sexuality, if tomorrow our relationship came to an end I cannot imagine having a relationship with another woman. I think it has been a very particular relationship and I know Stephanie feels the same way.”

It does sound like love? “I think that is the word for it. It’s a fantastic thing. It is a hard thing to find and it is certainly not something I thought would happen again after having been in quite a long relationship before?”

With a man? “Yes.” Had he had a long-term girlfriend before? “Not really. I have had girlfriends, but not for this long, no.”

Has it changed his perspective? “I think the thing is I have never had a relationship with a writer before. That was the quite terrifying thing for me. I think that has had a huge effect.”

I search for a smile, devilish or otherwise, in case he is joking. I don’t think he is. Arnott will laugh about most things. His writing is not one of them.

The Devil’s Paintbrush is published on May 28 at £15. Buy the book

Times Recommends

• The 100 Best Books of the Decade
• The Times picks the 50 best paperbacks of 2009
• The conversation: James Ellroy