His reputation so precedes him — ex-boyfriend of Tracey Emin, punk poet, art world hellraiser — that it’s odd to find Billy Childish in Chatham, Kent, in a modest semi with a kettle that whistles and enough health food to earn a gold star from Gillian McKeith. Two tiny kittens, as sleek as eels, dart between my legs and leap on to the piano that stands in the kitchen.

Childish, 47, likes “real things made of real materials” and lives surrounded by crafty knick-knacks and pieces of his art. Today he is wearing plus fours and a moustache of Kitchener proportions. He obviously thinks he has the talent to be famous, successful even, but he claims to be happy that he isn’t. He was one of the founders of the Stuckists, the art movement set up as a counterblast to the conceptualist pre-eminence of Emin and Damien Hirst. “I like being on the margins,” he says. “You work better there.” But then his bitterness towards Emin makes you wonder.

Childish gives me proof copies of two new books: one of woodcuts that he’s made, Thoughts of a Hangman; the other, Gun in my Father’s Hand, Selected Lyrics: 1977-2006. Surveying both, and the accomplished, intriguing art on the walls, you do wonder why Childish isn’t a bigger name, disregarding his association with Emin. He writes well: his poems have an emotional charge, while his songs neatly reveal his charming, odd, slightly askew view of the world. Childish is a true original. Perhaps the mainstream couldn’t contain him if he was famous.

In his time, Childish has sung more than 100 songs in almost the same number of bands (“I’m always the one to end bands; every new song should sound like it’s your first”) and proudly reads out the couplets which form The Day I Beat My Father Up: “I punched him clean down the stairs/ The blood was oozing from his golden hair”.

Hang on, this actually happened? “I was about 20,” Childish says in his Kentish twang. “My father was a very absent sort of chap. To me he was just a scary man with a beard. He left home when I was quite young and would come back and clear off again. My mother never divorced him. He bullied her. This particular day he picked an argument. I thought he was capable of killing her or me. He was in a sleeping bag under the table. I asked him to leave and he didn’t move.

He said I was a stupid boy, not a man.

“I dragged him to the stairs, shoved his clothes at him, and told him to get out. He came towards me and I was scared. I punched him in the face a couple of times and he fell down the stairs and knocked his head on the stone step at the bottom. I had planned to kill him on a number of occasions and now I had the opportunity where I could actually finish him off, but luckily I didn’t. He raised his hand up and said that he loved me. I thought it was for dramatic effect, not an actual declaration.”

Childish lost contact with his father for 15 years. They are in touch now, not super-close, but the odd call now and then. “I am angry with him but I have compassion for him,” Childish says. “The reason I can have a relationship with him is that I don’t see him as my father but as an individual. I’m glad I didn’t murder him that day. I couldn’t have lived with myself if I had.”

Are there traces of his father in him? “For the first 20 years of my life I emulated him in his drinking, womanising ways. But he gave me an example of how not to behave.”

He darts around subjects like his crazy cats. From a ten-minute ramble about childhood, I distil that he liked wearing hats and painting pictures. “I loved moustaches. I used to draw myself with one. When I was 14 I was really into war and Van Gogh.”

Then, without pause or segue: “I was sexually abused when I was 9 by a male family friend. We were on holiday. I had to share a bed with him. It happened for several nights, then I refused to go near him. I didn’t tell anyone. It affected me deeply. It made me very ashamed of my body. It made me think I was homosexual. From a young age my family had told me I was ugly, unattractive and retarded. I couldn’t read or write and my mother gave me a list of things I’d never be able to do.”

He sought comfort in art and drew his favourite Thunderbirds characters. “I was into history, the Stone Age, fortifications. When I was 8 I got into Jimi Hendrix.” He started drinking at 9, whisky and Coke from a hip flask. This developed into full-blown alcoholism, which he beat a few years ago. He was an extrovert child too, was bullied and — he claims — prevented from taking his A levels because he was considered too disruptive. After being expelled, Childish trained as a stonemason in Chatham dockyard (“pretty rough”). He was born Stephen Hamper, and got the name Billy Childish from a mate in one of the punk bands he was in when he was a teenager. He has adopted lots of names over the years. “Someone asked Christopher Robin why he had two names and he replied it was in case he lost one of them. I like that.”

The contrast with his academic brother who had gone to the Slade rankled, and eventually Childish got into Central St Martins. Within weeks he was expelled. “I was bright but I’m not very good at suppressing myself. That’s had ramifications through my life. Tracey loved me because I was uncontainable. I won’t obey.” He met Emin at a party. In Strangeland, her autobiographical memoir, Emin wrote about a man with whom she had a relationship who anally raped her. Childish says she is referring to him. “Yes, I think I was cruel to Tracey,” he admits. “But the cruelty she’s written about is fabrication. She was never subjected to anything.” He was seeing someone else during their relationship. “Tracey wanted me to smother her. But I had a stronger connection with this other woman.”