Scott Walker is, depending on how you want to look at it, modern music’s greatest underachiever or its greatest overachiever. Underachiever because, given the young Walker’s rich baritone voice and cool good looks, the man who sang The Sun Ain’t Gonna Shine Anymore was clearly destined to be his generation’s Sinatra — the one singer who could link the style of the old crooners with the sound of the new pop generation. He was obviously going to be the biggest star in the world. And he very much isn’t.

Overachiever because a man whose musical career began in what was essentially a boyband has grown to become the critically acclaimed musician’s critically acclaimed musician. The people who the rest of us think are on the cutting edge can only peer into the distant horizon to watch Walker’s receding boot-heels as he takes music further and further into unexplored territories.

Many of music’s brightest and cleverest can be found praising Walker’s work in Scott Walker: 30 Century Man, a new documentary on the singer, which opens in cinemas this month. David Bowie, Jarvis Cocker, Brian Eno, Damon Albarn and various members of Radiohead are interviewed. Their comments aren’t always insightful — “What do I know about Scott Walker?” asks Bowie. “What does anybody know about Scott Walker?” — but they are intriguing. Walker’s music has clearly captivated them, even if they struggle to explain why.

The fact is that Walker’s reputation rests not just on his music but on his determination over the past two decades to keep making it, pure and unadulterated by commercial considerations, musical fashions and market forces. In a music world where even the most credible artists make compromises, Walker’s refusal to deviate from his artistic vision is the object of admiration and no little envy. Many people talk the talk, but Walker walks the walk.

It wasn’t always that way. Walker burst onto the scene as part of what we would now term a manufactured pop band. None of the Walker Brothers was called Walker (Scott’s real name is Engel), and they weren’t brothers. The three young Americans took Britain by storm with hits including Make It Easy on Yourself and The Sun Ain’t Gonna Shine Anymore. They weren’t as big back home, but that didn’t bother the Anglophile Scott, who was as fascinated by European art-house films as by Duane Eddy and Link Wray.

Just like the Beatles, they were besieged by screaming teenage girls. Walker’s reaction to that was the first indication that he wasn’t your run-of-the-mill pop star. In 30 Century Man, Walker tells of one night when, after a gig, a girl took him back to her flat and he had a life-changing experience. She played him an album by Jacques Brel. Walker fell in love with the work of the Belgian songwriter and determined to introduce his dark cabaret songs to an English-speaking audience. Rarely has the phrase “musical differences” been more appropriate than in the case of the Walker Brothers’ split. Walker released four numbered solo albums heavy with Brel compositions and his own songs. Initially, they sold well, partly because of Walker’s earlier hits and partly because of his continuing profile. But the disparity between what Walker was trying to achieve and what a mainstream audience wanted from him — evident when he sang Brel’s grim My Death on the prime-time BBC light-entertainment vehicle The Billy Cotton Band Show — finally proved too much. Scott 4 failed to sell, and Walker was forced to face commercial reality.

He made a series of albums on which he caved in to record-company pressure to provide more accessible material, and as a result sank into a period of self-disgust and heavy drinking — at least, we assume that is what happened, as he also became increasingly reclusive — before reemerging with a series of widely spaced but extraordinarily powerful albums (Climate of Hunter in 1984, Tilt in 1995, The Drift in 2006) that have established his current reputation as a pioneering and uncompromising artist.

Walker’s habit of disappearing — professionally and personally — for whole decades at a time means that the idea of making a documentary about him seems courageous, if not foolish. But Stephen Kijak, director of 30 Century Man, says he is obsessed by Walker: “I’ve been a big fan since I first heard his work on a compilation in 1990. The songwriting, the voice — I was hooked. Anyway, easy isn’t interesting. Making a documentary like this is like wild-life photography — the sitting, the waiting.”

Given Walker’s reclusive reputation, you would expect his management to have rejected Kijak’s initial approach. It didn’t. It just didn’t get back to him at all. “Six months later, I tried again,” Kijak recalls, “and they were surprisingly open to the idea.” So surprisingly open that it took only four years for Kijak to start filming. To be fair, there may not have been much to film before that, as Walker was on one of his long hiatuses. “I was becoming very frustrated,” Kijak says. “His management kept saying, ‘We’ll let you know when there’s something visual going on’, and I was thinking, ‘Hey, I’m the director.’ But then they told me to come to the studio, and there was a man punching a side of pork, and I thought, okay, they were right.”

That man was Walker’s favoured percussionist, Alasdair Malloy, trying to recreate a sound that Walker wanted on Clara, a track on The Drift that is about Mussolini and his mistress. Walker eschews the comparatively easy world of synths and samplers in favour of creating his own extraordinary sounds. In 30 Century Man, we also see technicians building a large wooden box in the studio so that Malloy can hit it with a breeze block.

The image of Walker as an exclusively serious, reclusive artist takes a bit of a battering in the film, which shows a man quite capable of having a laugh in the studio and fully aware of the world beyond his artistic vision; but then, I’d already seen Walker humanised when I interviewed him earlier in his manager’s house in west London. Walker stumbled into the room, his progress hampered by his manager’s terrier, Dougal, which was wrapped round his right leg. Dougal’s intentions towards Walker were roughly similar to those of his female fans back in the Walker Brothers days — and I don’t mean that he wanted to play some Jacques Brel songs. After that, an unduly reverent approach was impossible.

“Everybody who works with me has a sense of humour,” Walker says. “They have to — otherwise we’d all go crazy. I place a lot of demands on them as musicians; it’s difficult work, so I can’t have them all brooding about the place the rest of the time.” He talks of “removing the emotion” from his work. “So much emotion in music is just bullshit. When I sing, I’m trying to be someone singing without any particular emotion or personality. It’s supposed to be neutral.”

This is perhaps the key to Walker’s work now. For the last three albums, his music has borne little relationship to rock or pop, or indeed to any other genre. He’s been out on his own for a while; but The Drift — great slabs of intense noise followed by equally powerful silences, vocals intoned almost tunelessly — takes us even further from any known, comforting reference points. And perhaps it’s this striving for a lack of emotion that so completely separates Walker’s music from everybody else’s.

Walker can’t throw any further light on it. He doesn’t know where his music comes from. He just knows that it takes a long time coming. His first long silence he explains by saying that he was “haunted by my bad faith”, the period when he caved in to commercial pressures, and a jokey mention of “imbibing a lot”. After that, the long gaps between albums are explained partly by the difficulty in getting record companies to finance such radio-unfriendly music, and partly by Walker’s writing methods. He says months can pass between the writing of one line of a lyric and the next. “I sit and I wait,” he says with a shrug. “It takes a lot of silence before the right line will come.” When he has tried to work faster, the results haven’t been as good. “I have to wait until something comes that I don’t expect, something that surprises me but that I know is right,” he says. “Or maybe I’m just really slow.” It’s clearly hard work; but, as Kijak mentioned, easy isn’t interesting.

30 Century Man opens on April 27 at Curzon Soho; Odeon Panton Street; Watershed, Bristol; Showroom, Sheffield

Explore Music

• Proms

• Festivals

• CD Reviews

• Live Reviews

Editor's choice

• The top 10 fan versus band scuffles
• Mamma Mia! The first review
• Times critics' 20 best films for kids