It begins, my unexpected journey into the darkness that threatens to engulf Neil Diamond, with pillow talk. I am among the 20,000 fans who have filled the O Greenwich for a joyful communion that reaches its climax with them singing him Sweet Caroline. Being press, I sit grandly at the front, in a section otherwise requisitioned by a group known as the Diamondheads, fans so hardcore that they have attended each of his concerts in his first European tour for four years.

For most of them this is just another night with Neil, although from the way they are swaying and clapping and subvocalising, you wouldn’t guess it. For one, however, a woman with a massive crush on the man, it is a very special day. After 30 years of devotion, she has finally spoken to the singer in the shiny black shirt now singing his heart out before us. She bumped into him on the Thames Clipper, the ferry that stops at the dome. Their encounter is now a bit of a blur, but she does remember he was carrying a pillow.

The next day, in a suite in the Dorchester, I ask him about the pillow, one of a pair, it turns out, bought from a British hotel. Since his songs are often about deracination (in I Am, I Said, the New York native complains of being lost between two shores) I ask if a private pillow is a comforter, a reminder of his own bed. He considers the point’s psychological dimension. “Mainly,” he answers politely, “I want a pillow I can rely on.” I have, of course, missed a clue. The 67-year-old, dressed now in sweatshirt, glasses and a baseball cap that make him look nearer his age, has asked to sit at a table not on the sofa. “That will be easier.” As I will discover, while his songs record psychic pain, they do not even touch upon the physical agonies he endures daily.

On Sunday, Diamond will abandon his musical comfort zone of middle-aged devotees for the Glastonbury Festival. He has been asking people which of his songs this unknown crowd may know. He admits to being nervous, but not to the charge that he is in Somerset for the credibility. After all, although his monstrous shirts – more cheese than cheese-cloth – denied him it in the Eighties, credibility was returned with interest by the critical and commercial success of his 2005 acoustic album, 12 Songs.

“I just want to delight the audience,” he says in a voice spread with gravel from the previous night’s triumph. “I don’t know if I can do that.”

Frankly, Jay-Z has more to worry about than Diamond. Everyone knows Diamond’s oeuvre, and some are man enough to admit to loving it. But until now they will not have guessed that he suffers from severe back pain, nor that much of his latest album, Home Before Dark, is informed by two years of anguish suffered by his girlfriend. While Diamond’s career enjoys a blissful renaissance – in America Home Before Dark has knocked Madonna’s Hard Candy from the top of the Billboard chart – his private life has been hellish.

I had already assumed that the dark title was a synonym for death. He confirms that. “The dark is death, the end. The dark is something to be feared, to be avoided, to be protected from.” In an interview he recently referred to the process of composing the new songs as the “jagged rock” he clung to during the illness of a close friend. I ask how this friend is. He decides to be candid.

“This was Rae Farley. The lady I lived with. She had back surgery and it wasn’t going well. She was in extreme pain for a year and the surgery did not really work. If anything, it made it worse. And I never left her side. I was within 20ft of her for the entire year that I took writing this album. So I had to overcome my anxiety for her situation and be there to help her and yet write this album.”

In the end he was in such distress that he completed the album in his recording studio. “But I was by her side for a year.

I considered it a test of my character, of my ability to work, of my ability to help her. But I needed to write. It was easily the most difficult album I have had to write because of this.”

Farley, a 35-year-old Australian whom he met a dozen years ago at a concert in Brisbane where she was working on a promotions stand, now runs his merchandising operation. A back injury damaged a disc in her vertebra and she spent a year in pain before electing to have surgery. “She walks but she is in constant pain and it has changed her life.”

He is anxious to return Los Angeles next week before embarking on a 37-date North American tour over the summer. He appears to accept that he can offer her only limited support. “I am doing the best I can but she is not . . . she feels this is something she has to deal with herself, which is unfortunate because I want to be there. I want to help her. So it has been a very, very difficult time, but I am an optimist and I think she will make it and come out whole, and get her life back.”

Several of the tracks on Home Before Dark suggest a fear of losing what he calls the “one more bite of the apple” that his relationship with Farley has provided. He is not, contrary to his 1966 anthem Solitary Man, a solitary man, although the term may apply to his music. Its sentimentality long ago led Diamond to be rejected by the critical elite and he has yet to be inducted to the Rock and Roll Hall of Fame. But his life, which now includes five grandchildren, is actually rather crowded. “I have only just recently come to realise I am not a solitary man,” he says. “I need people. I love having people around.”

After marrying in 1962 his childhood sweetheart, Jaye Posner, and having two children with her, in 1969 he wed a television executive, Marcia Murphy. Their marriage lasted 25 years during which they brought up two more children. It ended in 1996 with one of the largest and by all accounts most amicable, divorce settlements. Yet, he confesses that he is still haunted by its collapse.

“We had some amazing years, some of the best years of my life, but after 25 years it just petered out. You take each other for granted and the relationship disappeared into the ether and the love and companionship and the attention I need were not there any more.” Does he still beat himself up about it? “I do. I have a lot of guilt about mistakes I have made in the past in my relationships. I have been married twice and divorced twice and both of these women were rather extraordinary women and so it was my loss really.” And his fault? “Probably. Probably my fault. The music and my work was overwhelmingly my first choice, and then came my kids, who have forgiven me and love me because I keep apologising to them.”

Rae represented a third chance. “Another chance to do it right. To do it again and to make it right and to have the opportunity to have a loving companion at this point in my life again. Which I need. I cannot survive without it.” Her ill health, as well as jeopardising this chance, is a cruel reminder of the back surgery he himself underwent in 1979.

After doctors discovered a benign tumour on his spinal column that was slowly paralysing him, a neurosurgeon immediately operated but the procedure proved so lengthy that he “chickened out” and called in another “virtuoso” to complete it. Diamond did not know if he would walk again but slowly recuperated, graduating from wheelchair to walker, to cane. Offered the starring role in the 1980 movie The Jazz Singer he flew to Britain to sign the contract.

“I threw away my cane in LA and I never picked it up again. But the residual has been pain because of the healing process. I have dealt with that for 30 years.” I ask if he is hurting as we talk. “Not at this moment, no, but it has limited my ability to be physical. I can’t walk fast or my back will seize up. It takes the forms of spasms. I have a trainer and she will be working with me today and before every show. I am not in pain when I am standing, only when I am sitting and I get up, or I am walking quickly. But when I am fatigued there is the most extraordinary pain.”

Only the discovery of new drugs a few years ago, he says, has allowed him to continue his career.

And yet, as he performed, I detected not so much as a hint of arthritis. “When I am on stage I am 21. My body could be hurting minutes before but when I go on it is gone.” His trainer once explained about endorphins and serotonins. “And, maybe that is part of it but I think it is the passion and love for what I do.”

It is perhaps glib to warn Diamond that Glastonbury is not the most comfortable venue in the world. I caution instead that he should bring not just his pillow but a pair of wellingtons. “Everybody tells me that. But it won’t rain. I have a history of bringing my southern Californian weather with me. It will be a beautiful day. Check it out.”

So Diamond’s journey into the dark and back again resumes, a harrowing trek accompanied, at least, by a hell of a hummable soundtrack.

Neil Diamond plays Wembley Stadium tonight and the Pyramid Stage, Glastonbury on Sunday. Home Before Dark is out on Columbia Records

Explore Music

• Festivals

• CD Reviews

• Live Reviews

• Classical

Editor's choice

• U2 at Nou Camp Stadium, Barcelona
• The 50 Best Movie Villains
• The 15 golden rules of theatre etiquette