It was an irresistible invitation: to make an epic journey across Russia from Murmansk to Vladivostok for a total of 18 weeks, enough time to delve beneath the surface and discover some of the realities of life in a resurgent nation that for me – in Churchill’s aphorism – was still “a riddle wrapped in a mystery inside an enigma”. Perhaps, I told myself, I would be able to reveal the enigma, unwrap the mystery and solve the riddle. It was a journalist’s dream.

The invitation had come from one of television’s grandees, George Carey, who had managed to sell the idea of a five-part documentary series called Russia – A Journey with Jonathan Dimbleby to the controller of BBC2. He then sold it to me. But as soon as he had done so, I got cold feet, feeling suddenly overwhelmed by the sheer scale of the task to which I had committed myself.

Half in panic, I told myself I was taking a terrible risk with my life that was not so much physical as professional. To make the Russian programmes I would have to give up the security of a stimulating perch at the edge of the political arena as the presenter of ITV’s weekly flagship political programme, Jonathan Dimbleby. And for what, I asked myself: the insecurity of an unknown venture in an alien land without a compass?

To ward off the demons of insecurity, I told myself that I would become a 21st-century version of the Victorian traveller. Baedeker in one hand, journal in my knapsack, I would be open to every experience, shying away from nothing, however eccentric or bizarre it might seem.

I did not foresee in 2006 when I set out on the first leg of my journey that relations between Russia and the West would deteriorate so sharply and rapidly. Subsequently, I witnessed Putinism at close quarters, discovering and confronting the profoundly disquieting attitudes of most, though by no means all, Russians to the values and principles that a western liberal holds dear.

I was shocked by the accelerating corruption of the political process within Russia: notably the absence of free and fair elections, the supine torpor of the parliamentary body, the Duma, the muzzling of the media, the intimidation of the judiciary and a profound contempt for human rights.

But all that formed part of my “exterior” journey. In parallel, I gradually came to realise that I had also embarked on an “interior” journey and that it would have been wrong to conceal this. Throughout the two years I have spent on this project my perspective on Russia has been affected, if not shaped, by an emotional volatility that troubled me especially when I was away from home.

I found that, day after day, I was turning to my notebook to record my sharply fluctuating moods. After rereading those notes and recognising the extent to which these emotions (which would swing rapidly from delight to despair) had coloured my outlook, I decided that – except in bad faith – I could not pretend that it had been otherwise. But I can only make sense of these passages for the reader by outlining the background to what has been a roller-coaster period in my personal life.

A little over three years before I embarked on my Russian venture, I met an opera singer called Susan Chilcott who was the leading lyric soprano of her generation. In May 2003 we started to have an affair.

Some months earlier, after treatment for breast cancer, she had been given the all-clear. But very soon after our relationship began she discovered a secondary lump in her breast, and two days later her oncologist told her that the cancer had spread to her liver. No further treatment was possible; she had only a short time to live.

Although Sue protested that I should not turn my life upside down for her, I felt that I had no choice but to be by her side for the final months of her life. In so doing, I had chosen to leave my wife Bel, with whom I had shared 35 years of marriage and to whom I had always believed myself bound as a partner for life. I still do not adequately understand the intensity of passion and pity that animated my decision; only that I felt I had to follow my heart and what seemed to be my duty.

For the next three months I lived with Sue and her four-year-old son. They were precious days of intense joy mingled with deep sorrow until, on September 4, 2003, Sue died in my arms.

It was a tragedy for those who loved music and had heard Sue sing or seen her on stage. I found myself broken by a grief that was more dreadful than I had ever imagined such pain could be. For day after day I could barely bring myself to get out of bed in the morning. Nor did I care whether I lived or died. I could not rest, I could not sleep, I could not think.

With the benefit of hindsight, I feel now that only my wonderful family and a small handful of close friends saved me from going out of my mind. When I went back to work, colleagues at the BBC and ITV were astonishingly sensitive and forbearing.

I began to hope that I could return to our family home. But although my adult children, encouraged by Bel, were more loving and understanding than I could ever have hoped, I was in no condition to repair the damage I had inflicted on our marriage.

Bel understandably decided that she could take no more, and, a few months after Sue’s death, moved out of our farmhouse to start a new life. Now I felt doubly bereft and bewildered. The very foundations of my life seemed to have collapsed. I did not know who I was or where I was going, and I could see no way out of the long, dark tunnel in which I now found myself.

And then, in the spring of 2004, I met Jessica. Gradually and cautiously, over the following months we formed a relationship as she helped to nurture me back to a better state of health. But it took far longer than I had expected and I was still frequently stricken by waves of grief that welled up unexpectedly, drowning out every other sensation and not leaving me until I was utterly drained and exhausted.

This lasted, gradually becoming less frequent but no less intense, for months, a year, and then two years. Even in January 2006, when I had to make the choice between Sunday politics and the Russian project, I was in such a state of inner turmoil that my doctor warned that I would descend into a serious depression unless I put myself on a course of drugs. As it happened, I resisted this advice and that wave passed.

If the Russian journey was, in professional terms, an irresistible challenge, it became, in personal terms, a very simple decision. Jessica was at work during the week; I was working at weekends. It was clear that if we were to have a life together, I would have to recover my weekends for both of us to share. For this reason alone, the Russian invitation arrived at just the right moment. But it took a long time for me to restore the mental equilibrium that I had once taken for granted.

My journey through Russia played what, in retrospect, I believe to have been a crucial part of my recovery – a parallel journey.

The journey south from Murmansk to Kem on the edge of the White Sea was a 15-hour trundle by train. We passed through an unrelenting forest of birch trees and conifers, pressed up against the railway line and obscuring whatever they might have concealed: more forest, I presumed.

A little way down the carriage an attractive woman, who looked to be in her mid-forties, sat quietly opposite her daughter, their modest picnic neatly unfolded on the drop-down table between their bunks. Rather hesitantly, I went to interrupt them.

Sayukin, the mother, elegant but careworn in a black dress that was clearly her Sunday best, introduced her daughter. Karin was a pretty girl with an elfin face and olive eyes, who was anxious to practise her English but was enchantingly modest about her prowess. She had just been interviewed for a place at Murmansk University, where she wanted to study English, French, Spanish and two other, yet to be determined, foreign languages. She had top grades from her school, and she told me that she loved Shakespeare, particularly King Lear. Her dream was to qualify as an interpreter because she wanted to travel and see the world.

The problem, her mother intervened to explain, was that the family had very little money – too little, at any rate, to buy her way into one of the better courses in Russia. In fact, they were not at all sure they would be able to afford a place at any university.

Buy? I had assumed that universities were free and that a place was entirely dependent on merit. Not so, Karin explained, not in the new Russia. There were, of course, free places, but where there was strong demand, you had to buy your way to the front of the queue. Of course it was illegal, but that was how the marketplace in education worked.

Suddenly Sayukin leant forward to say with unexpected ferocity: “It is insulting if my child is a much better student, but someone who is not as good as her gets a place when she doesn’t. There are some colleges that you can enter only by paying fees, and she can’t begin to aspire to that. I find that really bad.” I commiserated, drank a toast to Karin’s future and headed back to my perch at the window.

On one side of the train a lake stretched into the distance, a flurry of whitecaps chasing towards the train. I noticed that the silver birches were swaying, not wildly but busily, in the wind and noted in my diary, with the anxiety of a fair-weather sailor, that the wind must be at force 5 gusting 6 on the Beaufort scale, perhaps 25 miles an hour.

This only mattered because Kem is on the edge of the White Sea. We were bound for a remote archipelago called Solovetskie Ostrava, renowned for its natural beauty, revered as a holy place, and notorious for the crimes of the Gulag that were perpetrated on its hallowed ground. The three-hour sea passage to Solovki, as the islands are universally known in Russia, is known for sudden storms even in summer.

By the time we reached the jetty, the wind was still rising – certainly force 6 and probably gusting 7 or 8. My trepidation grew when we joined the throng of people trying to board the dilapidated and rusting hulk of a ferry that was supposed to take us to our destination. It was perilously overloaded with tourists and pilgrims who had been transferred from three other smaller ferries deemed too frail to attempt the crossing in such weather.

Once outside the harbour, those passengers who could find no space in the cabin had to remain on the open deck, where most of them tried to find shelter on the leeward side away from the stinging spray whipped up by the wind and the sea. As a result, the ferry was listing heavily to starboard as the mounting seas rolled in on the port beam. The ferry became suffused with a mood of apprehension.

After 45 minutes or so we were in open water, breasting short, steep, ugly seas. We pitched and rolled, sometimes to what seemed to be the very extreme of 45 degrees. The boat shuddered, stayed poised on that edge for an eternity, then rolled back to start the same process all over again, as if to torture us with the possibility that what we feared was inevitable. Some passengers clutched each other in alarm. Others were crossing themselves; yet others, more resolute, started to sing what sounded in the roar of the wind like a Russian version of Abide With Me.

With the courage of real fear I decided to confront the skipper. I slid and slithered to the upper deck and staggered across to the wheelhouse. Pulling open the door, more drowned rat than ancient mariner, I yelled over the wind and engine: “I am a sailor and this weather is getting worse. This boat is overloaded and you are putting our lives in danger.”

In retrospect the words seem embarrassingly melodramatic but I believed then, as I do now, that I was telling the truth. At first – with that infuriating combination of indifference and contempt that I was starting to think was a genetic peculiarity of the Russian people – he affected not to hear me, but looked resolutely ahead at the confused and breaking seas. On my third exasperated attempt he simply said: “I shall not turn back.”

Beside him, leaning against the wheelhouse door, was his emaciated ship’s mate, sodden with alcohol, clutching on to a rail for support, his bloodshot eyes staring with benign vacancy into the middle distance. I reckoned there were more than 200 passengers on a boat designed for, at most, half that number. Two hundred souls and two crew, one of whom was a psychopath and the other roaring drunk. I eyed the life-rafts, securely fastened to the deck. If we were to capsize, none of us would have any hope.

I returned to the aft deck and, gesturing at those huddling on the leeward side, urged them to move across to windward to balance the boat. A few bold souls did as I asked – only to retreat again in the face of the wind, the rain and the sea. Not from heroism but from fear I chose to follow my own advice, and stood facing the elements in the forlorn hope that if the boat did capsize, I would at least be on the top side and not trapped underneath.

As it happened, the skipper was a skilled helmsman and managed to chart a course through the breaking seas that now raced towards us with white foaming crests that towered above us before sluicing water across the deck. I stood shivering and soaking, wishing that I was as fatalistic as our malevolent skipper. Beside me a woman read from her prayer book, never turning the page, a look of concentrated terror on her face.

It seemed an endless journey, but in fact it lasted no more than 3½ hours. Gradually, as we entered the lee of the islands, the sea and the storm subsided. We tied up alongside the jetty and, numbed by fear and cold, I felt a surge of that anger that sometimes accompanies relief: anger at the elements, at the skipper, at the state’s systemic indifference to health and safety, and at the Russians in general for being so intolerably fatalistic.

The only lasting irritation, however, was caused by a devout Orthodox Christian who told me, with the smugness of those who are blessed with certainty, that I should not have been afraid, that I had been shepherded by the Lord and that it was His will that we had survived; moreover, had we perished, it would have been God’s will as well. Fine, I wanted to retort, so long as you have faith, so long as you think you have been “chosen”, and so long as you believe that your afterlife will be better than your life on earth: I don’t.

The next day, with a gentle haze and a soft sea breeze and a few small boats bobbing in the harbour, I was reminded at once of the Isles of Scilly. But there the comparison stopped. Solovki is billed as one of the holiest places in Russia, but throughout the country’s turbulent and cruel history it has been the site of unholy horrors that – for me – shrouded the entire archipelago in a sinister embrace.

Apologists for the Soviet Union liked to believe that the Terror, of which the Gulag was the most ubiquitous example, sprang from a peculiarly demented excess of revolutionary zeal for which Joseph Stalin himself – not the system – should take the blame. In fact, the atrocities perpetrated in the name of the Bolshevik revolution started very soon after his predecessor, Lenin, seized control of the state apparatus.

As early as 1918, during the civil war, Lenin oversaw the publication of a document called The Resolution on Red Terror, which decreed, inter alia, that the “safeguarding of the Soviet Republic from class enemies” required “isolating them in concentration camps”.

The isolated monastery on Solovki promised to be an ideal location for this experiment. Dominated by high, buttressed white walls, from within which towers and onion domes sprout glitteringly into the evening sun, it seems at first glance to be as much fortress as holy place, more kremlin (as it is properly called) than monastery.

Solovki was the prototype for the Gulag empire developed by Stalin from Lenin’s blueprint. It was a model of its kind: when the official files were opened after the collapse of the Soviet Union, they revealed that, between 1923 and 1938, 100,000 prisoners were sent to the island, of whom 40,000 died either from disease and starvation or at the hands of the prison authorities.

Today a notice at the edge of the harbour invites new arrivals to respect the ecology of the archipelago by resisting the temptation to pick flowers or hunt animals, and only to picnic in designated zones. With no sense of irony, the administrator of District 50 (as Solovki is now known) writes: “Dear visitors, we hope that a keeping of the rules will not be difficult for you . . .” As if.

In summer the landscape is deceptively benign; hills and valleys form a natural bowl for a series of artificial lakes intersected by the intricate chain of canals built by monks in the 16th century to breed and harvest fish. On one summit, at the top of a hill called Sekirnaya, I was led into a glade that shelters a small white church with a long view over the White Sea. If it had been blessed with a different history, it would be the kind of haven where even an atheist might find sanctuary. But in the early days of the Gulag it was used as a punishment block, and the chanting of monks was replaced by the screams of tortured prisoners. There is no peace here.

My guide was one of Solovki’s senior administrators. Marina had given up a career as a nuclear scientist to settle permanently on the island, where she had dedicated her life to the service of the Orthodox Church as the – unpaid – head of Pilgrim Services. She knew every detail of what had happened in this simple church, which was now under restoration, and where, as they peeled away at the fabric, the restoration team had uncovered an inscription written by one of the inmates: “Comrades, remember Lenin’s legacy. Solovki is a school that teaches thugs to kill again.”

Echoing the Orthodox view, Marina told me with profound sincerity that the redemptive feature of Solovki was that it created, in her precise words, “many more martyrs for our faith”. I was instantly reminded of Monty Python’s Life of Brian – always look on the bright side of life.

On my last morning on Solovki I woke up early with a familiar feeling of dread. My room faced the Holy Lake beyond the eastern walls of the kremlin. A breeze ruffled the water, the light was soft, the sun still low in the sky, shadows shortening over the gleaming leaded spires and golden domes. I should have been elated by the simple beauty of the moment. Instead I found myself fighting a losing battle with despair.

I noted in my diary: “I wish I was away from here. I can’t understand the language. I can’t read the alphabet. I am like a blind man who is also deaf. I want to escape. I am lonely and homesick. I know I must school myself to discipline these feelings. I have to be a public person. I have to deliver. But it is draining me.”

Then, uncontrollably, my eyes started to prick and I was soon sobbing with self-pity, the monastery a blur in the distance. I feared that I was heading back down towards a depression that would coil around me and suck me into its vortex.

© Jonathan Dimbleby 2008

— Extracted from Russia: A Journey to the Heart of a Land and Its People by Jonathan Dimbleby, to be published by BBC Books on May 1 at £25 and to accompany the BBC2 series produced by Mentorn Media that will be broadcast from May 11 at 10pm on BBC2. The accompanying DVD will be available from June 16 priced at £24.99. Copies can be ordered for £22.50, including postage, from The Sunday Times BooksFirst on 0870 165 8585

Times Recommends

• Why do Pynchon, Ballard and Wallace provoke such online loyalty?
• The Sunday Times 100 best holiday reads
• What will we be reading this summer: the booksellers predict