Stalin died in 1953, his body now lying outside the walls of the Kremlin, but his ghost is still with us. That is one of the things I learnt over the past five years as I travelled back and forth between England and Russia to interview the last survivors of his Great Terror – a generation that is about to disappear. This was a unique opportunity because the average age of the people who told us their stories was 80 and many have since died. People handed over letters that had been hidden under mattresses, diaries – some written in code – and boxes of photographs.

Tonya, in her sixties, showed me an old towel that her mother had embroidered while in a labour camp. She had never spoken before about being the child of a gulag prisoner for fear of persecution. I soon realised that the first thing that happens when a family has suffered repression is that the children suffer “inherited” or habitual fear. It passes down the generations, reducing voices to a whisper and silencing tongues.

So secrecy and dread of reprisal did not end with Stalin, the dictator who is estimated to have persecuted more than 25m people during his tyranny. Fifteen years after the collapse of the Soviet Union, my team of researchers found the old anxieties very much alive in Vladimir Putin’s Russia.

Our task was to collect family archives for the Memorial Society, a charity set up under Mikhail Gorbachev to represent Stalin’s victims and create an online archive. In all, we collcted more than 200 archives and conducted more than 500 interviews. All this material would also form the basis of a book, The Whisperers (published by Penguin), about private lives in Stalin’s Russia. We asked more than 1,000 people to take part; only half agreed. I know that my book will be translated into as many as 15 languages, but Russian will probably not be among them. No publisher there will be brave enough. Putin’s attitude is increasingly one of “let’s move on”.

Those who did speak had to be helped to overcome their lifelong fear of talking to strangers. The intimate subject matter of our interviews had been a forbidden zone where most survivors of the Terror had never dared to venture before. What was life really like under his regime? What was it like to live with moral compromise? How was it when your father was arrested as an “enemy of the people”? How was it when a husband organised the disappearance of an unwanted lover? And how did you reconcile what you knew with what the state told you?

These were awkward questions about people’s interior lives. We asked about relationships, about love, trust and betrayal. We wanted to know about the internalisation of Stalinist values, what Mikhail Gefter, the Russian historian, has called “the Stalinism that entered into all of us”, the fear that kept the system going for so many decades. Playing back these tapes is distressing: they are filled with people just crying.

Perseverance, sometimes involving a dozen visits, got people to open up. Having lived in a society where millions were arrested for speaking inadvertently to informers, many older interviewees were extremely wary of talking to researchers with microphones, a device associated with the KGB, even though they knew we were from the Memorial Society, a highly trusted organisation that helps to find information about missing relatives. Some were frightened that they might “say the wrong thing” or get into trouble if they volunteered too much. A few withdrew from the project altogether on these grounds. Others reacted aggressively to questions about their political loyalties during the Soviet period. “What, are you trying to get me arrested?” they would say. At times it was hard to remember that this was not 1930s Russia.

Nona Panova, 78, from St Petersburg, became hysterical when she saw our microphone. She had been talking for about four hours before she noticed it. “But I’ll be arrested,” she cried. “They’ll put me in jail.” She still believed she could be sent to a gulag in northeast Siberia. In the event she changed her mind and continued with the project. Like so many, she said she would not sleep that night. This oral history is raw and alive.

It is that inherited fear that stopped the next generation joining dissident circles. Active resistance to the system was tiny. That fact is relevant in today’s Russia where Putin could sweep away the constitution without mass protest. There is a certain conformism in postSoviet society that plays into the hands of authoritarianism. After the years of glasnost in the 1990s under Gorbachev and Boris Yeltsin, many victims of repression now feel they are being increasingly marginalised under Putin.

Only a few years ago things were different. Yeltsin saw to it that these victims of repression had high status, passing a law that recognised “moral damage” by the state. They received federal compensation. Now that recognition is no longer there and these terms are being quietly dropped.

Meanwhile, the Memorial Society itself is coming under pressure with Putin going on the attack against NGOs funded by the West. Putin would rather talk about the achievements of the Stalin years – the victory of 1945 above all.

My sense is that younger Russians want to forget the Soviet period, but can Russia become a democracy without truth and reconciliation? There is frequent comparison between Stalinism and the Nazi dictatorship. But the latter lasted for only 12 years and Germans have been asking themselves awkward questions ever since. In the Russian language there are two words for a whisperer. One for somebody who whispers out of fear of being overheard (shepchushchii), another for those who inform or whisper behind people’s backs to the authorities (sheptun). In Stalin’s era the whole of Soviet society was made up of one kind or another. The consequences are lasting.

Explore Books

• Book Extracts

• Book Reviews

• Books Group

• Audio Books

Times Recommends

• Superfreakonomics by Steven D Levitt and Stephen J Dubner
• The Times picks the 50 best paperbacks of 2009
• Vincent van Gogh’s letters show the soul of the artist