In the film Doubt the white-robed priest asks: “What do you do when you’re not sure?” What the film’s writer and director John Patrick Shanley would like us to do is to talk about it afterwards — in excited little knots on the pavement outside the cinema, over restaurant tables, on the bus home and again the next day, when we’ve slept on it and thought a bit more.

The central, obvious dilemma of the film and of Shanley’s original Pulitzer prize-winning play is this: has Father Flynn (Philip Seymour Hoffman) been conducting a sexually abusive relationship with Donald, a boy in his care? And if he has, can the school’s principal, Sister Aloysius (Meryl Streep), prove it?

But her determination to nail this turbulent priest raises other, more important and difficult questions: how do we cope when our beliefs are challenged; can we weather uncertainty; what constitutes that woolly little word “abuse” and how to weigh it against kindness? In the most tense and riveting scene of many, Sister Aloysius shares her concerns for the boy’s welfare with his mother, a black woman working several cleaning jobs to pay his way through school. She seems inclined to turn a blind eye to the nun’s suspicions (I won’t divulge why). Is this a shocking dereliction of maternal duty or understandable expediency?

Despite its ponderous, flag-waving metaphors — sudden, swirling “winds of change”, lightbulbs inexplicably going out, sunshine blazing on to the priest’s face as he is interrogated — this is a disturbing film. It stays with me overnight and into the next morning, when I am ushered into a small salon on the third floor of a Knightsbridge hotel, where Shanley’s tall frame is folded into a plush armchair. The subdued, intimate light from a couple of table lamps is conducive to the small nap I would have had if, like Shanley, I had just flown in from New York. Has he slept? “Yep. On the plane, in the hotel room — slept anywhere anyone would let me.

“It’s a gift, sleeping, especially at one’s age [he’s 58],” he smiles. “Well, as the Sister says in the film, I’m not sure we’re supposed to sleep well’.”

I felt quite upset by the film, I tell him. There is a pause before he responds and an odd expression — a wince, possibly — crosses his face. “When you said that, it provoked a moment of grief.” He frowns. “Partly perhaps because you look like one of my family.” Another pause. “It’s grief for the loss of something that is gone: a lost world that I grew up in.”

That world was the Bronx; a place of settled, working-class, God-fearing, largely immigrant, wholly white families. He vividly remembers the first black pupil to arrive in his class and, more than 30 years later, when he came to write Doubt he would cast Donald as the only black kid in the strict Roman Catholic confines of St Nicholas, the fictional school modelled on his own alma mater, St Anthony’s. He went back a year ago to ask the Bishop for permission to film there (he was refused): “It is a good place, full of Latinos; warm. It wasn’t warm in my day.”

In 1964, the year in which Doubt is set, Shanley turned 14. President Kennedy had only recently been shot and the world was on the threshold of change. “Till then it had all seemed eternal,” Shanley says. “Within three years it was gone. Everybody picked up and left — the place was in flames.” The house he grew up in is now covered in barbed wire and surrounded by a security fence. “So we couldn’t film that.”

For Catholic communities the first warning “vibrations in the ground” came with Vatican II, the second ecumenical council that ushered in modernising reforms of the Church. In the film Father Flynn is charismatic, friendly and informal in sweat pants, playing basketball with pupils. Sister Aloysius is the face of resistance, worried about the erosion between Church and State and the advent of the biro. “Soon nuns would shed their habits and bonnets,” Shanley says. “I wanted to capture something about that lost moment.”

But the germ of the idea that would become Doubt came, not from memories of 1964, but from the run-up to the 2003 Iraq War. Shanley was struck by the lack of debate about the pros and cons of invasion. “There was no true discussion going on,” he says. “Whichever side people were on, there was such certainty. I thought, ‘Am I missing something?’ ” At times, he says, he felt very lonely. “You want to have a conversation where someone can have an effect on you. But there was nobody to talk to about our ignorance of what was going on.”

So Iraq fired him up; how come, then, he wrote about child abuse and the religious politics of the Sixties? “I wasn’t interested in the church scandals themselves,” he says. “I was looking for a polarising situation, one in which most people would not hesitate to condemn a person, and throwing those assumptions back at them.”

He wanted to explore the mechanism of doubt, he explains, unravelling facts and truths the audience might think are clear at the outset, then leaving them to explore the loose ends in their own way. “What was always important to me,” Shanley says, “is that the sense of doubt belongs to the audience. I’m not going to tell them what’s right or wrong — or how to think and feel.”

There is a denouement of sorts, though it depends how you interpret it as to whether you believe Father Flynn is guilty and of what, exactly: child sexual abuse, even the scent of it, is a stalking horse for our most complicated fears. “It’s such a bad term,” Shanley says. “Nobody goes through childhood without having felt strange energy from an adult — it’s part of puberty, entering the sexual world. When all those church scandals broke, people I’d been at school with would go, ‘Is this news?’ We remembered this or that priest and say: ‘He was always keen on the boys’."

He knew who the predators were at his school but, though he was regularly in trouble for rebellious behaviour, he did not suffer their attentions. But then he came from a secure family. He was the youngest of five children of Irish immigrant parents: “My mother and all we kids read a lot and the family held generous, liberal views.”

Was he close to his mother? “Intellectually, yes, but she was cool.” Was she frustrated? “Very much so. She’d been taken out of school at 15 to help to support her family and married at 18. I knew from the first day that she was waiting for us to leave.” What did she want? “To read her books and be left alone.”

His father had arrived in the Bronx at 24 and worked for the next 30 years at a meat packing company. “There was a savage side to him,” Shanley says, “the way he walked, ate, spoke — he was forceful. He would yell ‘I love you’ like a pirate.”

Is the son like him? “I guess. You need that to do what I do.”

What he does is write plays — off-Broadway until Doubt transferred and carried all before it, winning a slew of prizes — and film scripts; the screenplay for Moonstruck won him an Oscar in 1987. He often directs his own work and famously refuses to change a word of his scripts: “You have to be prepared to walk away,” he says.

When it toured the UK, the stage version of Doubt was not universally admired — reviews claimed that its series of dialogues was not sufficiently dramatic and that the narrative tension arrived only at the end. Did he draw from such criticism when he came to write the screenplay? “I have been waiting a long time for critics to show me the way,” Shanley says evenly. “I dream of having a mentor.”

You wouldn’t rush to accuse this man of intellectual humility, so it’s not surprising, when I ask if he believes in God, he says: “Yes, but I don’t need an intermediary.”

He has two adopted sons from his marriage, now ended. One lives with him, the other is a regular visitor: “Adoption is great,” he says, “As long as it is for the right reasons. No good having them because you want to be loved. I was fine with it either way; my kids don’t have to like me."

The writing in Doubt is accomplished and poetic — a legacy, perhaps, of Shanley’s Irish roots (“They speak in poetry there,” he says.) In his sermon, Father Flynn talks of public despair that can be shared. But what, he asks, do we do about the individual “stricken by a private calamity”? That was my favourite phrase in the film I tell Shanley as we part company. He inclines his head.

Doubt opens on Feb 6

Explore Film

• Film Reviews

• Trailers

• DVD

• London Film Festival

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010