I was on my way to the airport in LA last November, just after the start of the Hollywood strike, when the cabbie offered to swing by the studios to “take a look at the writers” — as though he were talking about the collection of sculptures that a billionaire had put out on his front lawn for the day. A mile later and there they were, masses of them, holding up placards. “Writers Have Souls Too” was the one inspiring the most honks. It’s a line I suspect that Marc Norman would like to have stamped on every page of this inspirational book.

Norman, Oscar-winning co-author of Shakespeare in Love, offers What Happens Next as a history of screenwriting in Hollywood from 1896 to 2006. His book is more than 500 pages long and rabidly intense. Put it down to get a drink and Norman will follow you to the off-licence, talking hard about the tricks writers have pulled to be the last one working on a draft in order to guarantee a screen credit, or recalling how Eric von Stroheim closed down the set of Greed for the day “until the guardsmen extras were furnished with authentic silk underwear”, or describing Gary Cooper, cool in the witness box in Washington during the anti-communist purge of screenwriters in 1947, slyly shrugging: “I could never take any of this pinko mouthing very seriously, because I didn’t feel it was on the level.”

He’s taken the writers and films he considers most pivotal — Raymond Chandler, Billy Wilder, Quentin Tarantino, Casablanca, Being John Malkovich, and many more — and tells the ever-shifting story of screenwriting through them, frequently charging on with details about other practitioners, flitting back and forth, all the time emphasising how tough writers in Hollywood have had to be.

It wasn’t until about 1919 that it occurred to audiences that such a thing as a screenwriter existed. In silent-era Hollywood, very little was written down. When DW Griffith made The Birth of a Nation in 1915, he simply formed the narrative in his head as he went along. “Nothing seemed to go together,” recalled a panicked observer on the set, “nothing seemed to fit.”

Norman describes Griffith pulling the story out of 150,000ft of negative in the cutting room, working through the nights holding up “strips of workprint to a lightbulb and cutting with scissors”. The film made $14m at the box office, and a personal profit for its distributor Louis B Mayer so huge that it set him up for life. The stakes were now too high to leave anything to chance. “From now on somebody would dream up movies and write them down before they were filmed.”

For every writer Norman has a whole peacock’s fan of anecdotes, some new, some familiar, artfully written to make the reader see — as though Norman were at times coming up with scenes for a film of What Happens Next itself. So, we see Herman Mankiewicz, low and paranoid after making Citizen Kane, listening to the 1941 Oscar ceremony over the radio at home. His wife Sara is on the bed; he is in a bathrobe. “Suddenly he heard his name announced. Welles’s name as co-writer couldn’t be heard for the screaming for ‘Mank! Mankiewicz! Where’s Mank?’ Hollywood was calling for him, his peers, giving their beloved, self- destructive gentleman . . . this one prize, this one moment. It was the only Oscar the picture won. Mank and Sara danced wildly around the bedroom, him on his bad leg.”

We see George Lucas trying to write Star Wars for two years, then three, hating every moment, paralysed with ennui, driving his wife Marcia nuts. “On bad days he snipped at his hair with desk scissors,” writes Norman — a perfect image, since Lucas has always had rather vain, theatrical hair, sculpted up and back like the Grinch’s. Norman just lets us imagine this hair, along with drafts of his ghastly script, clanging “dead into the wastebasket” to join the bits of tuna sandwiches brought up to Lucas by Marcia, also in a bloody mood.

We see Dorothy Parker sitting wryly in her Santa Monica pool-house with her knitting needles and her husband Alan Campbell, dictating dialogue while he typed. “ ‘And then what does he say?’ Alan asked. Dorothy’s answer was soft but audible. ‘Shit.’ ‘Please don’t use that word,’ Alan muttered.”

Norman hasn’t had a film of his own made since Shakespeare in Love in 1998. No wonder. He’s been camping in libraries combing through old memos and notes, diaries and interviews. His book is at its best, its most William Goldmanishly fun, when he loosens his collar and briefly confides his feelings on seeing a script of his own go into production and becoming “human, subject to compromise and intentions, original sin”.

In the end, Norman doesn’t really know why writers in Hollywood are forced to remind people they have souls. Maybe it’s simply because the very definition of being alive in LA is that you have a screenplay in your back pocket — every schmuck you meet is a writer. But what Norman does know is that everyone else involved in a movie can break a writer’s heart. A producer can ignore you, a director can ban you from the set, actors can mumble your words or simply out-dazzle them with their faces and bodies. Because no matter how brilliant the scene, some people just don’t have any respect. As proved by the story about a friend of Marlon Brando, who visited him in Tahiti and said how terrific he’d been in Apocalypse Now — the film that almost killed Francis Ford Coppola, who sat night after night at his typewriter in the jungle, ranting and raving and terrifying his wife, producing only two pages every 12, pot-addled hours, channelling Joseph Conrad down the centuries while assistants and children and Playboy Bunny extras ran to helicopters to get away. And Brando merely asked, “Is that the one where I was bald?”

The screen players

Marc Norman won an Oscar for Shakespeare in Love in 1999. In three decades of screenwriting, he has seen just two other scripts of his go into production, neither of any consequence. And that Oscar was no joke, either. Tom Stoppard, who “polished” the Shakespeare script, capped Norman’s earnest speech with an effortless display of wit that made it clear to the world where Gwyneth and Joe’s sparkle had come from. Norman is ideally qualified, therefore, to describe development hell. (Not a hot hell. Just a long one.)

What Happens Next: A History of American Screenwriting by Marc Norman
Aurum £20 pp553

Available at the Sunday Times Books First price of £18 (inc p&p) on 0870 165 8585

Times Recommends

• Superfreakonomics by Steven D Levitt and Stephen J Dubner
• The Times picks the 50 best paperbacks of 2009
• Vincent van Gogh’s letters show the soul of the artist