An obsessive in his quest for truth, W Eugene Smith once said: “I have in mind this dream of perfection.” His inability to attain this ideal left him restless, unfulfilled, addicted to drink and drugs, bankrupt, and his first wife and family destitute. He died aged 59 with just $18 in the bank. However, he left an enduring legacy – a vast archive of photographs, the product of a tortured, self-destructive life.

Smith was born on December 30, 1918, in Wichita, Kansas. His father ran a grain business that went bust, after which he committed suicide by shooting himself in the stomach with a shotgun when Smith was 17. His mother, who frequently took pictures of the family, introduced him to photography as a child. By 14 he was taking pictures professionally for local papers and in 1934 was published in The New York Times.

He enrolled at the University of Notre Dame, Indiana, in 1936 on a scholarship created specially for him, but quit after a year to work in New York. Here he began a tempestuous career that made him, in his own words, “red-hot” in the photojournalism world, but virtually impossible to work with owing to his demands for complete control over his work. He was fired from his first staff job at Newsweek. He had a stop-start career with Life magazine that included two resignations, shrapnel wounds incurred during the second world war (“I forgot to duck”) and a spell in a psychiatric institution. During his years with the renowned photographic agency Magnum, he only ever completed one project, repeatedly unable to limit his visions or work to a deadline.

Smith’s hallmark was complete immersion in his art. During the war he ate, drank and smoked with the soldiers on the Pacific Islands, sharing their pain and fear. For two months he lived with Spanish villagers, and gained unprecedented insight into life under Franco. He traced the work of Dr Albert Schweitzer in his African village hospital for nine years before producing a monumental photo essay. But disputes over complete editorial control saw him resign finally from Life. Smith became increasingly particular, spending endless hours in the darkroom, and obstructively argumentative with editors. A heady mix of alcohol and amphetamines would keep him working for days on end, after which he would often collapse. He was manic: playing music at full volume all night, buying thousands of books (his final collection was over 8,000) and writing rambling letters to family and friends.

Two of Smith’s most famous photo essays are Pittsburgh and Minamata. The former began in 1955 as a three-week project to produce 100 photos of contemporary Pittsburgh. It became a three-year odyssey: 17,000 negatives were whittled down to 2,000 prints, which made up innumerable, ever-changing layouts pinned to walls in Smith’s flat. Paid just $1,900 for its eventual publication in Popular Photography magazine’s 1959 annual, Smith felt it to be an inadequate representation of his vision – despite having creative control over the layout – describing it as “the final failure, the debacle of Pittsburgh”. He was broke; his housekeeper had taken on a second job to try to prevent the family’s inevitable financial ruin. Threatened with a lawsuit, Smith abandoned it all and holed up in a run-down New York apartment.

Smith travelled to Minamata, a small Japanese fishing village, for his final project. From 1971, with his second wife, Aileen M Smith, he gained the confidence of the villagers, who had suffered the effects of industrial toxic waste that had been dumped in their water since the 1950s. Aileen remembers “Gene”, a gentle giant who “had a childlike enthusiasm for everything, but as well as being a great photographer he needed people to save him [from his depression and addictions] along the way”. In contrast to the bolshie, difficult man that his employers knew, Aileen describes his way with the people of Minamata as “the total opposite of arrogant and overbearing”.

Enrica Vigano is curator of PhotoEspana’s new exhibition of Smith’s work. She spent nine months sifting through Smith’s photographs, layouts and notes, listening to his recorded memos. But would the indomitable Smith approve of what Vigano has achieved with his work? She hopes so, but as if Smith’s compulsive work ethic is catching, says: “If I’d more time, I’d continue, because his work is almost infinite.”

Eugene W Smith: More Real than Reality is at the Teatro Fernan Gomez-Centro de Arte, Madrid, June 3 – July 27; www.phedigital.com/festival

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010