How do you bring up a family when crack is sold in your foyer? Or when most of your neighbours scrape a living on food stamps and welfare payments? Keisha, a 26-year-old mother of two young children, explained that “if you have to sleep with a nigger downtown, then you got to do it. Because if you don't, they will put your kids on the street”.

Keisha was one of 27,000 residents in America's most famous housing estate, the Robert Taylor Homes in Chicago. The “projects”, as the 28 16-storey buildings were known, were built in 1962 along a two-mile stretch of land next to the Dan Ryan Expressway. It was a black ghetto, in which 96 per cent of adults were unemployed. To survive, you had to hustle. The local gang, the “Black Kings”, controlled every aspect of life. Guns, knives, and knuckle-dusters were routinely carried. Violence was the chief way to solve problems. This was life for men, women and children rejected by the rest of society. This African-American underclass was made in the USA.

Into this brutal world strolled a young sociology student named Sudhir Venkatesh. It was the late 1980s, and he was dressed like a fan of the Grateful Dead. He sported a ponytail. He wore a tie-dyed shirt. He was not even African-American, but had grown up in the lily-white suburbs of southern California, the son of immigrants from South Asia.

Why did he venture into an area that his university advisers had warned him against? He claims that he was simply curious. “How does it feel to be black and poor?” he asked the first posse of gang members who accosted him. In the face of their incredulity (and observing that two of these crack-dealing gangsters were armed with a gun and a knife), Venkatesh prompted them with the multiple-choice options: “Very bad, somewhat bad, neither bad nor good, somewhat good, very good.”

At the risk of seeming unkind, my first response was that Venkatesh deserved what happened next: the gang interrogated him, bickered over which one would “get the first shot”, and forced him to spend the night sitting on a freezing, urine-stained stairwell until morning. The leader of the Black Kings, a rather sympathetic thug known as J.T., then dismissed him with the words: “Go back to where you came from, and be more careful when you walk around the city.”

Venkatesh was too naive to follow this sensible advice. Instead, he turned up in front of the tower block the next afternoon bearing a six-pack of beer. It was the start of a decade-long relationship with this outcast group of Americans.

This book is a meandering memoir of his journey from unworldly, arrogant student to sensitive, distinguished professor of sociology and African-American Studies at Columbia University. At times I blushed on Venkatesh's behalf. During his first discussion with Ms Bailey, a veteran African-American community leader, for instance, he began lecturing her on the causes of urban poverty. Ms Bailey quickly interrupted the academic bullshit being spouted by this callow youth by asking him whether he would be studying “white folk”. The question confused him, until she explained that if anyone seriously wanted to understand “how black folk live in the projects” and why there was so much poverty, crime, and unemployment, they had to study white bigotry and discriminatory cultural practices and political institutions. “You've got a lot to learn, Mr Professor,” she told him. He accepted the challenge and spent years following the Black Kings as they administered beatings and sold crack cocaine. He interviewed pimps and prostitutes and hustled residents, in turn, for interesting stories about their lives.

By the end of his story Venkatesh has learnt a lot. I even warmed to him, especially when he worries whether he has done justice to the complexities of life for people struggling to survive in this harsh environment. But can he justify deceiving his chief respondent, the gang leader J.T., into believing that he would write his biography when, instead, Venkatesh has written part of his own autobiography? Venkatesh obviously revels in developing his status as the “rogue sociologist”.

There are greater problems. Venkatesh gives very little political context, let alone any solution to the problems encountered by millions of Americans on low incomes. Corrupt officials are alluded to only briefly; discriminatory policies are mentioned, but not really examined. Lengthy descriptions of gang-led aggression are not related to the broader context of violence in America. In the words of the rapper Ice Cube, criticising American foreign policy: “We do car-jackings, but that's very small when compared to country-jackings.”

Venkatesh tells us a moving story about President Clinton's visit to the projects in 1994. A substantial number of the 27,000 residents of the Robert Taylor Homes threw their energies into a frenzy of beautification to honour this popular President. Flower gardens were planted. Children were bought new clothes. The Black Kings even stopped selling crack for a day. But we are not told why people made all this fuss.

Venkatesh is the author of a couple of extremely useful books on the causes of American poverty. The lack of analysis in this volume is disappointing. It is a lost opportunity, since this book is a rollicking read. It just leads us nowhere.

Gang Leader for a Day, by Sudhir Venkatesh
Allen Lane, £18.99

Explore Books

• Book Extracts

• Book Reviews

• Books Group

• Audio Books

Times Recommends

• The 100 Best Books of the Decade
• The Times picks the 50 best paperbacks of 2009
• The conversation: James Ellroy