How would you like to be the “voice of a generation”? Sounds all right, doesn’t it? As the “voice of a generation”, you would be not only famous, but also really important, your every changing mood sending ripples through our culture. Surely it would give you a great sense of freedom, too, knowing that whatever you did would be lapped up by eager crowds.

Except it turns out that being the “voice of a generation” isn’t liberating at all. It’s some kind of artistic prison sentence. Only a few weeks ago, we were able to read Bob Dylan’s thoughts on the matter in the first volume of his autobiography, Chronicles. The very idea sends him off on a rant: “The big bubba of the revolution, leader of the freeloaders, kaiser of the apostasy.” He speaks of being “backed into a corner”.

And now, here is With the Lights Out, a four-disc box set spanning Nirvana’s brief but extraordinary career, and the three CDs (the other disc is a DVD) offer us a compelling story. On the first CD, we witness Kurt Cobain’s musical vision becoming ever clearer; on the last CD, we hear him returning to that vision; in between, we hear him being rudely interrupted by becoming the “voice of a generation”. We have always known that the idea sat uneasily with him, but it has never been so clear just how much the early music of Nirvana had in common with his last songs, and to what extent Nevermind was an aberration.

The fact that it was also a 10-times-platinum landmark, loaded with brilliant songs that changed the face of rock music, makes referring to it as an aberration seem rather odd, but listen to With the Lights Out all the way through and you will be forced to the same conclusion. For us, Nevermind was a work of genius; for Cobain, it was a detour. He was perfectly capable of writing the best radio-friendly rock songs of his era, perfectly capable of summing up modern life in a couplet, but that wasn’t what he had to do, it wasn’t the music he needed to get out of his system. It wasn’t the point.

It is much the same awkward lesson that we have had to learn about Dylan. The thing that we love about him isn’t the thing that he cares about. What makes him so special to us isn’t special to him. Like a child waving an autograph book at our favourite footballer and being told to sod off, we suddenly realise that our hero isn’t exactly who we want him to be.

Although 68 of the 81 tracks in the box set are previously unreleased, this includes a goodly number of alternate takes on well-known songs — solo demos, band demos, radio sessions and other live performances. The real feast of new material, for most listeners, lies on the first CD, where we hear Nirvana (before Dave Grohl’s arrival on drums) exploring different styles, but quickly finding their musical feet. Back in 1987, Help Me I’m Hungry gives us our first taste of their quiet/loud dynamic — almost certainly before Cobain could have heard the Pixies, which suggests that rather than being a fundamental influence on Nirvana, their songs merely served as confirmation to Cobain that he was heading in the right direction.

If You Must, a track from 1988, sounds eerily like an outtake from In Utero — one of many musical connections between the band’s earliest days and their last album. Another comes when Pen Cap Chew and a cover of Leadbelly’s Grey Goose see Cobain’s guitar-playing sounding uncannily like Mick Ronson on Bowie’s The Man Who Sold the World album (the title track of which Nirvana covered on their Unplugged album).

The full, world-conquering pop power of Nirvana is unleashed on the second CD. There is a version of Smells Like Teen Spirit that is — apparently — almost the first time the band played through the song together. If this is true, it is an astonishing confirmation of their intuitive brilliance. Although poorly recorded, it is virtually fully formed: Grohl and the bassist, Krist Novoselic, aren’t learning the song, they are already driving it. Grohl’s “Wake up, world” intro was clearly the first idea that came into his head. This is the kind of jaw-dropping moment that justifies the whole idea of box sets.

On the third CD, we hear Cobain reacting to the way that we reacted to Nevermind. There’s a comment in his journals: “Nirvana can’t decide whether they want to be punk or REM.” Well, they tried being REM and it didn’t fit, so now it’s back to something a lot more like punk. The music is sometimes ugly, painful to listen to, but Cobain is going back to where he was heading before he was so rudely interrupted by becoming a megastar. He is trying to find his soul again. Judging by his suicide note, he never thought he found it, never felt that he’d recaptured the excitement of the band’s early days — but judging by the music here, he really did.

Much of the pleasure of this final CD comes from the visceral excitement of hearing Nirvana exploring ideas together, but the high point is in the final few tracks, when a solo, acoustic Cobain sings All Apologies, You Know You’re Right and Do Re Mi, a “new” song that may be the best clue we have to where he was heading next. The only real disappointment is that there aren’t more clues as to what might have been the next Nirvana album, or — as Courtney Love has hinted — the first Cobain solo album. If we are to believe Love (and that is, as they say, a big ask), she has dozens more cassettes of unreleased Cobain songs. With the Lights Out was delayed for years while Love battled with Grohl and Novoselic over the nature of the set, and some believe that she is planning a separate solo Cobain box set at some future date. There is another comment in Cobain’s journals: “I cried to Seasons in the Sun.” If the idea of him tearing up to Terry Jacks’s hit seems a little bizarre, it’s not half as bizarre as seeing Nirvana play the song. This is one of the strangest moments on the DVD. To accentuate its strangeness, Cobain is on drums, Grohl on bass and Novoselic on guitar.

Much of the DVD is taken up with a 1988 band rehearsal in a small room in Novoselic’s mother’s house. Rather than maintaining eye contact with his bandmates, Cobain turns away from them and sings with his face only inches from a wall. There may be mundane practical reasons for this. He may have been trying to minimise feedback by keeping his guitar turned away from the nearby amps. But it looks as if you are watching Pink Floyd’s existential concept album, The Wall, on a very small budget. And it drives home why Cobain had such problems with his band’s phenomenal success. He’s not singing his songs to anyone; he doesn’t care where they go; all that matters is that they come out of him.

With the Lights Out is released on Geffen tomorrow; £39.99