Happy 400th anniversary to Shakespeare's sonnets. They are still going strong. You can buy dozens of different editions, ranging from pocket keepsakes for lovers to hefty hardbacks with hundreds of pages of scholarly explanations. Last summer you could hear Simon Callow recite the whole sequence of 154 at the Edinburgh Festival. And this month in Berlin the avant-garde American stage director Robert Wilson has teamed up with the singer-songwriter Rufus Wainwright to turn 24 of them into cross-dressed theatrical dreamscapes. So what is the source of their enduring power?

On May 20, 1609, a publisher called Thomas Thorpe entered in the Stationers' Register his right to publish “a booke called Shakespeares sonnettes”. A few weeks later, browsing the bookstalls in the yard of St Paul's, you could have found the little volume and purchased it for sixpence.

Probably the greatest love poems in English literature (though John Donne runs them close), the sonnets introduced to the language such phrases as “shall I compare thee to a summer's day?”, “the darling buds of May” and “remembrance of things past”. They express almost every permutation of love, from the first leap of the heart at the sight of the beloved to the last ache of sorrow and bitterness in the face of death or, worse, betrayal.

Reading the sequence through, there seems to be a story - though, in contrast to Shakespeare's plays, the twists of the plot and the nature of the characters are shadowy and mysterious. The poet begins by addressing a beautiful and high-ranking young man. The youth is in a position of power and the poet in one of supplication. Absence, travel, scandal, melancholy, estrangement and reunion are variously implied. The young man appears to have an affair with the poet's mistress, thus abusing the bond of friendship. Then the poet is discomposed by a rival who wins the patronage of the fair youth with his “well-refined pen”.

Again and again, the sequence returns to the great battle between love and time. The mood becomes autumnal (“Bare ruined choirs where late the sweet birds sang”). Time is relentless (“Like as the waves make towards the pebbled shore,/ So do our minutes hasten to their end”), but the act of writing offers the hope of immortality (“So long as men can breathe or eyes can see,/ So long lives this and this gives life to thee”).

Then the poet turns his attention from the “lovely boy” to the “dark lady”. Dark-complexioned and sexually voracious, she inspires a complex mix of emotions: desire, fondness, self-abnegation, misogyny, a lingering sense of the sour taste that comes after sex (“The expense of spirit in a waste of shame/ Is lust in action”). One moment the poet is bitter, the next dazzlingly playful, as he parodies conventional love poetry (“My mistress' eyes are nothing like the sun”) and puns on the multiple senses of the word that is also his own name: Will.

We think of love sonnets as the most personal of poems. The little book called Shakespeare's Sonnets is a source of endless biographical fascination because it seems to be the one work in which its author speaks in his own voice. There is, however, no intrinsic reason why a sonnet should not be a dramatic performance, just as a play is. It may be that for an Elizabethan poet to dash off a sequence of sonnets was a kind of exercise. Shakespeare could have invented the “plot” and “characters” of his sonnets.

We simply do not know whether the sonnets are dramatic performances written out of sheer imagination or poetic reimaginings of real figures and events. Because Shakespeare is so guarded, the circumstances of composition have provoked centuries of speculation. The young man to whom most of the poems are addressed may or may not be synonymous with the mysterious “Mr W. H.” who is named in the collection's dedication “to the only begetter of these ensuing sonnets.”

The traditional candidates for the role are the Earl of Pembroke and the Earl of Southampton, though neither was a “Mr”. A provocative case has been made for the possibility that “Mr W. H.” is actually a misprint for “Mr W. S.” and that in the dedication Thomas Thorpe, the publisher, is merely acknowledging Shakespeare as the “only begetter” of the sonnets (“begetting” was a common metaphor for authoring).

Unlike Shakespeare's narrative poem Venus and Adonis, the bestselling literary work of the Elizabethan age, the sonnets were not reprinted or frequently quoted. The vogue for sonneteering had passed its prime by 1609. Some scholars have supposed - without any direct evidence - that they were actively suppressed because of their risky sexual orientation.

Dozens of male Elizabethan poets wrote sonnet sequences, but only Shakespeare and a certain Richard Barnfield addressed their poems explicitly to a man. Barnfield wrote in the explicitly homosexual tradition of ancient Greek pastoral poetry, whereas Shakespeare's sequence emphasises the spiritual aspects of the poet's love for the fair youth. The only sonnets in the collection where “Will” is actually in bed with a lover are addressed to the dark lady. Taken in their entirety, the sonnets associate heterosexual desire with consummation and disgust, homoerotic attraction with spirituality and an intensity that derives in large measure from the impossibility of consummation. It might be better to read the opposition between dark lady and fair youth as a dramatic device: one is a “character” representing desire in its sexual manifestation, the other in its idealising and spiritual one.

That is what I always tell my students - and myself - as I sit down to reread the sonnets. Don't be drawn into the trap of supposing that they are autobiographical: that is an illusion of Shakespeare's art. But it's very hard to stop yourself. When I worked on them for my book The Genius of Shakespeare in the 1990s, I became convinced that I had identified the dark lady: she was the wife of John Florio, the Italian tutor in the household of the Earl of Southampton. When I returned to them recently for my book Soul of the Age, I became convinced that I had identified the rival poet: he was John Davies of Hereford, the greatest calligrapher in England and a hanger-on in the circle of the Earl of Pembroke.

Each time the sonnets had worked their magic: they had made me project a story of my own into their narrative. They work like love itself by making you want to join your story to that of another. And that, of course, is why they are the greatest of all love poems and why they are still so fresh after 400 years. When Shakespeare writes: “Let me not to the marriage of true minds/ Admit impediments” the two minds that are joined are no longer his and his lover's. When I read the poem, they are mine and my lover's. When you read it, they are yours and your lover's.

Jonathan Bate's Royal Shakespeare Company edition of the sonnets is published this month

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010