ASKED IN 1965 TO DRAW THE line between song and poetry, Bob Dylan had a neat, if perhaps disingenuous answer. “Anything I can sing,” he said, “I call a song. Anything I can’t sing, I call a poem.”

That was in the days when to be called a poet was the highest compliment that a songwriter could receive: from Sam Cooke to Neil Young, from Rod McKuen to Smokey Robinson (Dylan called him “America’s greatest living poet”), songwriters were the lyric voices of their time, the voices of our conscience, our grief, our inexplicable joys, our deepest fears and, most of all, our loves — which meant that they had to be something more than mere pop singers, they had to be dignified by the label “poet”.

Yet the truth is that, whether on the street or in our most private moments, no meaningful distinction can be made: poetry and song have always worked in the same way, towards the same ends: making music from rhythm and melody, to be moving and memorable. No surprise, then, that poets and musicians have always collaborated; what is odd is that so little attention has been paid to the link between songwriting and fiction.

Songs have always told stories. From the border ballads to Schubert’s Winterreise, from the Icelandic sagas to Dylan, song has been a vehicle for narrative — although more often than not that narrative has come from a more magical world, and from a different timezone, than the tales that we usually spin in novels and short stories. In the 1960s and 1970s, many musicians tapped into the ballad tradition, either by resurrecting and renewing old songs themselves, or by using their techniques to make new stories, new myths.

Often these new songs involved a journey and a transformation and, at their best, they were supremely effective fictions: Joni Mitchell’s Hejira, Leonard Cohen’s Famous Blue Raincoat, Dylan’s own Tangled Up in Blue or the underrated Brownsville Girl — are all narrative works of the highest order, distilled into an intensely lyrical experience: proof, if proof were needed, that song is as much akin to the short story as it is to the poem.

Nevertheless, when the 1960s musicians branched out, they tended to publish poetry collections. (Cohen was the one successful novelist.) These days, singer-songwriters are more likely to publish short stories or novels: Steve Earle, for example, is a marvellous prose writer, whose short story collection, Doghouse Roses, garnered wide critical acclaim, while Willy Vlautin’s novel The Motel Life is just as spare and heartbreaking as the work he does with his band Richmond Fontaine.

I cannot help thinking that this is a positive development: songwriters know how narrative works, they need to be capable of extraordinary economy, they understand the importance of mystery in any narrative work. “Anything I can sing, I call a song,” a contemporary songwriter might say, “and anything I can’t sing, I call a story.”

As it happens, I write both poetry and fiction — although until recently, I had never tried my hand at song. In fact, I found the idea of writing song lyrics terrifying, for reasons both technical and emotional. This changed when the Chemikal Underground record label invited me to take part in a project called Ballads of the Book, in which a number of Scottish writers produced lyrics for leading Scottish musicians, including Vashti Bunyan, Idlewild, the Trashcan Sinatras and King Creosote.

Not an entirely new idea, perhaps, but what made this project so interesting was that both poets and prose writers were invited to contribute. Well over half the lyricists on the album are best known for their fiction. Speaking for myself, I found that my own song lyrics work felt closer to fiction, in both shape and impulse, than to poetry.

Indeed, it was poetry, in the way that I normally understood it, that I most wanted to avoid — the reason that Dylan praises Smokey Robinson so extravagantly, I think, is that his lyrics never create the necessary distance that one finds in poems, they simply happen, directly, organically.

What I wanted was a core simplicity, with the merest suggestion of a storyline: luckily for me, the settings — by Norman Blake and Mike Heron — brought this out beautifully. At the same time, the music draws out narrative elements that I had played down, filling the spaces that I had avoided taking up with something quite outside my imagining, so the song is neither mine, nor not-mine, but a shared story, with a life of its own.

Songs from Ballads of the Book will be performed on January 30 at the Glasgow Royal Concert Hall Strathclyde Suite (see celticconnections.com for details). The CD is out in March on Chemikal Underground records.

Explore Books

• Book Extracts

• Book Reviews

• Books Group

• Audio Books

Times Recommends

• The 100 Best Books of the Decade
• The Times picks the 50 best paperbacks of 2009
• The conversation: James Ellroy