I seldom revisit Liverpool, but it often visits me. I wasn’t born in the city, but grew up there until I left, at 17, to go to university. So I had all those primal experiences that form a personality there: going to school, riding a bike, passing exams, discovering passions, looking at art, learning to drink, learning to smoke (failed), meeting a girl, meeting more girls, learning to drive, loving books, leaving home. But perhaps even more than sex, drink and cars, and, I suppose, books, it was walking about the city that made me.

Liverpool has a unique architectural character – unique on a global scale – but it is also very different to its near neighbour, Manchester. Liverpool was always mercantile and commercial, and the rich shipping barons, sugar merchants and edible fats entrepreneurs loved flaunting their wealth in the city centre, building fine houses, parks, pubs and galleries. Even today, many first-time visitors are astonished at the grandiose scale and swagger of Liverpool’s financial district, with its branch of the Bank of England, its stock exchange and its Victorian office buildings, whose daring structures predicted Chicago and New York skyscrapers by several decades.

By contrast, Manchester was an industrial and manufacturing centre: uncompromisingly workaday, with gloomy, sooty warehouses and mills at its heart, and the prosperous folk fleeing the city to live in leafy suburbs. Not for nothing was there an old saying, “Salford lads, Manchester boys, but Liverpool – gentlemen”. But by the time I was growing up, all that had changed, somewhat for the worse. Liverpool was now a city of scallies, Scousers and grumbling Irishmen. Of course, there were musicians and comedians too, but they soon left for the South. I cannot say there were shoeless waifs and beggarwomen on the streets, but it was close.

The dramatic decline of Sodom and Gomorrah excepted, no city in history has experienced such a shocking and rapid descent as Liverpool. First the Luftwaffe; then the disappearance of the Atlantic trade; then the shift of Europe’s centre of gravity; not to mention the fantastical, obstructivist disinclination of the population to adapt, plus a slew of demented politicos; all these turned the Venice of the North into its Marseilles.

But walking these streets, I learnt about architecture and how it affects the personality of cities and of the people who live in them. Still today, I often begin lectures by explaining the Liverpool building scene. Here is one arrangement of stone, iron, bricks and glass that is exciting, dignifying and pleasurable; here is another that is ugly, nasty and crushes the spirit, sucking light and energy from the universe. What’s the difference? Design.

Liverpool had lots of the latter, especially in the poorer, bomb-damaged districts, where grim French system-building techniques were employed with a tad too much enthusiasm, making the wretched point-blocks that helped turn the name Liverpool into journalistic shorthand for modern urban misery. The Liverpool City Plan of 1965 was conceived by Graeme Shankland and Walter Borin in a spirit, now seen as misguided, of new-age euphoria. It proposed to eviscerate the centre and penetrate it with flyovers and fly-unders. Through a mixture of lassitude, good fortune and an inspired campaign by my own tutor, Quentin Hughes (a raffish Liverpool Welsh soldier, antiquarian and architect, alas, now gone) it was never wholly realised. But, in the same year, the city did tear down the old market, replacing it with the artless and soulless St John’s Precinct. Beryl Bainbridge, a local girl, said “Someone’s murdered Liverpool and got away with it.” Sure, it's a rough old place.

But against this was the fabulous architectural inheritance of a happier age. Fortified by Nikolaus Pevsner’s Pioneers of Modern Design and Penguin’s bestselling anthology of Mersey poets, The Mersey Sound: Adrian Henri (in fact, a Birkenhead Frenchman), Brian Patten and Roger McGough, I made it my business to understand architecture. I saw Frederick Gibberd’s Catholic cathedral – a Scouse version of Oscar Niemeyer’s Brasilia – going up. They called the conical structure “Paddy’s Wigwam”. It stands above the crypt of Edwin Lutyens’ polychrome and perfumed Byzantine design which, Liverpool being Liverpool, never got finished.

At the opposite end of Hope Street was Gilbert Scott’s awesome Anglican cathedral, made of local sandstone in science-fiction Gothic of eye-popping scale, standing on a rock, surrounded by a terrifying canyon that accommodates, among curiosities and condoms, a monument to William Huskisson, the Liverpool politician who was the first person to be killed by a train. At another end of the site stands John Foster Jnr’s perfect, miniature Greek Revival temple, inspired by this rich boy’s Mediterranean sightseeing on the Grand Tour. On a good day, when you stand by it, the river glitters distantly.

Of course, they have still not finished Scott’s cathedral either, but made enough progress by the time I was 15 to furnish me with a true sense of what a sublime frisson really meant – as well as a keen understanding of architecture’s power to exalt, as well as dismay. But then there were more sedate experiences, too: Rodney Street and Gambier Terrace – Georgian architecture as fine as any in Dublin or Bloomsbury; the Albert Dock – Jesse Hartley’s superlatively muscular brick warehouses with iron Tuscan columns, abandoned back then, but now the bright home of Tate Liverpool and any number of WAGs’ bars.

Then there was Quarry Bank, my old school, a genuine Liverpool curio: a Gormenghast house in a leafy park where teachers in mortar boards admonished us in Latin, while also driving Mini Coopers; and the Philharmonic pub – which taught me Art Nouveau while serving halves of bitter. These buildings still today form the townscape of my dreams – and my nightmares. That’s the effect growing up in Liverpool can have. But I am not alone. He did not live there, yet Carl Gustav Jung also dreamt of Liverpool. It’s not just architectural, it’s archetypal.

But there were 20th-century buildings, too. That famous waterfront is, to my mind, at least as powerful a composition as Manhattan. Walter Aubrey Thomas’s Liver Building, Briggs and Wolstenholme’s Mersey Docks and Harbour Board and Willink and Thicknesse’s Cunard Building is the trio that became known as the Three Graces – a difficult trio to follow, as recent experience has shown.

In 1966, Denys Lasdun, who designed the National Theatre in London, built the University Sports Centre, the radical, sloping concrete profile of which was (to an adolescent aesthete) a thrilling and fine rebuke to the prim and orderly arrangements of Abercrombie Square’s adjacent terraces. Next door was the Electrical Engineering Building, by modernists YRM, in severe white tile. I loved this one: if I half-shut my eyes when the sun was shining, I could dream I was in Palo Alto. Liverpudlians, even assumed ones, are notoriously sentimental, but, equally, very inclined to nourish dreams of leaving.

Editor's choice

• The End: the worst movie endings ever
• Introducing The Times Proms Podcast
• Heath Ledger tipped for Oscar