BY RIGHTS I should hate Robert Venturi and Denise Scott Brown. History has it that they invented that dreaded, horrifying evil . . . Post-Modernism. In most architects’ offices, thank God, Po-Mo, the architecture of the Seventies and Eighties, is still nostalgia way beyond the pale. One glimpse of a pastel façade or an “ironic” historical reference hurtles me back to my Eighties childhood of the Prince of Wales and classicism triumphant, when all there was to satisfy the soul was wallpaper-thin theme-park stuff. Back then, like some Soviet Russian supermarket, you could have any architecture you wanted, so long as it was Po-Mo.

But as the historian Robert Miller says, blaming Venturi and Scott Brown for Po-Mo is a little like blaming Edison for disco music. The husband and wife, now in their seventies, are in London next week, defying fashion to give a rare lecture. The P word is a dirty word in their house too.

“I am not a Post-Modernist and have never been a Post-Modernist,” declares Venturi to me, wearily. “Forget about all that ancient and boring stuff. We didn’t have any control over it.” They have, they say, always called themselves modernists.

They’ve had a lifetime of being misunderstood. Po-Mo is the Frankenstein’s monster that burst from misreadings of their two most famous manifestos — Complexity and Contradiction in Architecture (1966) and Learning from Las Vegas (1972, with Steven Izenour) — both of which changed the course of 20th-century architecture. The first said it was OK to be a modernist and like history. Venturi, bravely at the time, declared his love for Renaissance, Baroque and Mannerist Italy, and called for a return to, well, complexity, contradiction and symbolic meaning, all lost in the impoverished abstract modernist landscape of the Fifties and Sixties.

To Ludwig Mies van der Rohe’s famous “Less is more” quote, Venturi infamously replied “Less is a bore”, and proceeded to make a classically inspired house for his mother in Chestnut Hill, Philadelphia. The controversy it caused! But architects fell upon the book.

Learning from Las Vegas said it was OK to be a modernist and love Vegas tack, the American capitalist roadside vernacular of burger bars, neon signs and billboards. Thirty-five years before Rem Koolhaas wrote about learning from Junkspace, Venturi, Scott Brown and Izenour cast Caesar’s Palace as a modern Hadrian’s Villa, and burger bars and neon as modern versions of ancient Greek temples or Christian churches with their narrative mosaics, murals and frescoes: “just signage — but we call it high art.” Again they came up with a handful of catchy mantras, such as “the decorated shed”, or architecture with a wallpaper-thin façade; and “Main Street is almost all right”, meaning, look at and appreciate the ordinary, populist, supposedly ugly middle American landscape sprawling around you. The controversy it caused!

These books loosened up architecture. Without them there’d be no Gehry, no Libeskind, no Herzog & De Meuron. They’d be no bad Po-Mo either. But that’s only because most architects lacked Venturi and Scott Brown’s wit and erudition, forgot the “it’s OK to be a modernist” bit of their message, and used the books as an excuse to turn on modernism’s progressive politics, rummage clumsily through history and plaster the globe in cheap tack for two decades.

The pair had opened a Pandora’s box. He said learn from Borromini and Vegas, not copy them. Symbolic depth, not pastiche. Use architecture to communicate with the public democratically — “as a sign, rather than as space” — but don’t dumb down. It’s about finding the right exterior language for the interior function.

Their own decorated sheds are variations on the theme of sign/billboard/advert outside and “main street” behind. The Best Catalogue Showroom, an out-of-town shop, sits behind an almost literally wallpaper façade, patterned “for its obvious appeal to the viewer”, to grab the eye of passing car drivers. Their Trubek and Wislocki houses in Nantucket pick a woody, classical-meets-New England barn style. The Museum of Contemporary Art extension in San Diego is bloated Spanish style. Behind the stage-set façades they function well, mostly thanks to Scott Brown’s thoughtful planning.

Sometimes, though, it all goes horribly wrong. All that knowing façadism can become wearying; the jokes and signs can miss the mark. A big aluminium antenna for Philadelphia’s Guild House sheltered housing to signify its TV-glued inhabitants inside? A ship’s wheel rose window for a harbourside house? Please. Houston’s Children’s Museum, with cartoon kids for caryatid columns, called “caryakids”? Boom, boom.

The National Gallery’s Sainsbury Wing remains their most controversial work. This was, of course, the building which replaced the original “carbuncle” denounced by Prince Charles. There was a lot riding on it. “We poured our heart and soul into the building,” Venturi says. But they found themselves foreigners walking into a parochial nest of vipers. In those days you were either a modernist or a traditionalist. These two were both.

Their building offended “snobbish” British critics, “but part of problem is they didn’t know enough history,” Venturi retorts. They called it pastiche, but that was partly the point. It’s a comment on the Eighties architecture style wars. OK, Charlie, you want history? Here it is, like wallpaper on a roll. OK, modernist critics, you want pastiche? Here it is in spades. The gallery, says Venturi, “a billboard done in Portland stone”, playing with the dull façade of William Wilkins’s 19th-century gallery, as if to say, is this what all that fuss is about?

I don’t care for its more obvious tricks, such as the overtly false trusses not holding up the interior’s staircase ceiling. But behind the façade there’s a functioning building, whose galleries, based on John Soane’s Dulwich Picture Gallery, show up the Renaissance paintings beautifully.