The Serpentine Pavilion, like summer this year, is late — very late. Still, at least it’s actually bothered to turn up. Re-spun by the Serpentine Gallery as an “autumnal pavilion” instead of the summery teahouses the gallery has commissioned in the past seven years, it opens in Kensington Gardens on Friday.

Don’t blame head-in-the-clouds designers for this tardiness, though. The architect Kjetil Thorsen (from Norway’s most illustrious firm, Snøhetta) and the world-straddling Danish artist Olafur Eliasson — famed for The Weather Project, his wildly popular 2003 “giant sun” installation for Tate Modern’s Turbine Hall — were only drafted in in the spring, after the designs of this year’s original choice, the German engineer Frei Otto, proved to be too ambitious to be realised in time.

In this case, though, our loss is also our gain. Otto’s structure — now pencilled in for next year — would doubtless have dazzled with technical virtuosity, only to be expected from a man famed for virtually inventing, with his rollercoastering stadium roof for Munich’s 1972 Olympics, the flamboyant, gravity-defying, Gehry-esque architectural forms we now take for granted.

With Eliasson and Thorsen’s pavilion, though, we are treated to “not just the wow effect,” says Thorsen, “which simulates a fun ride — this is nothing like a rollercoaster”, but something far subtler, something far more refined, thought-provoking, and perfectly suited to its very English landscape setting.

Which makes it sound awfully well-mannered and dull. Not a bit of it. Like past Serpentine Pavilions the steel frame structure, clad in plywood panels, delivers the requisite killer looks. The gallery has already dubbed its giant, tilted cone bound in a spiralling ramp “the spinning top”, though it’s as much like a flying saucer crash-landed from a 1950s B-movie: ominous, like a Trojan horse, and melancholically gloomy (in a very northern latitude, kind-of-optimistic sort of way).

For where it differs from past pavilions is in gravitas. This is the cleverest pavilion yet. You walk in and out of it, on a ramped walkway that twirls in and out of the building right to the top. Each of the previous pavilions had its thing, its formal point — Oscar Niemeyer’s languid, Brazilian elegance in 2003, say, Alvaro Siza and Eduardo Souto de Moura’s malevolent crouching beast in 2005, Daniel Libeskind’s waltzing spikes in 2001. But only two have previously added intellectual depth to formal pizzazz. MVRDV’s plan in 2004 to cover the gallery building with an artificial hill, and last year’s balloon by Rem Koolhaas and Cecil Balmond, explored similar ideas to Eliasson and Thorsen’s, although the former was unrealised and the balloon ended up deflated in execution, if not concept.

The series, initiated by the Serpentine Gallery director Julia Peyton-Jones in 2000, has become the undisputed zenith of Britain’s architectural year — not that there’s much competition. Mogadon modernism rules the UK, but this is one patch of the nation where, for two months a year, “value engineering”, PFI and per square metre rental opportunities are not the governing laws; where A-list intellectual architects with precious little hope of getting hired in “real” Blighty rule the roost and unfettered architectural ideas can take flight without taking root in this most select of London boroughs (after their temporary installation, the pavilions are privately sold to part-finance them).

This one is no different. It may be cast in autumnal hues of, says Thorsen, “unfashionable brown”, but it sure packs a shapely punch. Eliasson and Thorsen’s pavilion fuses concept with form in a tight, explosive intellectual package, exploring the very idea of a pavilion itself. Both men are fascinated by how architecture, the shaping of space, acts as a membrane between you and the rest of the world, the subject and the object.

Thorsen’s work for Snøhetta — such as Egypt’s Alexandria Library and the recently abandoned Turner Contemporary gallery for Margate — gently manipulates our perception of space. The same goes for Eliasson’s projects, such as The Weather Project or Your Black Horizon, his collaboration with the architect David Adjaye in which the participant is submerged into a world dark save for a mesmerising horizon.

Eliasson’s work demands that one’s body and all five senses are immersed in the work and coaxed by it into engagement, so that the “artwork” isn’t so much out there, but in you — think of that Tate Modern sun. Lately Eliasson has edged into architecture, designing — with Henning Larsen Architects — the skin for Reykjavik’s National Concert Centre and collaborating with Thorsen on the new Opera House in Oslo, while his plan for Washington DC’s Hirschhorn Museum ditches the tourist’s artistic routemarch in favour of an “immersive” gallery experience which, he says, “the visitor constructs as much as the curator”.

Their pavilion, then, is a device not for viewing, but experiencing. Its launch pad is Britain’s one great contribution to spatial design, the 18th-century landscaped park. Cultivated, aristocratic minds then employed eminent designers such as Humphrey Repton and “Capability” Brown to create country estates — Stourhead, for one — as theme parks for the intellect.

These were picturesque landscapes containing a string of architectural spaces — Gothic pigeon houses, Chinese pagodas, Palladian bridges, faux ruined temples and, of course, pavilions — to stimulate the sublime, a rush of beauty to the head. Kensington Gardens is a little axial for Eliasson, a tad French with its straight, Haussmann-esque avenues, rather than the rambling, picturesque lanes and studiedly “casual” views of Hyde Park across the road. But it and the Serpentine Gallery, built as the Gardens’ original teahouse pavilion, are basically of the same heritage.

Their pavilion is a walk in the park all contained in one building — a spiralling promenade through a string of spatial experiences; enclosure, exposure, vertigo, cosiness, echoey coolness, corridors, caves, balconies, promenades and magnificent framed views of trees and leaves as the park shifts from summer through autumn to winter.

And just wait till you get inside. Particularly intense is the huge, grotto-like cave interior, a lair for trolls, studded with “geological” polygonal columns like the Giant’s Causeway and lit by a twisted oculus and a lonely Juliet balcony. Here, alongside the pavilion café’s cups of tea — which, given the interior gloominess, and the current climate, I anticipate to be on the nippy side (bring jumpers) — “intellectual experiments” are to be dispensed by the pavilion’s events programme.

Whereas the 18th-century pavilion basically reinforced a particular way of seeing, artistic convention today demands a more questioning eye. Nothing in this pavilion is as it seems. Its shape seems simple. But look hard and its complexity becomes apparent: its cone tilts, no two of its triangular scales are the same, planes that seem parallel shift away from one another, that cone’s shell is warped, the hole at its zenith tilted and elliptical.

Perspective is constantly shifting, an effect accentuated by the whirling ramp, whose outside “wall” of string, trembling nervously in the breeze, comes and goes disconcertingly. You can be sure of nothing. That walk in the park suddenly got rather intense.

Serpentine Gallery Pavilion, Kensington Gardens, London W2 (020-7402 6075), from Friday to November 5

Editor's choice

• U2 at Nou Camp Stadium, Barcelona
• The 50 Best Movie Villains
• The 15 golden rules of theatre etiquette