Picasso at the National Gallery? Unthinkable. Scantily clad mannequin figures in the hallowed home of Rubens, Rembrandt and Raphael? Impossible - until now.

The nation's premier collection of Old Master paintings is to make a foray into the Tate's modern art territory, its new director Nicholas Penny said yesterday.

A Picasso exhibition and a provocative installation work in which visitors become peeping Toms peering into prostitutes' rooms - re-creating the Amsterdam Red Light District - would usually be Tate Modern's pitch. Instead, both shows will be staged next year by the National Gallery.

The Red Light installation will be more controversial than anything previously staged by the National Gallery, Dr Penny said.

This may mean the end of an attempt to define rigid boundaries between the two galleries, and the end of the Tate's monopoly on modern art and shock. Only 12 years ago, the two public institutions agreed that 1900 marked the year that modern art was born - where the National's collection should end and the Tate's begin.

The two galleries exchanged Monets, Matisses and Pissarros that were painted on either side of 1900. The National sent its Picasso still-life, Fruit Dish, Bottle and Violin - because it was painted in 1914 - to the Tate, where it has remained. In return it has works such as Van Gogh's Farms near Auvers of 1890. Dr Penny said: “The idea is not to have an agreement. We are not happy with 1900 as a final, absolute point of the end of the National Gallery.”

The exhibition Picasso: Challenging the Past, which opens in February, will feature works borrowed by the National from a variety of museums. The Tate is not among the lenders so far.

It will explore the influence of Old Masters, including Velázquez, upon Picasso, and the ways in which he sought to challenge their art.

Some 60 works by Picasso will be divided into sections dedicated to the self-portrait, the female nude, and the still life, among others. The exhibition will show how he borrowed from the Old Masters directly and at other times by allusion. Picasso made the implicit case that it was he, in the 20th century, who most forcefully reinvigorated the European tradition.

More controversially, the National will give space in November 2009 to a walk-through installation by Ed and Nancy Kienholz, artists from the United States. The Hoerengracht (Whores' Alley) includes lifesize mannequins in various stages of dress or undress, each with a glass frame around their faces. Asked whether visitors will be shocked by the work's explicit subject-matter, Colin Wiggins, the gallery's head of education, said: “It depends who you are.”

The installation will be displayed alongside “genteel and intimate” 17th-century Dutch paintings that touch on the theme of prostitution.

Robin Simon, editor of the British Art Journal, condemned the programming of such a provocative piece.

“I wouldn't worry about removing the 1900 cut-off, which is pretty arbitrary. There's no reason why the National Gallery shouldn't show Picasso, but it has no business whatever in dealing with contemporary installations which are far better put somewhere else. The National Gallery shouldn't be wasting its space or resources on contemporary installations.”

The agreement of a 1900 division made in 1996 was for an initial four years and never formally renewed, it emerged yesterday, although some 60 swapped paintings have not been returned to their original owners. It followed the two galleries' agreement in 1927 that the year 1870 - when Impressionism and the first of the modern movements began - should mark the beginning of the modern era.

Rigid time divisions hardly make sense in artistic development terms. Monet, for example, is shared by both galleries, with later images that bear a more abstract quality viewed as “very much of the 20th century” and therefore suitable for the Tate. His Woman Seated on a Bench, of 1874, belonged at the National whereas his Water-Lilies of 1916 was one for the Tate.

A spokeswoman for the Tate suggested that the issue had not yet been laid to rest: “We are in discussion about the 1900 divide.”

Editor's choice

• The 100 Best Films of the Decade
• 70 facts you didn't know about Marvel
• The 50 Biggest Movies Of 2010