The Cliff - Performing Arts Centre on the Fjord

by Leif Arne Rønningen

The Cliff is a revolutionary concept for an arts centre on the fjord. It will create a new and exciting multi venue scene for audio and visual experience. The realization of The Cliff requires extremely low-delay networks between different venues, multi-view collaboration surfaces, and both internal and external multi-view video arts scenography.

Networked music collaboration, which is related to augmented reality (AR), has been carried out for decades, using available technology. In most cases, only 2D video is used, and unwanted long delays are often experienced. However, a number of successful productions, including the networked Thora paa Rimol opera [1], have applied the LOLA system. The LOLA (LOw LAtency audio visual streaming system) project aims to enable real time musical performances where musicians are physically located in remote sites, connected by advanced network services. [1].

The Cliff [2] represents a new concept in performing arts centres that can be transported to visit venues at the seaside, and is intended for networked collaborations via a digital network. Applications are typically music drama, theatre, concerts of all kinds, multimedia shows, and art exhibitions. Furthermore, The Cliff’s exterior is a sculpture that can be changed dynamically by light, colour and water art.

The interior can be used in all weather, and the exterior when weather allows. In particular, on dark winter evenings when parts are covered by snow, spectacular sculptures can be created.

The surface of The Cliff will consist of many irregular multi-view surfaces, built of lens arrays that will be projected on by lasers from inside. The images generated will be visible from both inside and outside. A large number of miniature cameras, hidden in the surfaces, will provide the sufficient number of views to obtain near natural spatial feel. A networked processing system synchronizes all projections. Scenes can be built anywhere in the space, and projection walls, ceilings, floors and objects support the scenography and provide collaboration surfaces for networked collaboration.

A major advantage of video based scenography is the extreme dynamics of scene objects, which allow changes in milliseconds rather than seconds or minutes as with traditional scene technology. The marginal production cost is also much lower with video scenography and pre-recorded music.

Even if, in principle, the design and construction of The Cliff could start today, there remain both technical and artistic research challenges. The large, laser scan based multi-view displays needed, with views on back and front, are not yet available. And there is still a way to go before the perception of the virtual presentations can be near natural. Another challenge is to enhance the artistic value of the distributed or networked production of music drama compared with more traditional productions.

NTNU (Norwegian University of Science and Technology) and SINTEF (the largest independent research organisation in Scandinavia, and the natural contract R&D partner for the private and public sectors) have world-leading competence in ICT and ocean science and technology, strengthened by the ‘oil age’. NTNU also have widely acknowledged activities in performing arts. Their combined expertise puts them in the ideal position to establish The Cliff as an extraordinary world-class performing arts centre.

Aida by Verdi as networked AR performance
For the opera Aida, the scene in The Cliff is planned to be a hill of stones, built as a promontory to a wall, with the audience on three sides. The multi-view surfaces around the promontory are used for video scenography and collaboration surfaces for two additional venues. All other walls, ceilings and floors are also used as vision projection screens. The live performance is distributed and takes place at three venues each of which can accommodate over 1000 audience members. Most of the visualizations in The Cliff can be viewed externally from the city or the Trondheim fjord:

• The main soloists and characters play live in The Cliff. The performance is recorded, transported and shown on large, mobile, multi-view screens in the collaborating venues.
• Trondheim Symphony Orchestra plays live in Olavshallen (Trondheim) and is recorded and shown in The Cliff. When the orchestra plays overtures or the main role, video is presented on the periphery of The Cliff space, surrounding the audience entirely. An extract is shown in the Nidaros Cathedral on mobile screens. In all cases, superb 64 (or higher)-channel sound with extremely low delay is spread from the periphery both in The Cliff and Nidaros Cathedral.
• A large mixed choir (of priests, soldiers and others), performs live in Nidaros Cathedral (Trondheim). Both ensembles and their surroundings are recorded and shown in real time on the multi-view surfaces in The Cliff, and in Olavshallen.

To meet the hard delay sensitivity given by the distributed performing musicians, the technical delay requirements are visual (optical) end-to-end delays of less than 30 ms, and audio delays of less than 10 ms. To accommodate these requirements, a 100 Gbps IP/Ethernet fibre network with switches takes care of the transmission between the venues, including high performance image and sound processing equipment that needs to be optimized to support the low delays.

Links:
LOLA system :
http://www.conservatorio.trieste.it/artistica/ricerca/progetto-lola-low-latency
Nettmusikk2014 [In Norwegian]:
https://openwiki.uninett.no/_media/ nettmus:nettmusikk2014-brukerstyre_2013-11-13.pdf
Thora paa Rimol [In Norwegian]:
https://openwiki.uninett.no/_media/nettmus:thorapr.pdf
Collaborative Surfaces and Spaces (ITEM@NTNU) :
http://www.ntnu.no/telematikk/people/personalpages/fac/leifarne/ collaborationspaces
The DMP Architecture, Specification and Philosophy (ITEM@NTNU) :
http://www.ntnu.no/telematikk/people/personalpages/fac/leifarne/the_dmp_architecture_specification_and_ philosophy
NTNU Faculty of Information Technology, Mathematics and Electrical Engineering :
http://www.ntnu.edu/ime
NTNU Oceans :
https://www.ntnu.edu/oceans/about-ntnu-oceans

References:
[1] Parts of the Thora paa Rimol opera as a neworked collaboration (augmented reality) version, were played between Dokkhuset Scene and Orgelsalen, Trondheim, 9th of December 2014, Project Nettmusikk2014, http://kwz.me/JJ.
[2] L.A. Rønningen: “The Cliff. Performing Arts Centre on the Fjord”, Memo, NTNU/Item, May-June 2015.

Please contact:
Leif Arne Rønningen
NTNU, Norway
E-mail: This email address is being protected from spambots. You need JavaScript enabled to view it.
http://www.ntnu.no/ansatte/leifarne