by Fons Kuijk, Rufael Mekuria and Pablo Cesar
REVERIE is a general-purpose tele-immersion system that enables individuals to interact in a shared 3D synthesized environment. It is a meeting place for virtual characters representing real participants and receptive virtual agents (autonomous humanoids). Participants observe this virtual reality from the dynamically changing point of view of their virtual embodiment (an avatar or a replicant) and 3D spatial audio completes the impression of being immersed.
Tele-immersion enables individuals that are geographically apart to interact naturally with each other in a shared 3D synthesized environment in which each of the participants can be represented by a virtual character. Since tele-immersion systems are not restricted to the recordings of static cameras, they enable participants to see the virtual environment from the dynamically changing point of view of their virtual representation. REVERIE’s virtual environment is one example of such an ambient, content-centric Internet-based environment where people can work, meet, participate in live events, socialize and share experiences as they do in real life, but without time, space and economic limitations. Users are represented either by an avatar (a geometrical model of a human that can be animated by means of parameters) or by a replicant (a surface model of the user captured with multiple cameras and reconstructed as photorealistic 3D meshes or point clouds that are transmitted in real-time). In this environment we may also find agents, fully autonomous virtual humanoids that do not represent a human user in the real world, but act like one.
Figure 1: A fully autonomous agent acting as tour guide for a group of students being represented by avatars.
Figure 2: REVERIE scene with four participants, two of them represented by replicants and two represented by avatars.
The REVERIE system [1] is a distributed architecture of web-connected clients that can be configured to run the modules needed for a particular application. Modules of the REVERIE system analyse captured data from the real human user and model the interaction between virtual and real, including facial expressions, gaze and gestures, as they are essential aspects of non-verbal communication. The components can be deployed on user computers, as they do not require a heavy setup (Kinect, webcam and microphone) and therefore are well suited for a web-based communication system.
Non-intrusive real-time recognition processes enable the speech, the gestures, the emotions and the facial expressions of users to be recognized by the system in a dynamic and natural fashion. This way, real-time puppeting of avatars is made possible along with mouse-keyboard-triggered actions such as navigation or requesting to speak. Depending on the activity of the users, the avatars produce rich human-like behaviour that is visually understood by other users and virtual agents.
Using similar control structures augmented by reasoning components, the REVERIE system manages autonomous virtual agents that may act as tour guide or lead a game [2]. Moreover, agents are capable of reacting autonomously to the users’ activity in various ways, detecting when a user is losing engagement or participating in the interactions and showing appropriate behaviour (gestures, gaze, speech) in response.
Unlike avatars and agents, the replicant representation is a real-time captured model of the user for which transmission of 3D geometry poses a large challenge in compression. Therefore, the REVERIE system contains a framework for geometry compression to address the challenge of immersive communications. The framework includes three different codecs that have different properties and can be used in different situations depending on the need of the application [3]. As spin-off we have contributed a base reference software platform for MPEG (where CWI leads an activity of the 3D Graphics group) that can be used for collaborative development of technologies for compression of time varying point clouds that includes colour and prediction schemes. This work led to an update of the MPEG-4 requirements for including time varying point clouds for supporting 3D tele-immersion scenarios and will allow industries to rapidly develop and integrate these technologies in their industrial systems.
The user experience of the REVERIE system has been evaluated in a field trial including avatar and replicant representations. In this study, participants in groups of four (two being represented as replicants and two as avatars) interact in a virtual room to complete a collaborative task. We learned that a replicant representation has some advantages: the user represented is instantly recognized, the representation demonstrates what the user does and where s/he is looking, and one can see the user’s emotional state. Because of this, participants recognized the usefulness of the replicant representation in the successful completion of a collaborative task. On the other hand, an avatar representation, which uses less bandwidth, is preferred when limited facilities leads to low-quality replicants (e.g., having artefacts or missing user body parts), which degrade the user experience. Completing tasks with low-quality replicants would require users to rely solely upon verbal channels. Avatar representations are also useful to enable users to hide from view and still have a virtual embodiment that mimics their actions in a “natural” way by pose, gesture and emotion recognition.
The results of these field trials highlight the potential of the replicant representation in virtual worlds, and the relevance of design, implementation and evaluation of geometry codecs to support this representation.
This work has received funding from the European Union’s Seventh Framework Programme FP7/2007-2013 under grant agreement no. ICT-2011-7-287723 (01/09/2011 - 31/05/2015). In this project fifteen renowned research institutions and industrial players with balanced expertise covered all the technical aspects needed to achieve the challenging project goals: STMicroelectronics, Queen Mary College, CTVC Ltd, Blitz Games Studios, Alcatel-Lucent, Disney Research, HHI (Fraunhofer), TP Vision, CWI, Telecom ParisTech, DCU CLARITY, Synelixis Ltd, ITI (CERTH), Microsoft Innovation Center Torino and WooX Innovations Belgium.
Link:
http://www.reveriefp7.eu/
References:
[1] P. Fechteler et al.: “A Framework for Realistic 3D Tele-Immersion” International Conference on Computer Vision / Computer Graphics Collaboration Techniques and Applications (Mirage 2013), Berlin, Germany, June 6-7, 2013.
[2] F. Kuijk et al.: “Autonomous Agents and Avatars in REVERIE’s Virtual Environment”, in Proc. WEB3D 2015, 20th International Conference on 3D Web Technology, June 18-21 –Heraklion, Crete, Greece.
[3] R. Mekuria, et al.: “Enabling 3D Tele-Immersion with Live Reconstructed Mesh Geometry with Fast Mesh Compression and Linear Rateless Coding,” IEEE Transactions on Multimedia (TMM), 16(7): 1809-1820, 2014.
Please contact:
Fons Kuijk
CWI, The Netherlands
E-mail
