PhD Position F/ M Leveraging Radiance Fields Rendering for Asymmetric Collaboration in Extended Reality

Contexte et atouts du poste

Inria. Two Inria teams, located at different sites (Rennes and Sophia-Antipolis) will be involved in the PhD. The Hybrid team (Dr. Ferran Argelaguet) will ensure the direction of the PhD and bring their expertise in VR, 3D user interfaces and avatars. The Grapheco team (Dr. Geroge Drettakis) is a world leading team in image-based rendering, although not directly involved in the co-supervision on the PhD, will provide support for the fast radiance-field rendering methods.

IMT-Atlantique. Two members of the INUIT team (Lab-STICC, Brest) will take part on the co-supervision of the PhD. Dr. Cédric Fleury, will bring the expertise on virtual reality and computer-supported cooperative work and Pr. Guillaume Moreau will bring the expertise on augmented reality.

Both partners are also involved on the EquipEx+ Continuum , the PhD will benefit from the VR and AR platforms at Inria Rennes/IRISA and the Lab-STICC.

Context and Methodology

Augmented Reality (AR) has been largely explored for remote assistance under a wide range of applications domains, such as industrial maintenance or home assistance. In such a context, one user, having access to an AR system, asks guidance from one or multiple remote users (collaborators). To ensure optimal collaboration, first, remote users need a precise understanding of the workspace of the AR user (workspace awareness). The awareness of the workspace enables the remote collaborators to understand and assess the environment in which the AR user is located. Second, users should be able to be aware of the actions of the other users (user awareness) and should be able to communicate with traditional interaction modalities, such as voice, gaze and gestures. User awareness is required to ensure efficient communication and interaction among users, as users should be aware of the actions of others. Finally, workspace and user awareness must be synchronized, as interactions are linked with elements of the physical workspace.

However, due to the difficulty to ensure free exploration for remote collaborators, workspace awareness still remains an open problem, and it is typically supported either by virtual replicas/reconstructions or video feeds. However, with the appearance of radiance field rendering methods such as Neural Radiance Fields (NeRF) or more efficient point-based solutions, real-time and high-fidelity reconstruction of physical workspaces are becoming possible. This PhD will leverage fast radiance-field methods to enable efficient collaboration in asymmetric configurations, in which AR users asks for support to remote collaborators in virtual reality.

Thus, t his PhD aims to explore how fast radiance field methods can be used to enable workspace awareness, and investigate the user representation that will maximize user awareness.

Mission confiée

Radiance-Field methods, which have the potential to achieve better quality than 3D static reconstructions without the need of specific instrumentation, still present a number of challenges for its direct application to remote collaboration scenarios.

The first tackled challenge will be (C1) progressive workspace reconstruction. In order to reduce bandwidth and preparation, the radiance-field reconstruction should be done in real-time using head-worn RGBD sensors (e.g. Hololens 2), thus the exploration of the AR workspace will determine the quality of the reconstruction. The method of provides the required speed for this to be achieved. The first objective will be to (O1) propose interaction methods to allow remote users to drive the data gathering (e.g. pilot the AR user so pertinent views are gathered).

(C2) Handle uncertainty in the reconstruction. Due to the progressive reconstruction, it might be partial or incorrect, remote users should be aware of this uncertainty. (O2) Propose communication protocols and feedback solutions to validate the reconstruction and/or provide ground-truth to the remote collaborator (e.g. couple radiance-field rendering with a video feed or display the uncertainty of the reconstruction quality). 

The final challenge will be to (C3) ensure user awareness . All collaborators will require to be aware of the actions of others. However, the interaction capabilities of each user will be different, while the AR user will be constrained by the physical workspace, remote collaborators could have “super powers” (e.g. teleport, go through virtual objects). In such context, the user representation will be play an important role of the subjective perception of the experience and potentially impact on the user awareness. (O3) Assess the role of the user representation and interaction capabilities for user awareness.

Principales activités

The first six months of the PhD candidate will be do a thorough analysis of the state of the art and develop a VR radiance-field-based rendering prototype. The next twelve months will be focused on reconstruction fidelity assessment and the study of referential awareness, dealing notably with uncertainty. The next twelve months will be focused on the study of user awareness, this will require the study of the user representation and interaction capabilities. The final six months will be focused on the writing of the manuscript and to the preparation of the defense.

Compétences

The candidate must have MsC in computer sciences, with a focus either on virtual reality, human computer interaction or computer graphics. In addition, the candidate should be comfortable with as much following items as possible:

The candidate must have good communication skills, and be fluent in English. 

Avantages