Work Package 1
Advanced capturing and creation of animatable volumetric video
Leader: Peter Eisert (HHI)
Objectives
WP1 aims at developing novel technologies for capturing and processing 3D models of persons while preserving their natural mimics and gestures, thus avoiding the uncanny valley effect. It addresses the following objectives:
- To capture real persons by multiple cameras and represent them as sequences of temporally registered 3D meshes (volumetric video)
- To develop novel post-processing, animation and synthesis tools based on deep learning
- To develop new hybrid representations combining volumetric video with graphical models for photorealistic animation of humans
- To create a database of 3D mesh and skeletal animation sequences representing different human motion snippets, which are combined and interpolated for highly realistic face and body animation.
The goal of the work package is to generate natural representations of real persons, which can be fully animated enabling new forms of immersive multi-user collaboration and storytelling. Volumetric video capture that supports high realism and the reproduction of fine details as well as subtle motion and expressions is combined with computer graphics models, which allow for interactive animation and editing. The resulting hybrid representations enable the photorealistic rendering of real humans while providing the possibility of pose modifications orthe creation of entirely new content from sample data. By relying on real captured motion sequences, which are re-combined and interpolated during animation,realism is targeted that is difficult to achieve with state-of-the-art CGI models.
