In the Spring of 2019, I was asked to collaborate with NYC’s Technodramatists to design and build a live facial motion capture system and interactive 3D content for a one-woman adaptation of Shakespeare’s Comedy of Errors. The challenges inherent with real-time motion-capture and live animation were increased given that the audience would also see the performer physically performing live in front of them, making the relationship between expressiveness and performance of the utmost importance.
Over several weeks I developed a content pipeline and interactive project in Epic Game’s Unreal Engine that was custom suited to the needs of the project. Working with the actor and director, we chose the traditional Italian masked theater of Commedia dell’arte as a major creative & visual reference. This reference helped us establish and play with the relationship between the Augmented Reality projections and the performance itself, as well as opportunities to influence the storytelling and character relationships inherent in Shakespeare’s Comedy of Errors.
Client: Technodramatists
Executive Producer: Lorne Svarc
Actor: Claire Tyers
Director: Noa Egozi
Lighting & Projection Designer: Kris Stone
Claire Tyers expressive performance was amplified with face-tracking and live animation with Unreal Engine and custom made CG masks.
Actor Claire Tyers fearlessly portrayed more than 6 different characters using Augmented Reality to capture and project multiple masks to expand on what is possible with live theater technology. Facial animation data was captured with a latest generation iPhone Xs. Claire was able to control character switching with a Nintendo Switch controller installed on the performance rig that was reprogrammed to communicate with our Unreal Engine software.
A short breakdown video showcasing the different masks that were used and some of the considerations that went into our interactive development.
Facial motion capture was executed with a custom app installed on an iPhone Xs. Our software took advantage of new functionality introduced by Epic Games ARKit integration, and their incredibly helpful Face AR Sample Project, which is freely available on Epic’s site. Many thanks to Epic Games for the work that went into this integration and execution.
Overview of the Unreal Engine project layout (It’s a pretty simple layout actually!).
Each mask was calibrated via custom blueprint logic that was used to tweak & amplify the mask performance to the actor’s specific expressive facial qualities.
We tracked more than 50 facial data points to drive animated blendshapes in UE4. A combination of gyroscope and tracking data were used to recreate head and neck rotation values.
Claire Tyers gave more than 10 public performances of the 1st 4 scenes from the play. The system was nimble enough that under her control she could control the pacing and flow of every performance, allowing for improvisation whims and breaking the 4th wall.