SentiMask SDK
3D face tracking for augmented reality apps and digital characters control
SentiMask is designed for development of augmented reality applications, which use real-time 3D face tracking technologies for motion capture and controlling 3D digital character's facial expressions or mapping animated avatars on user's face. The technology works with regular cameras and common PC or smartphones.
Available as a software development kit that provides for the development of 3D face tracking systems for Microsoft Windows, Android, iOS, macOS and Linux.
Features and Capabilities
- Real-time face detection and tracking.
- Facial pose, landmarks, shape and expression estimation.
- 3D facial mesh generation.
- Gender and age estimation.
- Glasses, beard, mustache and hat detection.
- Works with regular webcams and smartphone cameras.
- Integration with other software like 3D modelling software or game engines.
- Reasonable prices, flexible licensing and free customer support.
SentiMask provides real-time 3D face tracking and facial expression estimation using video from a regular webcam or smartphone camera. The possible applications of the SentiMask technology include:
- Motion capture for 3D characters' face animation in entertainment applications, like computer games, communication apps etc.;
- Augmented reality applications, like virtual makeup, appearance changes evaluation, etc.
- Facial features analytics for interactive applications, which provide user experience based on person's gender, age and/or facial expression.
The SentiMask 2.0 technology has these capabilities for 3D face tracking applications:
- Real time performance. SentiMask technology performs facial features detection and tracking from live video in real time. The technology provides fast performance on a regular PC or smartphone.
- Facial features estimation. SentiMask algorithm is able to recognize facial pose, landmarks, shape and expression from a video. The advanced facial expression analysis performs 23 different estimations, like checking if left or right eye is closed, left or right eyebrow up or down, jaw moving left or right, etc.
- 3D facial mesh generation. The algorithm reconstructs a 3D facial mesh (wireframe model) from a facial image. A custom texture can be applied to the mesh, or the mesh points can be used as a reference for changing the appearance of an animated character. See the video tutorial for creating custom masks with a graphics editor.
- Gender and age esimation. SentiMask 2.0 algorithm can estimate person's age or gender from the face image.
- Facial attributes detection. SentiMask 2.0 algorithm can detect if a person has beard, mustache, glasses or hat.
- Common camera required. A video for the 3D face model reconstructions can be captured with an off-the-shelf camera or a smartphone. No depth sensors or other advanced hardware needed. The recommendations and specifications contain more detailed information about camera setup.
- Integration with other software. SentiMask generates 3D point cloud, facial rotation angles (roll, pitch, yaw) and estimations of facial expression. This data can be used in a custom application or passed to a 3D modelling software like Blender, as well as game engines. See the video tutorial for using SentiMask with Blender 3D.
