Powered by MediaPipe: The Eyes of Motion Games
None of this would be practical without MediaPipe, Google's open-source framework for building real-time multimodal (video, audio, etc.) machine learning pipelines. MediaPipe ships production-grade, pre-trained vision models that run at high frame rates, even on ordinary laptops and phones, and its Web solutions run entirely in the browser using WebAssembly and WebGL acceleration.
MediaPipe is built on the MediaPipe Tasks API, a modern, unified interface that makes state-of-the-art machine learning easy to integrate. Developers simply load a model, pass in a video frame, and read back structured results, such as 33 body landmarks or 21 hand points, within milliseconds. The models are pre-trained and ready to use, yet creators can also train custom classifiers with MediaPipe Model Maker for their own games. Because MediaPipe is cross-platform, the same pipeline can power a webcam motion game today and a mobile app tomorrow. Best of all, everything runs on-device: frames are analyzed locally, which keeps gameplay fast, responsive, and private.
MediaPipe provides the "eyes" of modern motion games. Its main capabilities include:
Hand Tracking
Detects and tracks 21 landmarks per hand, including every finger joint, knuckle, and fingertip, in real time. This enables precise finger-level interaction: pinching, grabbing, counting fingers, drawing in the air, or playing a virtual piano.
Pose Estimation
Tracks 33 skeletal landmarks across the body, from head to ankles. This is what makes full-body motion games possible: jumping jacks, squats, dodge moves, dance routines, and posture-based gameplay are all built on pose tracking.
Face Mesh
Maps 468 points on the human face, capturing expressions, blinks, head orientation, and mouth movement with remarkable fidelity. Face-driven motion games can make your smile, frown, or raised eyebrows part of the action.
Face Detection & Face Landmarks
Fast, lightweight detection that locates faces in a frame, the essential first step for any face-based interaction, even on low-powered devices.
Holistic Tracking
The full stack combined: hands, face, and body tracked simultaneously in one synchronized pipeline. It is ideal for whole-body immersive experiences.
Beyond Vision: Audio and Customization
MediaPipe also supports audio processing, and tools like Model Maker let creators fine-tune and adapt models to their own domains, so your game can even learn to recognize your custom gestures. Add a custom gesture to a motion game and you have a control scheme no one else has: the signature move is yours alone.
For game developers, MediaPipe is a gift: state-of-the-art computer vision, optimized for speed, free and open source, running on-device. It's the engine behind modern webcam motion games.
Everything in this section is documented first-hand by Google. Explore the framework and its vision solutions directly: MediaPipe ↗, Hand Landmarker ↗, Pose Landmarker ↗, and Face Landmarker ↗.