post under development, as of 2022 Jan 07
NDLR: This post reflects my own thinking about motion capture and applications, thanks to fruitful discussions with my friends/colleagues from abroad and from my institution.
Motion capture, also known as MOCAP, is dedicated to collect data enabling to compute the displacement of the object under investigation. This object can be solid rigid, or the complex assembly of different parts, including living subjects. The knowledge of the trajectories/displacement leads to the biomechanical analysis of the subject, or the modification of its appearance for animation purposes. MOCAP can also be considered in computational photography, as the first capture frame is completely different compared to the final result after computational pre and postprocessing. MOCAP data can also be used to drive numeric twin or to generate digital companions… or to magnify and transmit incredible gestures, from artists to top level athletes.
Main Goals 2020
- Practical systems for capturing motion: with or without optics, marker/markerless
- Allow (some) editing of motion
- Can now be used as measuring tools / reference data
- Applications :
- Digital Health in Motion: Gait, Prosthesis, Bionics, Well-Being, Performances, Synchronization …
- Sports and leisures: Optimal position, Interactions, Trajectories …
- VFX (Visual Effects) Cinema: 3D Characters and Avatars
- Cobotics: Fitting models on real 3D motion, Exoskeletons
- e-Games and Virtual Reality
- IHM : Human Machine Interface… and more
Motion has been rendered by Louis and Auguste Lumière with « Sortie de l’Usine Lumière à Lyon », (45s) in 1895…
… but contested by Edison and Dickson with the kinetoscope box showing Dickson’s greeting (10s) moving… but not on screen.
In 1901, Etienne-Jules Marey was freezing the movement from 57 smoke channels hitting various objects, producing the first ever fluid mechanics experimental study.
While the first motion analysis by photography has been realized by Eadweard Muybridge, « The farm », Stanford (1878). 16 « old fashioned » (1 shot, silver halide glass plate) cameras were triggered by a rope while the horse riding in front of white painted wall.
And Sir Stanford kept on going research afterwards and created Stanford University on « the Farm » site.
- This is now possible doing it with color slow motion combined with dedicated 4×4 vehicles with counterbalanced seat and gizmo stabilized camera.
- or even live for bio-mechanics analysis during competitions when placing reflective markers on rider and the horse surrounded by cameras (illustration from Qualisys, Goteborg Sweden).
The first motion cartoon was delivered by Emile Cohl in France (1905)
- Drawing each frame on paper
- Shooting each frame onto negative film => picture with blackboard look.
- Made up of 700 drawings, double-exposed (animated « on twos »)
- Running time ~ 2minutes
Then the rotoscope appeared in 1915 thanks to Max Fleicher.
Animators traced characters over recorded actor’s motion, frame by frame. Rotoscopy was used for Human Characters in Snow White and Seven Dwarfs (1937).
In 1937, the multi-plane camera enabled displacing foreground and background to simulate relief perception, the illusion was perfect at these times.
Creates illusion of depth and true 3D.
Background and foreground moving in opposite directions creates effect of camera rotation.
Animated puppets and robotics
In 1993 Jurassic Park movie was created thanks to animated puppet dinosaurs using stop-motion armatures equipped with sensors measuring joint angles. Computer graphics model are then driven with key frames obtained from armatures.
Virtual visual effects… VFX
Visual effects, using « green background », image synthesis and image computing really appears in the late 1990’s with Matrix (1999). Embedding the multi vision angles high resolution digital photography with crispy images lead to show the same movement with different angles in the same time. In 2021, in « Matrix resurrection », it is even now possible doing it with high speed cameras replacing the single digital shot from the 1990… Face expressions dedicated to the character playing will be introduced in particular with Avatar (J. Cameron, 2009). These integrations require huge graphic processing units and tremendous data storage.
Basic and complex motion capture with passive markers
Photogrammetric observation of passive marker appears in the beginning of 2000’s. Basic retro-reflective markers are placed on the subject and reflect the light shined from the camera to the objective lens of the same camera. If the marker is viewed at least by two camera, its 3D location is generated after intrinsic and extrinsic calibration of the measurement volume. More complex markers are also used in the cinematographic industry to reach easy location of the subject parts, e.g. for the Marvels Iron Man in the late 2010’s.
Active markers just appeared in the 2010’s, by using driven LED on the subject instead of retroreflective markers (e.g. Phasespace). Leds are shining in synchronization with cameras exposure clock.
More recently, Qualisys also developed active (or passive) dedicated markers to ensure rigid body tracking and skeleton generation using only 6 markers clusters (Traqrs, Qualisys, Goteborg, Sweden).
Markerless motion capture
The main problem of the markers setup is to position the markers on the subjects… when possible. For « into the wild » measurements, it is then required to avoid positioning markers and also for non collaborative subjects. Cute systems with low cost cameras, or huge systems with lots of cameras can be used, thanks to live image processing, even on board mobile phones. The user will then choose between accuracy of the 3D results, and the simplicity/cost of the system. Recent paper from our team (Desmarais et al) describes the different algorithms possible to reach the human pose estimation. This research is part of the KeenMT project, to allow 3D estimation with low cost systems in harsh environment, with controlled measurement errors by comparison to golden standard e.g. markers Qualisys MOCAP.
Exemple of video analysis rendering using HumanPose
Carnegie Mellon University (Pittsburgh, Pennsylvania) also developed the digital Dome…using 480 cameras with simultaneous flow to ensure 3D recording…. in a dedicated place.
MOCAP – Systems
post under development, as of 2022 Jan 07
- Leonid Sigal, University of British Columbia, Lecture3, 15-869, Human Motion Modeling and Analysis, Fall 2012
- Kavita Bala, Cornell University, Lecture 32 CS4620/5620 Fall 2012
- Parag Chaudhuri, IIT Bombay, CS 775: Advanced Computer Graphics, Lecture 11 : Motion Capture
- Prof. Vladlen Koltun, Stanford University, Motion Capture CS 448D: Character Animation
- Qualisys, QTM help QAcademy and webinar and demo files, Goteborg Sweden, www.qualisys.com
- Trinoma, 38 Avenue des Cévennes 48000 Villefort, France, https://trinoma.fr/
- Optitrack, NaturalPoint, Inc. DBA OptiTrack, www.optitrack.com
- Vicon Motion Systems Ltd, www.vicon.com
- Phasespace, www.phasespace.com
- MAP Visibility Estimation for Large-Scale Dynamic 3D Reconstruction (Dataset), Hanbyul Joo, Hyun Soo Park, and Yaser Sheikh Carnegie Mellon University
- Desmarais, Y., Mottet, D., Slangen, P. and Montesinos, P., “A review of 3D human pose estimation algorithms for markerless motion capture,” Comput. Vis. Image Underst. 212, 103275 (2021).