100% AI-Generated Locomotion Animations Using a Video-to-Animation Workflow
Idle, walk, sprint, vault, cartwheel, a roll, and a wall run that looks genuinely cursed. None of it was keyframed or pulled from Mixamo. Every motion started as an AI-generated video of a guy on a treadmill.
That's a three.js WebGPU playground where the whole animation set comes from 9 clips made with H3 Max on fal (8 image-to-video, 1 extend-video). The motion is lifted out of those videos and fitted onto a mannequin with custom IK and blending.
How the pipeline works:
- Generate the reference: one image of a person, a short prompt per move, and H3 Max renders the clip
- Extract the motion: GVHMR recovers 3D body motion from the video onto an SMPL-X rig
- Build the playground: Opus 5.5 put the scene and controller together in about 3 prompts, and cut the parkour clip into vault up, jump gap and fall
- Make it speed aware: different variations kick in when standing, walking or running, and longer jumps are stretched procedurally
- Extend with text: the cartwheel, roll and wall run came from extending the video moves with UniMate, an open-weight text-to-animation model that runs locally
According to the creator, a new set of animations takes around 20 minutes, video generation included. A demo and a free open-source parkour controller are supposed to follow.
The rough edges are on screen: cartwheel off a ledge and the character just ragdolls (badly), mantling locks you in place on approach, and the wall run is funny more than usable. Also check licences before shipping, because GVHMR weights and the SMPL-X body model are for non-commercial use.
GVHMR: zju3dv/GVHMR
UniMate: huggingface.co
161 points · 6 comments
Mentions
@RobertBeckebans This yesterday: zju3dv/GVHMR Works really well!
100% AI-Generated Locomotion Animations Using a Video-to-Animation Workflow Idle, walk, sprint, vault, cartwheel, a roll, and a wall run that looks genuinely cursed. None of it was keyframed or pulled from Mixamo. Every motion started as an AI-generated video of a guy on a treadmill. That's a three.js WebGPU playground where the whole animation set comes from 9 clips made with H3 Max on fal (8 image-to-video, 1 extend-video). The motion is lifted out of those videos and fitted onto a mannequin with custom IK and blending. How the pipeline works: - Generate the reference: one image of a person, a short prompt per move, and H3 Max renders the clip - Extract the motion: GVHMR recovers 3D body motion from the video onto an SMPL-X rig - Build the playground: Opus 5.5 put the scene and controller together in about 3 prompts, and cut the parkour clip into vault up, jump gap and fall - Make it speed aware: different variations kick in when standing, walking or running, and longer jumps are stretched procedurally - Extend with text: the cartwheel, roll and wall run came from extending the video moves with UniMate, an open-weight text-to-animation model that runs locally According to the creator, a new set of animations takes around 20 minutes, video generation included. A demo and a free open-source parkour controller are supposed to follow. The rough edges are on screen: cartwheel off a ledge and the character just ragdolls (badly), mantling locks you in place on approach, and the wall run is funny more than usable. Also check licences before shipping, because GVHMR weights and the SMPL-X body model are for non-commercial use. GVHMR: zju3dv/GVHMR UniMate: huggingface.co