Tora: AI for Precision Motion Control in Videos

Tora, developed by Alibaba, lets you control object movements in videos simply by drawing paths. Ideal for creatives and filmmakers, Tora brings precision to video editing, letting users control each element’s movement.

How Tora Works

1. Trajectory Extractor: Breaks down your drawn paths so the algorithm understands the desired motion.
2. Spatial-Temporal Diffusion Transformer: Builds the video using the processed path data.
3. Motion Guidance Fuser: Adds the motion details from the path directly into the video, ensuring fluid and natural-looking movements.

Recent strides in Diffusion Transformers (DiT) have made high-quality video generation possible, but using them for motion control is still new. Tora introduces the first DiT framework that brings together text, visuals, and movement paths to create customizable video scenes. This setup includes the Trajectory Extractor, the Spatial-Temporal DiT, and the Motion Guidance Fuser. Tora’s setup allows you to control video dynamics with flexible durations, formats, and resolutions. Extensive testing shows Tora’s precision in creating realistic motion that mirrors real-world physics.

Examples of Tora in Action

You can see the video examples live on the website https://ali-videoai.github.io/tora_video/

• Two Roses: Draw a swirling path, describe two roses swaying, and they’ll follow the path naturally.
• Fish on Mars: Draw a path left for a fish, right for Mars’ background, and watch a fish swim across a Martian landscape.

Availability

Tora’s code is open-source and available on GitHub, making it free and easy to use for creatives and developers.

There is a demo but it requires registering with a mobile number https://www.modelscope.cn/studios/xiaoche/Tora

Tora provides creatives, filmmakers, and animators with precise control over video content. It’s open-source and free, available on GitHub, and offers a reliable way to add smooth, controlled movements to any project.