LatentSync: Next-Level Lip Syncing from ByteDance

LatentSync is an innovative tool for creating lifelike lip-synced videos directly from audio input. Unlike old-school methods, it skips complex intermediate steps like 3D modeling or facial landmarks. Instead, it uses latent diffusion models, which focus on delivering high-quality, time-based consistency in every frame.

The system was developed by ByteDance, the company behind TikTok, in partnership with researchers at Beijing Jiaotong University. ByteDance has a strong background in using AI to push the boundaries of video and audio tech, and LatentSync is another step in that direction.

Key Features of LatentSync

Here’s what sets LatentSync apart:

• All-in-One Workflow: It skips intermediate steps, creating more natural and smooth results.
• Smooth Frame Transitions: Uses TREPA (Time Representation Alignment) for clear and consistent lip-syncing across video frames.
• Accuracy That Beats the Rest: Hit 94% accuracy in testing—higher than most existing methods.
• Crisp Visuals: Works in latent space to churn out high-res, detailed videos.
• Powered by Stable Diffusion: Combines the strengths of Stable Diffusion to match audio and visuals precisely.

Real-World Applications

LatentSync’s potential goes beyond fancy tech—it’s practical:

• Dubbing Made Easy: Perfect for syncing voices with video when translating content into other languages.
• Better Digital Avatars: Brings more realistic expressions to virtual characters, making them more engaging in games or on social platforms.
• Sharper Video Calls: Improves how audio and video align during virtual meetings.

Open to Everyone

The code is available on GitHub (inference code and checkpoints), so developers and researchers can dig into it or tweak it for their own needs. It’s open source, which means anyone can take advantage of this powerful tool.

You can also try it on fal: https://fal.ai/models/fal-ai/latentsync

And there's already a ComfyUI wrapper https://github.com/ShmuelRonen/ComfyUI-LatentSyncWrapper

Uploading some of my LatentSync tests here.

LatentSync is a promising leap forward in lip-sync tech. While it’s packed with exciting features, more independent testing is needed to see how well it holds up in everyday use. If its capabilities are confirmed, it could become the new standard for audio-video synthesis tools.

Last modified 05 January 2025 at 11:44