Amazing Sounds for Dream Machine Videos (ElevenLabs)

If you've been looking to enhance your AI-generated videos, making your characters speak and bringing your scenes to life with realistic sound effects, ElevenLabs has just the tool for you. Their innovative Video to Sound Effects feature takes the guesswork out of audio production, allowing you to focus on crafting compelling visuals while AI handles the soundscape.

The company built a demo to show the power of their API. You can upload any video and add AI-generated sound effects.

What's amazing, is that the video actually comes out smaller in size while retaining quality, at least that's been the case with my Dream Machine videos. Upload them in about 2.9 MB mute, get them in1.7MB with a neat sound effect.

How the API Works

1. Extracts 4 frames from the video at 1 second intervals (all client side)
2. Sends the frames and a prompt to GPT-4o to create the custom Text to sound effects prompt
3. Uses the prompt to create a sound effect with the ElevenLabs Text to Sounds Effects API
4. Combines the video and audio on the client side with ffmpeg.wasm for a single file to download

Pretty clever if you ask me!

Try the Demo

A demo is currently hosted on Vercel at 👉 videotosoundeffects.com and anybody can try it out free.

The site offers 4 versions of sound effects, you can download all four or just one that fits best. Below I'm going to show you my Dream Machine generated videos with ElevenLabs generated sounds.

If you're tech savvy, you can use the API on GitHub. You're going to need to obtain your API key from ElevenLabs though.

Dream Machine Videos with ElevenLabs Sounds

Man Walking

A silhouette of a man standing then walking in a dimly lit corridor. Moody atmosphere, fluorescent lights, reflections on the floor. Sense of mystery and suspense.
I've picked the sound variant which resembles noises in prison, kind of creepy. This vid is made using the two Midjourney's images.

Prompt in Midjourney:

Cinematic scene with a mysterious man in a dimly lit hallway. High contrast lighting with strong backlight creating a silhouette effect. Shallow depth of field focusing on the man's face. Dark, moody atmosphere with teal and orange color grading.

Prompt in Dream Machine with Enhance Prompt ON:

man slowly walks away

Then all I had to do was head over to the demo side and upload it there and get my sound effect. But what's more, sound effect includes music as well. I'll show you in my next video.

Steampunk Universe

This steampunk-inspired clockwork universe is generated from a text prompt entirely (see below). As you can hear, ElevenLabs engine found that a melody would fit best for this video. Other options were kind of white noise effects.

Prompt in Dream Machine:

A steampunk-inspired clockwork universe, where planets and stars are gears and cogs, comets are streaks of steam, and constellations form intricate mechanical patterns.

Next up, an underwater sound.

Man Underwater

This is a text to video generation. Luckily, Dream Machine is very good with slow motion effects, which is what I wanted here. The underwater noise is pretty accurate.

Prompt in Dream Machine:

Underwater shot of a fit tanned man with long hair, cinematic, slow motion.

Finally, let's check out the natural sound effect here. It went quite well with my magical but photorealistic flower video...

Large Blooming Flower

This was a text-to-video output, which came out excellent. The sound fits perfectly.

Prompt in Dream Machine with Enhance Prompt ON:

A large blooming bluish-pink flower in hyper-slow motion, shallow depth of field, backlit, with a dreamy atmosphere.

Furious Woman

This one I've decided to make into a bigger video with more images and background music. The full version I'll showcase and explain the process in a separate post. For now, here is one pissed off loud lady! Dream Machine did not need a prompt for this, just uploaded 2 MidJourney's images. Prompt for MidJourney.

So there you have it, folks! ElevenLabs' Video to Sound Effects technology demonstrates the growing potential of AI in content creation. By seamlessly integrating AI-generated audio with visually striking videos, creators can now easily produce more immersive and engaging content. This opens up new possibilities for storytelling and audiovisual experiences across various media platforms.

Last modified 03 July 2024 at 10:25