For my second try at doing a video entirely powered by AI, I decided to go with a card introduction, and went with one of my all-time favorite @splinterlands cards: Oshuur Constantia. As before, here's the final result before going into my process:
Process
This video was a little bit simpler than the first, as my "storyboard" required less action. I decided to do this because I realized that while Runway is a very powerful AI video tool, it's still quite difficult to create precise and high-quality motion (or at least it's not something I've quite figured out). Also, I saw Runway had a "lip sync" feature and wanted to try it out.
The steps were as follows:
(1) Creating high fidelity images in Midjourney
(2) Generating audio in Runway (for lip sync & non lip sync parts)
(3) Lip syncing audio to image to create a speaking video in Runway
(4) Generating other videos in Runway
(5) Stitching everything together in Clipchamp
Midjourney
As before, I used Midjourney to create my images. This time I skipped ChatGPT and went straight to Midjourney, using the character reference feature to get MJ's help in generating high-fidelity images that closely match the original art.
One new thing I discovered was that using the "--cw" parameter, which determines the character weight, it can make it difficult to make a photorealistic rendering of the character if the reference image is an illustration. This can be balanced by adding a photorealistic image as a secondary "style" reference (the paperclip, instead of the person icon). The "style" weight parameter is "--iw" and goes from 0 to 3, with a default value of 1.
Ultimately however I was able to get the images I wanted by doing a series of experiments and occasionally alternating my character reference image in the prompt. Keeping "--cw 100" (100 is the max for character reference) ensures that MJ maintain consistency with the original image (style, clothing, colors, etc.).
For the image used for the lip sync, I created an image where Oshuur was looking straight at the camera, as my attempts to add lip movements to other images (e.g. where she's slightly looking to the side) failed miserably.
Creating the audio & videos
Runway has a "Generative Audio" feature that allows you to select from a library of voices. You simply provide a script, create generate, and it does the rest. The voices are extremely natural and lifelike.
Lip-syncing is then quite simple, and just requires adding the image you want to animate (in my case the one where Oshuur is staring right into the camera) and selecting the audio file you just created.
After this, I went on to create a few more videos using alternate art for the frames where Oshuur is speaking in the background.
As with the last video, I used Runway's Gen-3 Alpha Turbo model which does a good job of quickly create high fidelity videos from the provided images.
Stitching it together in Clipchamp
As before, I opted to use Clipchamp which is simple and easy to use. With it I'm able to quickly put together the image, video, and audio files together.
Conclusion & Resources
This video took me a total of about 2-2.5 hours. It would've been much less (about an hour) but at the last minute I decided to add extra frames where Oshuur describes her abilities, and also ran into an issue with the audio (described below). Overall though it was easier because there was considerably less motion and a much simpler storyboard.
Resources used:
- Midjourney to create consistent high fidelity images (https://www.midjourney.com/)
- Runway to animate the scenes (https://app.runwayml.com/)
- Clipchamp to stitch the video together
- Youtube music library for royalty-free music (https://www.youtube.com/audiolibrary); Unfortunately while attempting to use music from Pixabay, even though it was marked as "royalty-free", it triggered the copyright check on YouTube, so I had to switch out the track I originally had with one from the Youtube music library (which isn't bad, but is rather limited)
Hope you enjoyed this post and the resulting video. Let me know if you have ideas for other videos you think I should try to do, and let me know if you find this one interesting & useful.