Generating images with AI, as empowering as it sounds, can sometimes be an arduous process, especially when you have a particular idea and getting something close isn't straightforward. The same applies to generating videos, only with even greater challenges. And here are AI-generated videos' strengths and limitations.
Sora, a video generation AI model, took everyone by surprise when Open AI released it in February. It was the beginning of something new in the AI community that would transform how videos are made, as well as give more power to people with inadequate experience and equipment in professional videography or filmmaking to bring their creative ideas to life.
Shy Kids, a digital production team based in Toronto, was selected by OpenAI to produce promotional short films. They created "air head," a mind-blowing AI-generated short film that gives elaborate insights into what you can achieve with Sora. The truth behind it all, however, is that a lot more than just a well-built prompt fed to Sora was involved.
In reality, many of the things we see—the promotional videos—for these revolutionary tools we see these days often allow one to assume that "what they show us" is totally doable by anyone. What they don't show us is the actual work to bring that stunning demo to life.
Although Sora was used to generate most of what you see in "air head," professional production work is what makes it, say, "real." VFX, editing, colour correction, storyboarding, rotoscoping, and the like all result from a lack of control when working with such a tool as Sora.
Control is still the thing that is the most desirable and also the most elusive at this point. … The closest we could get was just being hyper-descriptive in our prompts. Explaining wardrobe for characters, as well as the type of balloon, was our way around consistency because shot to shot / generation to generation, there isn’t the feature set in place yet for full control over consistency. Patrick Cederberg, a post-production artist at Shy Kids, said.
When shooting videos or making films, you are able to control many parameters, such as clothing colours and the positioning of props. When generating videos, however, you don't have that much control, even though you can describe it to the AI. You mostly have to depend on the AI's imaginative capabilities to deliver what you really want, and that can be laborious as you may need to keep going back and forth.
In some instances, you may even find the AI introducing unwanted elements. In "air head," for example, a producer described how the model would routinely generate a face on the balloon that the main character has for a head, or a string hanging down the front. These had to be removed in post, another time-consuming process, if they couldn’t get the prompt to exclude them. Source
Getting the timing right can be another issue in some cases, as can getting the desired camera movements. And then one may need to do the work themselves to fix these shortcomings by coming up with workarounds in production, just as Shy Kids had to with "air head."
The shot above is an example of what Sora generated and what the end results look like after a lot of work has been done. A yellow baloon could have been requested as the head of the character, but what Sora generated was a red baloon and an actual human head. You'll also see some colour correction was done—for consistency.
At the end of the day, hundreds of videos may be generated, but only a very few could turn out useful.
Something rather interesting about Sora is how it responds to requests that can lead to potential copyright issues. For example, if you ask it to give you a video in the style of a certain artist of film, it will refuse such a request. So if you asked it to generate Spider-Man clips, it wouldn't.
More concerning, though, is how Sora knows what kind of content to not generate or how it figures out what you're trying to do when you want to wiggle your way to make it generate something it should. Does it imply that Sora was trained on such data? Surely, we may never get a direct answer to that, as AI vendors keep information about their training data close to their chest and discreet, for obvious reasons anyway.
For "air head," the Shy Kids made a small behind-the-scenes video describing some of the problems they encountered in making it.
As it stands, developing professional videos with Sora isn't exactly as easy as they make it look. A lot goes into making really good AI-generated videos, and it involves a lot of "behind-the-screen" manual work. Notwithstanding, Sora will get better, and perhaps we will have more control using it with time.
By the way, make earnings with your content on Hive via InLeo while you truly own your account. If you're new, sign up in a few minutes by clicking here! And here's a guide on navigating.