The Rift: Oshuur's Watch | Splinterlands Movie Mock Trailer

Alright, well let's just get this started by saying this was way harder than I thought it would be... It took forever, partly because lots of life things got in the way, but also because the project was very ambitious and I found it difficult to get quite the results I wanted. I'll be the first to say that I think it leaves a lot to be desired, but I wanted to at least share the final result and my process for those who are interested in AI image and video creation.

Without further ado, here's the final product, with some details below it on the process.

Tools used

As before, I used Midjourney for image generation, Runway for video generation, and Pixabay for music and sounds. This time however I chose to try a professional video-editing tool called DaVinci Resolve, which is extremely powerful.

Image generation

The image generation process was similar to before, with Midjourney as the centerpiece.

One thing I struggled with was maintaining a "lifelike" quality to all my images. Somehow it was very difficult when providing certain types of prompts (e.g. getting fantasy creatures in the mix) or when trying to alter characters I'd created (e.g. Oshuur). I found MJ would step away from realism and images started to look more and more like they were part of a high quality videogame rather than a cinematic movie.

I generated over 2000 images in order to get the ones that I wanted, and occasionally gave up on trying to get the perfect image because I felt it simply wasn't possible to tune it just right.

"--cw 100" was my friend. This is a parameter that enables you to set a character reference image so you can use it in other image generations. However I felt it often lacking as well. With that said, it's certainly doable to maintain character consistency.

Video generation

Using Runway for video generation has gotten me to appreciate the difficulties and limitations of AI video at this time. It's certainly far, far, far from perfect. The amount of motion you're able to get, the precision, the speed, even camera control (which is arguably where it does best) is very hard to control and finicky. And a lot of the visual effects are mediocre at best.

I think I generated over 500 videos in order to select the clips I wanted. One thing I quickly realized is that movie trailer clips are typically 5 seconds or less, which means that a 2-minute trailer may have 25-30 scenes (or more, as it's often fewer than 5 seconds). Getting this many clips to line up properly, make sense as a rough "storyline", and be of sufficiently high quality was HARD.

The storyboard I created was quickly thrown out as I realized I needed something considerably simpler or I'd never finish the job.

I used a bit of the "Act One" lip sync (for Oshuur's father, when he talks to her) and though I was planning to use this more, I frankly just found it a bit too hard to work into the script.

Editing with DaVinci Resolve

DaVinci Resolve is an absolutely phenomenal video editing tool. Frankly what I'm happiest about with this project is how much I've learned about video-editing using this tool. And the best part is that it's totally free to use: https://www.blackmagicdesign.com/products/davinciresolve.

Resolve is FULL FEATURED. There is literally nothing this product can't do when it comes to video-editing. It does come with a learning curve however, so it took me a while to figure out hotkeys and what everything in the UI means (not everything's self-explanatory) but once you get it, it's not too difficult to do simple things. I think I uncovered about 1% of what it can do however, and have gained massive appreciation for film editors by getting a peek into what it takes.

What's next

This project took... weeks to do. Of course I wasn't working on it every day (I have a very demanding full time job!). So I worked on it early mornings and evenings when I could, but I got blocked many times as the results weren't as polished as I wanted. It was hard finding the music that fit the theme, finding video clips that synchronized with to the music, getting video renderings that moved accordingly, pairing it all with the right sounds, etc. I'm happy to have a final product, but have a THOUSAND things in there that I wish were better.

I won't be able to make more of these, so I may return to other styles instead such as the style I used for the Doctor Blight mini-movie. That one was significantly easier as there was a single character, single style, less realism - and yet the storyboard was arguably more complex.

I may also try a different video AI tool. While Runway is very powerful, it also left me disappointed many times with the results, so I'd like to see what else is out there.

In any case, I'm happy with the journey as I learned a ton in the process that I can take into future projects. Hopefully if you've read this far you also feel you've gotten some value from these experiments I've done and you'll share your own projects with the community.

Till next time. VONAK!