#viducpp

Lyrics, design, and editing by S.T.T. Music: The DeepDreamNights Robot Band, Via Suno Visuals made with Vidu q3 ref-2-vid, try it here with code VIDUQ3R2VCPP for 100 free credits.

While I've had the Daem-On characters in my back pocket for awhile, and the song as well, the actual above video was about 36 hours from start to finish, with the usual manual video editing and manipulation mixed with Vidu's new ref-to-video model.

The big improvement with Q2-Pro is the ability to use videos of up to 8 seconds as reference prompts, which can be used to tighten up/fix problems with otherwise good gens, update vids made with older gens, and can be used as a reference for character and camera movement.

Beyond that, Q2-Pro gives a lot more coherence, as several of the Daemes simply would not work properly under previous models.

Looks like I don't have an excuse not to make those shows now.

Vidu just launched two models, Q2-Pro with video-prompting and reference-to-video, and Q3, which is currently text-to-video and image-to-video only, but can do up to 16 seconds with dialog and sound effects.

As the video above is longer than 16 seconds, obviously, editing is involved, as it always is when something is more than a quickie one-off image or clip.

But, this does open up a lot of potential.

The basic process of putting together a clip is simple. You start with an image, and then you script out your prompt.

Long time fans might recognize these dinos and some of their interview quotes from here. So I had character art ready, and my own writing to start with.

Generally, where I started actually wasn't in Q3, but Q2-Pro, where I'd use the reference-to-video options to create starting frames. This is helpful because even with reference the AI has to reinterpret things and there's always some style shift. Using the ref-to-video to create base frames makes it so scene-to-scene those changes aren't random, as you're essentially giving the AI an image that's ready-made for Vidu's use.

Vidu even gives you a built-in method to send the images directly to the prompt with its frame extraction feature, but I recommend to always also download it (without the watermark), because you will want to go back to that image, have the option to edit it, etc.

From there, it's a matter of prompting:

Dynamic anime fight scene, multiple cuts and angles. 2005, traditional hand-drawn cel animation, masterpiece sakuga cinematic animation quality. A green-skinned, skullfaced barbarian queen with a huge scythe duels against an orange-skinned goat-anthro with pink flames for hair and trim fighitng with a golden trident. the goat-anthro wears a sparkling burlesque costume and has a long snake-like tail. SHOT 1: the camera circles the fighters slowly, establishing their relative positions and forms. SHOT 2: cut to closeup on the skullfaced barbarian, her voice is sinister and high pitched, dripping with sarcastic evil SKULLFACED: "A goat that's self-braising, how very quaint!" she is voiced by Sarah Skeletoria. SHOT 3: cut to the goat-anthro, front view, middle-closeup, she takes a defensive stance with her trident. GOAT-ANTHRO: "I won't let you hurt anymore dames, Skullsa Doom!" she has a valleygirl accent and sexy, confident voice. SHOT 4: Goat-anthro charges Skullfaced with her trident forward, Skullfaced barely dodges, the trident rips skullfaced's cloak. flame-themed anime motion background. She is driven by righteous fury. GOAT: "And I won't let you hurt Audi!" SHOT 5: Skullfaced swings her scythe, but goat-anthro backflips away, dodging just in time. her flame-hair forms a long arc as she dodges, enhancing the sense of grace an motion. SKULLFACED: "I don't require your permission, Baph-lessk!" SHOT 6: They continue to duel, evenly matched.

In the form of a script, essentially. You'll notice I used the same reference names for the characters, and that I avoided pronouns. This helps reduce confusion. As does phonetically spelling out stuff that's difficult to pronounce.

There is an important early-series bug that needs to be advised on. If the clip isn't the right length for the required dialog, the characters will often flub or babble lines. This means that you're going to want to time-talk-out your dialog before generating (Q3 is credits-hungry compared to Q2-Pro), or you get what I got here:

Which is the clip going off the rails partway through. If you're trial-and-erroring it will get expensive, but with editing, and by re-dubbing with recorded voices, you'd get around that easily.

Generally, I just generate shorter clips for editing, but for showing off the system, a full 16 second demo seemed best.

Vidu does improve their models over time, and I expect this to be ironed out in the next few weeks, but in the meantime, keep it in mind.

The potential of the tool is immense. Not just because of the coherence and quality of the animation produced, but because it's set up for actual production. A team of a handful of people could produce an entire show on a scrap of a budget with this.

If you want to try it yourself, go to Vidu here and use promo code ROBOTQ3 for 100 free credits.

As a member of Vidu's creative partner program, I was able to participate in the beta and got a lot made during that window, so there will be a lot more about Vidu's new model soon.

Currently, the new Q1 model is a 1080p, 24-frames-per-second, 5 second model that functions on the image-to-video and text-to-video processes, with reference to video hopefully to come sooner than later.

I took an assortment of vaporwave landscapes I'd made in midjourney and used them as start-and-end point images, along with prompts along the following lines:

walk through POV footage, live action film of a surreal forest filled with strange glowing plants and fungus

With img-to-video, minimal prompting works fine for something like this.

The quality is a big jump from version 2, with a lot less blurring and smudging, especially with animation. For a lot of actions setting the movement amplification to large is needed, however, since Q1 is conservative with its movements.

In terms of quirks, start-and-end-frame image-to-video tends to produce a less crisp and less controlled result than just using a start frame.

As there is presently no "extend" function on vidu videos, the motion speed and carryover changes clip-to-clip, so doing a long shot like this one requires adjusting clip speeds, chopping the start and end frames, and the like. Even so how details are interpolated can change shot to shot, requiring a lot of tinkering to minimize the clip-transitions.