#fantareal

There's been a lot of buzz in r/StableDiffusion about a thing called ControlNET, which lets the user transfer a pose from one image to another. I finally got some time to check it out. I followed the video tutorial here on what to download and how to get started. If you're new to it too and not getting the best results, know you're not alone and check out some results of my unicorn test.

I started with an AI-generated unicorn I liked. For the image with the pose to transfer, I found this photo of a running horse on Freepik then edited it to face the same direction as the prompt image and to eliminate the background wall, which was causing weird effects in my generated images.

MODULE: DEPTH

ControlNet Module: depth ControlNet Model: control_sd15_depth [fef5e48e] This is the "Annotator result" with a "Midas Resolution" of 1173:

A grid I made with Automatic 1111's "X/Y/Z Plot" script that shows the effects of different settings:

MODULE: HED

ControlNet Module: hed ControlNet Model: control_sd15_hed [fef5e48e] This is the "Annotator result" with a "HED Resolution" of 300:

Results:

Effects of changing HED Resolution:

Shared settings:

Prompt: a graceful unicorn with a pastel rainbow curly mane flowing in the wind and slender furry Clydesdale legs standing in a field at sunset, nature photography, 35mm, film grain

Negative prompt: floating horn, five legs, extra legs, bad anatomy, deformed, ugly, malnourished, starving, scrawny, 3D render, big butt, cow, tired eyes, dead eyes, bulky, fat, plastic, toy, doll, CGI, airbrushed, painting, digital art, cartoon, oversaturated shadows, oversaturated, saturated colors, blue legs

Steps: 20 Sampler: Euler a Seed: 1170935863 Size: 512x512 Model hash: 00839a9f17 Model: kkw-FANTAREAL-V1.0 ControlNet Enabled: True ControlNet Guidance Strength: 1

And here's a grid showing the big effect that different seeds can have:

I started with what should've been a simple idea for an AI image-- a tattooed sphynx cat-- but it proved trickier and anticipated. For my initial attempt, I used the "KKW-FANTAREAL V1.0" model. The interface I'm now using is "Automatic-1111".

I was seeing too much questionable anatomy and not enough tattoos.

In general, mixing tattoos and cats mostly got me tattoos OF cats... and whatever the hell that human-armed cat thing is.

After a while, I found the time to try out the "Instruct Pix2Pix" model, which is meant for uploading images and telling it how to change them (ex. "Add sunglasses"). I used my favorite of the previous sphynx images, hoping to quickly tell the computer to slap a cool dragon tattoo on the chest.

Here, I ran into some interesting problems. I'm suspecting a bias towards pictures of humans since trying to use a simple prompt such as "Add a dragon tattoo to the cat's chest" generated plenty of obscene-looking flesh monstrosities featuring the cat AS a tattoo instead of putting tattoos ON the cat. The combination of "chest" and "tattoo" in particular had me scrambling to fill up that negative prompt field to try purging human anatomy from the equation. I tried "shoulder", "under the neck", "below the neck", but that didn't get me the cool chest tats I wanted.  Sometimes the cat's ears mutated into arms too.

These are some of the less horrific instances of cat-as-tattoo from this batch:

Even when the results were more reasonable, I still wasn't getting much of what I wanted:

So, I decided to just go back to FANTAREAL and inpainting. That proved much more effective:

I expanded my inpainting to include more:

Sometimes, some human BS still tried to sneak in...