#toxick

I’m a digital artist first.

I have been since before the rise in popularity of generative AI, and I will be… probably forever. Below are some examples of my older art, so you know I'm a "real" artist:

You also get an official internet pinkie-promise from me that I'm an artist. I would not betray the sanctity of the pinkie-promise.

Okay, now about the trolls.

I had this idea for a completely realistic troll generator - a bit like thispersondoesnotexist. I thought it would be cool to see what a diffusion model might produce if it were trained on an Alternian dataset and not a human one!

I went to go see if there were any decent troll models already that I could use as a base, and I noticed a few things about the existing Homestuck models:

They could only really do heavily-stylized characters - any attempt at realism resulted in much lower quality images, and I really wanted realism.

Any models that could do realism seemed to be trained on random people's cosplays, and all of the other models seemed to be trained on random people's fanart. :(

There were only models for specific characters. There was no model that understood the troll species as a whole.

Taking a quick moment to explain what LoRAs are, as well as my stance when it comes to training them! Feel free to skip this part if you already know!

A LoRA is essentially a way to fine-tune an image generation model to be able to produce a certain character, style, or other concept. When people complain that their art was scraped, stolen, and trained on without their consent, they're usually talking about LoRAs. While anyone who posted art online prior to 2022 or so has their art somewhere in the base models, only reasonably famous artists had enough representation in the dataset for their style or characters to be replicated. Everyone else's work just got diluted out. A LoRA on the other hand requires much less data and lets you target that character, style, or concept. I am completely opposed to people doing this to random artists without their consent. I like AI, don't get me wrong, but I am just one artist. I understand that many other artists do not share my views on this technology, and I believe that those artists' wishes should be respected. Even if it's not illegal per se, it's a dick move. Training on your own work, though? Well, that's fair game as far as I'm concerned. Prior to this project, I had made a few small models myself for my art style and OCs, so I knew a little bit about how the training process worked, and I was feeling pretty confident that I could pull this off.

First Dataset:

I began with my own digital paintings, and then I did some 3D renders specifically for this project to beef up the dataset. When I realized I'd run out of troll art, I also did silly edits of my not-troll art so I'd have even more data.

At this point, my hopes were high, but I tried to cram way too many concepts in at once. While it trained, I saw the preview images, and became increasingly convinced that this model was going to be trash.

First Model:

It... Was pretty trash, actually.

Most looked super plasticky like the 3D images - I had so many renders in there that the model simply concluded that that texture was what trolls usually looked like. But some were passable, just missing important features. I was convinced at this point that I could get it working, because it was at least in the right ballpark.

Second Dataset:

I took the highest quality outputs from that dataset and began adjusting them manually. I didn't need a lot of detail yet because I was still teaching the model the basic recipe for a troll.

I also created 3D renders of humans in the hopes that the model would learn "Ohh, okay, this plasticky texture is because it's a render, NOT because it's a troll..."

Second Model:

I had... Clearly overtrained it on the 3D look. My attempts at normalizing the data with a 3D human had backfired spectacularly, and to top it off, they all had same-face syndrome.

It also shed light on a really big problem: It was attempting to give trolls fins at random, even if I didn't prompt for them, even if that troll was not a sea-dweller. This was unacceptable to me, partly because I'm somewhat of a perfectionist, but mostly because I hate having to regenerate something ten times just to get one decent image. I wanted a model that would just... Work.

I realized at this point why people were only making LoRAs for individual characters. Trolls as a species have twelve distinct morphotypes, thirteen if you count mutant and lime separately. Even if we simplify that down to just two categories, land-dwellers and sea-dwellers, the model doesn't learn "A sea-dweller is a kind of troll with fins and a land-dweller is a kind of troll with no fins". The model learns "A sea-dweller is a kind of troll. A land-dweller is a kind of troll. Also, 50% of trolls have fins." For it to actually learn which traits correlate with each other, I was going to need more data.

... A LOT more data.

Most character LoRAs can work with just ten images. The super complex LoRAs, the ones that encode multiple outfits, usually take thirty. The general advice is that anything more than 50 images is overkill. And I was going to need at least 50 per caste to get something even slightly usable...

I cut everything back and trimmed my dataset to just a few castes, deciding that if I could get a passable burgundy (for the warm colours) and a passable teal (for the cool colours), then I could move the colours around to get any non-psionic land-dweller. That just left special training for the goldbloods and violetbloods (since violet is easy enough to shift to fuchsia).

Subsequent Models:

I would spend the next several months making horrible-looking trolls, editing them until they only looked bad, and feeding them back through the model. That new model would produce bad-looking trolls, which I would edit until they looked okay, and train again. The new model would then produce okay-looking trolls, which I could edit until they looked good, and train again...

Subsequent Datasets:

It was also during this process that I accidentally made the teal-blooded troll whose silly transparent glasses frames I would fall in love with and end up using as the face of this blog!

And then I generated... Her.

If you've checked out my blog, you might recognize this sea-dweller as my first ever post here! Indeed, hundreds of images and several training sessions later, this was the first generated troll who actually met my standards without needing significant post-processing.

Final Dataset:

After that, it was smooth sailing. I could use burgundies to make bronze, violets to make fuchsias, goldbloods to make... More goldbloods... And teals to make everyone else. I was able to phase out all of my low-quality images and gradually replace them with better, more detailed ones. I kept my art in there, of course, I'm far too prideful to replace it. And excitingly, I could bring back special concepts that I had tried and failed to implement before, like robotic limbs and blind eyes.

Final Model:

And what I eventually wound up with is the model I use today for this blog! It's still got plenty of peculiarities, of course. Sometimes a troll will get an extra finger. I'll still occasionally spot fins on someone who should not have them. But for the most part, the outputs are pretty solid.

Well... Sometimes.

If I were generating all of these trolls with Midjourney or ChatGPT or some other online AI service, it would be terrible for the environment (and my wallet). This is because it's datacentres that are bad for the environment. Specifically the datacentres owned by big tech companies. They do a lot more than just host AI, which is why their impact is so huge. I am massively in favour of imposing environmental regulations on all datacentres, not just the ones used for AI.

But the nonexistent trolls that I post on this blog? I'm generating all of them on my home computer. It's not connected to the cloud or any paid AI service in any way. Since it runs on my own hardware, it doesn't use any water. It does put some strain on my GPU, but not any moreso than playing a high-end videogame does. My power bill has not changed.