Ameteur Bog Mummy

I am currently building a multi-decade chain of papers on computer vision and image generation to have this discussion, and I would like to do so in a moderately calm manner.

Unfortunately if I want to cite a journal article from 1983 in a tumblr post, the copyright holders of this article about image restoration and pixel randomization want $248 per section of excerpted text (<500 words).

This is a 2024 paper by one of the Adobe developers who worked on those compositing tools going over how the tools work. For the record, this model was trained on licensed and public domain images only and the tools are run on device, so the copyright and environmental complaints that people frequently make about AI don't apply to these specific tools.

That paper uses research from this 2015 paper on AI detection of composited images. That paper uses the ImageNet dataset.

ImageNet's dataset is a combination of images sourced from image searches starting in 2009 and description tags generated by Amazon Mechanical Turk workers. The images are not owned by ImageNet, but were scraped from various internet sources in the late 2000s.

That project uses algorithms and processes described in this 2007 paper on the utility of a general image database and image segmentation.

This 2000 paper on image segmentation developed some of the models used in the later paper, based on a collection of 1000 Corel stock images.

That segmentation was based on statistical models in this paper from 1994, which used a small training dataset collected by the researchers.

That 1994 paper made use of the model in this foundational paper from 1984 on predictive pixel selection for algorithm-based image restoration.

That paper helped to refine the boundary-finding methods used in this 1986 paper, which was an improvement on methods from the 1980 Hildreth paper.

Both Hildreth and Canny cite this 1971 memo from the MIT Artificial Intelligence lab, which describes the process of using a computer to find lines in an image.

Personally I like this note from a revised version of that 1971 memo that shows that we are still currently dealing with some of the same problems in computer vision that people were 50 years ago.

"This program has no idea what a reasonable line-drawing should look like when it represents an image of polyhedra. Instead it is very general and will find arbitrary line-drawings. Observing the particular way in which things sometimes go wrong, one quickly comes to the conclusion that higher-level understanding of the scene being analysed could greatly improve the line and vertex creating phase of this program. As thing stand now this understanding comes only after the line-finder has done its work."

Okay so what's my point?

My point is that there is a long chain of models, statistics, and research that, stretching back to the beginning of computer vision, has been centered around figuring out the likelihood that one pixel next to another pixel is black instead of white. The computers have never been very smart, they have never understood context, and the improvements we have made from the line finder in 1971 to "harmonize" in photoshop in 2026 is a very traceable chain of refining how the statistics are calculated.

They hallucinate extra elements, they don't know what shadows are.

Computers are still stupid, they just do math a lot faster than they did in the seventies.

Harmonize is apparently a new "AI" tool available in photoshop that is capable of matching lighting, texture, and other qualities in photos that are composited together. One of the things that the flesh pit guy is currently being dragged for (aside from really seeming to be a pretty tremendous asshole) is using this particular type of AI tool as a time saver on his ongoing art project.

I want to have a conversation about this, by which I mean I want to make some arguments about this.

I'm writing this specifically about the harmonize compositing tool in photoshop (and similar tools like upscaling). This is not about using grok or generating whole images or whole elements of images, this is not about whether or not the flesh pit guy is an asshole. I will grant that he is an asshole and I personally find whole images generated with AI tools like grok to just kind of look bad and be really boring.

Work with me on this, and let's accept the premise that adobe's Firefly is an ethically trained model (up to 5% of the images used to train firefly may be midjourney images that were licensed to adobe as stock images after being generated by midjourney, but adobe pays creators standard licensing fees to train data and does not train on client data), and that the tools are run on-device and do not consume any more resources than creating a 20-layer document in photoshop would.

(Also I acknowledge that training a model uses a lot of power and resources but creating a video game or an animated movie uses power at a much higher level than playing the game or watching the movie on your own machine; i've got to limit the scope somewhere so I can actually ask the question I've got)

Again, we are granting the following before you respond to the poll:

If the model was trained on ethical data and

If the tool runs locally and does not use more power than your computer uses and

The art the tool is used on is a personal, individual project and does not lead to poor treatment of workers

Direct from cellphone, Autofocus

Posterize (edge-finding like the tools in the original linked master’s thesis)

HDR Toning (photorealistic high contrast preset)

Selective Color Adjustment

Glowing Edges (again, another edge-finding tool)

Rendered lighting effects (spot, high intensity, high angle)

Color halftone

Gaussian Blur applied to the color halftone (similar to how AI upscaling works, and is based on the 1986 paper listed above)

Selective color replacement

Cutting backgrounds 100% with tools that use the AI included in my 2012 CS6 license

I think the last two images here and the edge-finding images do a really good job of demonstrating how AI/Machine Learning image processing tools are statistical tools.

The cut-out on the left was done hastily with a magic wand selection tool with a color selection threshold of 20. What that means is that you click on the image and the tool picks all of the pixels that are contiguous with the pixel you clicked on and are within the tolerance. If you broke down the image pixel by pixel you'd be able to get the exact count of what percentage of the image is what color, you could, in theory, remake the image as a barcode of just the proportions of each color using this kind of image analysis. The image is math, the computer sees it as math, the algorithms see it as math, the models see it as math.

The cut-out on the right demonstrates how the "AI computer vision" makes educated guesses (predictions) about the image. That was cut out using a magnetic lasso tool, which you guide around the vague shape of the image with your mouse. The program creates a selection line that follows color variations in the edges of the image that you're cutting out. On a very high contrast background (so if I was standing against a white wall) this can be very accurate. On a lower contrast background, there's more variation as the program guess wrong about which way to move the selection line.

The program doesn't see the photo as a picture of a person being cut out, it sees a collection of pixels of various colors and calculates what the probability is that a pixel of one color is likely to be next to a pixel of another color based on the the difference in color of the pixels.