#image tagging

I used multimodal generative AI to tag my archive of 2,500 unsorted images. It was surprisingly effective.

I’m a digital packrat. Disk space is cheap, so why not save everything? That goes double for things out on the internet, especially those on third party servers, where you can’t be sure they’ll live forever. One of the sites that hasn’t lasted is ffffound!, a pioneering image bookmarking website, which I was lucky enough to be a member of.

Back around 2013 I wrote a quick Ruby web scraper to download my images, and ever since I’ve wondered what to do with the 2,500 or so images. ffffound was deliberately minimal - you got only the URL of the site it was saved from and a page title - so organising them seemed daunting.

A little preview- this is what got pulled out tagged "maps".

The power of AI compels you!

As time went on, I thought about using machine learning to write tags or descriptions, but the process back then involved setting up models, training them yourself, and it all seemed like a lot of work. It's a lot less work now. AI models are cheap (or at least, for the end user, subsidised) and easy to access via APIs, even for multimodal queries.

After some promising quick explorations, I decided to use Google’s Gemini API to try tagging the images, mainly because they already had my billing details in Google Cloud and enabling the service was really easy.

Prototyping and scripting

My usual prototyping flow is opening an iPython shell and going through tutorials; of course, there’s one for Gemini, so I skipped to “Generate text from image and text inputs”, replaced their example image with one of mine, tweaked the prompt - ending up with ‘Generate ten unique, one to three word long, tags for this image. Output them as comma separated, without any additional text’ - and was up and running.

With that working, I moved instead to writing a script. Using the code from the interactive session as a core, I wrapped it in some loops, added a little SQL to persist the tags alongside the images in an existing database, and set it off by passing in a list of files on the command line. (The last step meant I could go from running it on the six files matching images/00\*.jpg up to everything without tweaking the code.) Occasionally it hit rather baffling errors, which weren’t well explained in the tutorial - I’ll cover how I handled them in a follow up post.

You can see the resulting script on GitHub. Running it over the entire set of images took a little while - I think the processing time was a few seconds per image, so I did a few runs of maybe an hour each to get all of them - but it was definitely much quicker than tagging by hand. Were the tags any good, though?

Exploring the results

I coded up a nice web interface so I was able to surf around tags. Using that, I could see what the results were. On the whole? Between fine and great. For example, it turns out I really like maps, with 308 of the 2,580 or so images ending up with the tag ‘map’ which are almost all, if not actual maps, do at least look cartographic in some way.

The vast majority of the most common tags I ended up with were the same way - the tag was generally applicable to all of the images in some way, even if it wasn’t totally obvious at first why. However, it definitely wasn’t perfect. One mistake I noticed was this diagram of roads tagged “rail” - and yet, I can see how a human would have done the same.

Another small criticism? There was a lack of consistency across tags. I can think of a few solutions, including resubmitting the images as a group, making the script batch images together, or adding the most common tags to the prompt so the model can re-use them. (This is probably also a good point to note it might also be interesting to compare results with other multimodal models.)

Finally, there were some odd edge cases to do with colour. I can see why most of these images are tagged ‘red’, but why is the telephone box there? While there do turn out to be specks of red in the diagram at the bottom right, I’d also go with “black and white” myself over “black”, “white”, and “red” as distinct tags.

Worth doing?

(Originally posted here, but copied and pasted with loving care.)

Given circumstances: you’ve been amassing an arsenal of irreverent web-based imagery (since before the term “meme” was collectively used to describe them and let’s split semantic hairs over this later).

The problem: something has jarred a memory of one of those, let’s conservatively estimate, 100,000-odd images and you suddenly feel that burning animal need to find and share that image (and using Google Images would just be cheating).  

So you need to come up with a system of cataloging, sorting, tagging, or otherwise just discriminating between images in a more efficient manner than flipping through endless lists of thumbnails.  There are a few curve balls, of course — these images run all of the gamuts from small to large, from static to Peter-Jackson-movie-length, from run-of-the-mill JPEGs to static and animated GIFs to PNGs with, I don’t know, maybe a TIFF thrown in here and there for good measure.   The category/tagging system needs to be portable and future-proof.  The system needs to run more or less autonomously, as hand-holding through each image run — or even through batches of images from the same source — is too tedious (and inelegant) to entertain.  While I have a few different operating systems at my disposal, the bulk of the effort will be spent on my Windows 10 machine, so it’ll need to operate within that system (in retrospect, if I had a more robust Unix box than my Raspberry Pi, it would’ve been an ideal choice).  The cost should be kept minimal or, for all practical purposes, nil.  And, ideally, you should learn something in the process.

The solution — current solution, really, as this is pretty clearly an evolving process which will benefit from enhancement and refinement as time goes on — is a hybrid of quite a few different free software packages: Darknet, YOLO9000, Tesseract, ImageMagick, and ExifTool.  Individually each is a tool — at least useful, at most invaluable — together they work wonders.

First up, Darknet.  Darknet runs in C, is compatible with CUDA to take advantage of whatever graphics card power you have at hand, and is relatively simple to compile and get running.  …Well, not so much if you’re operating in Win10, even with the enhanced Unix-ish tools Microsoft has been letting drip into the OS.  Compilation has been tricky, and since I have just a sliver of hard disk space left, installing and compiling with CUDA hasn’t been remotely possible (though I should be getting a new hard drive to alleviate this today).  Still, even running purely under the CPU, this implementation of Darknet is a fire-and-forget solution, so an extra 10-30 seconds per image isn’t a deal-breaker.

While Darknet does the actual computation and image processing, YOLO9000 is the engine’s visual dictionary, the information that ties a certain blob of pixels to a label (“shetland pony” or one of 9418 other classifications, whereas Darknet’s default dataset has only 20).  There’s always going to be a trade-off between the size of your label set and performance, but YOLO, and YOLO9000 in particular, seem to inhabit the sweet spot, the balance point between complexity and performance.

It should also be noted that image recognition/identification is only one part of this solution — what do we do when a significant percentage of these images to be catalogued are Twitter/Facebook screenshots?   Ideally, I’d like to have an optical-character recognition feature, one that can take an image of text and turn it into searchable text. And that’s where Tesseract comes in.  Tesseract is the gold standard for open-source OCR, and it really needs no help at all to give you solid, useful results.  With just the slightest bit of image tweaking, you can enhance the quality of the text recognized and returned, which brings us to….

ImageMagick is a suite of image-processing tools fired from the command line, something that doesn’t sound all that useful on first examination, but its utility becomes more evident the more roadblocks you run into during development.  If you want to adjust an image’s levels and contrast (to, say, make an image more readable by Tesseract), ImageMagick makes it happen.  Need to bounce an image between formats, resize it, add text, combine images…?  It’s Photoshop for the command line, with all the utility and waterfall of features that implies.  

ExifTool is another command-line utility with an overwhelming feature set that solves problems you never realized you had.  ExifTool interfaces with a feature embedded in some digital images called the Exif tag — most of the time this is a handy way of extracting data about a digital photo, anything from its dimension or bit depth to even a geotag if you need to know precisely where the image was taken.  For our purposes, it’s just important to know that these Exif tags are (a) only for JPEGs and some TIFFs and (b) can be read, modified, and indexed natively by Windows.  And that’s the last bit of information that ties everything together.

The process that binds all of these systems together is nothing more complicated than a hundred-line (well, 30% is comments) batch file.   Drag and drop up to 100 images (this is a limit I’ve run into and can’t seem to get around yet but I’ve had bigger fish to fry) onto it, and each image will fall into this algorithm:

Is the image a JPEG file?  If not, create a JPEG of the image and use it as the work file.  If the original is an animated GIF file, it pulls off the first frame of the GIF to use as the work file.

If it doesn’t already exist, create a “results” folder in the folder where the original image came from.

Copy the image to a Darknet work directory.  (Darknet has an issue with a relative link in its code that needs fixing — without it, it’s going to cause problems finding label images if you try to run Darknet outside of its home directory.)

Run Darknet — using the YOLO9000 image weights file — on the work image, outputting the detection results to a text file.

Check the “predictions.png” file Darknet should’ve created with the image labels and bounding boxes shown on it.  If it’s the same size as “predictions.png” was before Darknet was run, then Darknet probably crashed, so skip to the next image.

Translate “predictions.png” to a JPEG version in the “results” folder and rename it to match the original image.  

Take the list of identified objects (which Darknet output to that text file when it ran) and embed the results in this new JPEG file under the “XPKeywords” Exif tag so it shows up in Windows.

(Optional, but right now, implemented: take that text file and embed it into the Exif tags of the original image.)

Take the work copy of the image and feed it through Tesseract and into another text file, returning every character Tesseract found (or thought it found) in the image.

Embed that text file into the “XPComment” Exif tag of the file in the “results” folder.

(As above, optional, but right now, implemented: take that text file and embed it into the Exif tags of the original image, too.   Now, regardless of what happens to the “results” folder or the contents therein, you’ve got the original image tagged with the image-recognition items and the OCR text.)

Clean up all of the extraneous text files, just for decluttering’s sake.

Move on to the next image.

And now, if all’s gone well, you should have a whole slew of images with tags and text searchable from the Windows Explorer interface.