#test harness

It seems to enable the setup and teardown of VMs through several different providers including vagrant, docker, vmwarefusion. It runs tests on them. Looks like there is a toggle for PuppetEnterprise versus Foss which is type=foss.

Last week and into this week I've spent writing my "test harness".

The purpose of the test harness is to:

Run a bunch of vision processing programs in sequence (a 'chain')

Run these programs in different sequences (multiple 'chains')

Test different possible combinations of values and command line arguments you could give these programs (within each chain!)

Spit out some statistics at the end (Receiver Operating Characteristic curves) and turn them into pretty graphs and display those graphs alongside the final images and the ground truth images to compare them with.

I've nearly nearly almost completed the test harness stuff now. I would say that I thought about it all carefully before I started writing code for it - and that would be a complete lie. Maybe I can pass it off in my writeup as 'evolutionary design'. Luckily, I chose to write it in Python, and design choices in python usually aren't much more effort than "hm, which of these modules do I import?"

I'm quite pleased with the test harness as it is now however. It does some neat stuff. What little design I did was to write the configuration file first, which became my software requirements. The configuration is done through a YAML file. To start off with, you define different sets of input images like this:

Then do the same thing with ground truths (define a set of ground truths for each of the input sets above).

At the end of the config file, you define the programs you want to use in chains, and then the chains themselves:

Here I'm defining the 'simple_test' chain, which runs a single program, 'simple_shadows', which is defined above. 'simple_shadows' does some very dumb thresholding stuff: converting input images to greyscale, any pixel with a brightness value less than an adjustable threshold (set by the '--shadow' command line argument) is marked as shadow. Any pixel brighter than the threshold set by '--object' is marked as an object, and the rest is marked as background (white colour).

The config file defines a list of different parameters to send to the '--shadow' and '--object' arguments of simple_shadows. The test harness builds a tree structure out of these arguments, to get all possible combinations of them. Then it 'flattens' that tree by following every branch of the tree until it hits the end, and all the nodes it's been through on the way become part of the list of arguments sent to the simple_shadows program.

With those two arguments having 8 and 7 different parameters to try respectively, the chain runs simple_test 56 times for every single input image (that has an associated ground truth). Multiply that by the 8 images I have that currently have ground truths, and the number of folders and output images equals a lot.

You would think that that is a bad thing because it's computationally expensive and uses up lots of memory/hard drive space, but no -- the 448 images in 56 folders spat out by the test harness take up a whopping 4MB of space, because they're just greyscale PNG images with 2 or 3 different shades each. And it takes a couple of seconds to run.

When the test harness is done, it runs a program to compare the output images of the chain against my ground truth images, and spits out a bunch of statistics into a CSV file per directory.

Those CSV files are read by another script that uses matplotlib to create graphs. Which in theory should look pretty, but I haven't quite figured out how to play matplotlib yet:

Urgh.

Couple of hours later, I'm getting graphs that look like this which show the results of my two chains so far. Apart from doing simple thresholding, I'm also testing out Otsu's thresholding method (plus different levels of gaussian blurring), which is a little smarter.

The 'simple_test' and 'otsu_test' chains combined (simple thresholding and Otsu's method) are still getting me some pretty crappy graphs:

But Otsu's method on its own at least gets results that are more consistent with ROC graphs:

I'm at a confusing standoff with my code, because I can't tell which part is wrong. ROC curves should have the false positive rate going up fairly consistently with the sensitivity, following that diagonal line. The steeper the slope, the better the algorithm's performance. However, I've got points all over the place, which shouldn't really happen. In the simple thresholding case, I could put it down to it just being really bad performing. Otsu looks more promising (true-positive rate is high, but so is false-positive rate), but those outliers are still bugging me.

Oh well. If all else fails, I can still eyeball the images ;)