#code coverage example in software testing

Recently i got into rust again. Partially for fun, partially to learn, partially because i always liked rust and ideas behind it, partially because it will soon be of use to me. I've been writing some code in it, learning to fight the borrow checker and the whole data-oriented mindset.

Today i was bored, so i decided to write some simple unit tests for my project, using the built-in features of cargo test and #[cfg(test)] etc.

Turns out i had a MAJOR bug in my program logic which caused my data structures to lose anywhere between 1 entry to 99% of stored data after an insertion operation. It somehow went unnoticed in all prior development, despite the fact that i used the library in some example executables.

The bug couldn't have been found by the compiler, since it was a program logic bug - no memory safety was harmed, all memory operations were perfectly safe. They just happened to drop references to a bunch of data nodes LMAO.

I was very unlikely to find the bug on my own, since - as i've said - i was sure the logic was working. All the programs i wrote seemed to return correct results.

What really struck me is that everyone talks about testing, and i have tried unit testing my projects in the past, be they in C, C++ or whatever, but i never got into the habit of doing it since it usually felt boring and never really provided me with much benefit - a minor bug there, an off by one here, a missing if statement somewhere else.

This, however, was the first time where a bug happened that i was very much unlikely to catch without unit tests, and said bug was actually a major one that would completely nullify usefulness of my project! I have never in the past experienced such direct and significant benefit from writing a few test cases.

It also felt funny in a way, since that's specifically what testing is for. Hell, i work in software testing (sort-of). Despite that, i usually let the message of TDD preachers fall on deaf ears. I guess i will not be doing that anymore lmao.

I felt very glad that i wrote the tests. In fact, fixing that bug also made me rethink the code i was using in that particular place, and realised it was VERY inefficient (i wrote it this way since borrow checker was being mean to me and i was just glad once the code seemed to work lmao). I learned a new standard library function that allowed me to write not only correct, but also more efficient code!

Afterwards i installed llvm-cov cargo addon to check my coverage, and noticed that some of the more complex branches of code were missing from it. This in turn made me write more test cases.

In the end i was so hyped up (by testing, of all things, lmao) that i added CI to my repo and can now enjoy a green checkmark next to my commits and log that says "all tests passing" :)

Never mind that "clean" code can be slow.

Off the top of my head, I could give you several examples of software projects that were deliberately designed to be didactic examples for beginners, but are unreasonably hard to read and difficult to understand, especially for beginners.

Some projects are like that because they are the equivalent of GNU Hello World: They are using all the bells and whistles and and best practices and design patterns and architecture and software development ceremony to demonstrate how to software engineering is supposed to work in the big leagues. There is a lot of validity to that idea. Not every project needs microservices, load balancing, RDBMS and a worker queue, but a project that does need all those things might not be a good "hello, world" example. Not every project needs continuous integration, acceptance testing, unit tests, integration tests, code reviews, an official branching and merging procedure document, and test coverage metrics. Some projects can just be two people who collaborate via git and push to master, with one shell script to run the tests and one shell script to build or deploy the application.

So what about those other projects that aren't like GNU Hello World?

There are projects out there that go out of their way to make the code simple and well-factored to be easier for beginners to grasp, and they fail spectacularly. Instead of having a main() that reads input, does things, and prints the result, these projects define an object-oriented framework. The main file loads the framework, the framework calls the CLI argument parser, which then calls the interactive input reader, which then calls the business logic. All this complexity happens in the name of writing short, easy to understand functions and classes.

None of those things - the parser, the interactive part, the calculation - are in the same file, module, or even directory. They are all strewn about in a large directory hierarchy, and if you don't have an IDE configured to go to the definition of a class with a shortcut, you'll have trouble figuring out what is happening, how, and where.

The smaller you make your functions, the less they do individually. They can still do the same amount of work, but in more places. The smaller you make your classes, the more is-a and as-a relationships you have between classes and objects. The result is not Spaghetti Code, but Ravioli Code: Little enclosed bits floating in sauce, with no obvious connections.

Ravioli Code makes it hard to see what the code actually does, how it does it, and where is does stuff. This is a general problem with code documentation: Do you just document what a function does, do you document how it works, does the documentation include what it should and shouldn't be used for and how to use it? The "how it works" part should be easy to figure out by reading the code, but the more you split up things that don't need splitting up - sometimes over multiple files - the harder you make it to understand what the code actually does just by looking at it.

To put it succinctly: Information hiding and encapsulation can obscure control flow and make it harder to find out how things work.

This is not just a problem for beginner programmers. It's an invisible problem for existing developers and a barrier to entry for new developers, because the existing developers wrote the code and know where everything is. The existing developers also have knowledge about what kinds of types, subclasses, or just special cases exist, might be added in the future, or are out of scope. If there is a limited and known number of cases for a code base to handle, and no plan for downstream users to extend the functionality, then the downside to a "switch" statement is limited, and the upside is the ability to make changes that affect all special cases without the risk of missing a subclass that is hiding somewhere in the code base.

Up until now, I have focused on OOP foundations like polymorphism/encapsulation/inheritance and principles like the single responsibility principle and separation of concerns, mainly because that video by Casey Muratori on the performance cost of "Clean Code" and OOP focused on those. I think these problems can occur in the large just as they do in the small, in distributed software architectures, overly abstract types in functional programming, dependency injection, inversion of control, the model/view/controller pattern, client/server architectures, and similar abstractions.

I've used the term "Documentation Driven Development" in conversation a few times over the years, and people usually seem to just get what I'm talking about. It's like "test driven development", and every bit as disciplined, but for docs instead of tests. I certainly did not invent this concept, and it's very simple.

I also don't think that many folks realize just how serious I am about this. And the repeated truism that bad documentation is the worst thing in open source seems to indicate that it's not being evangelized enough.

Here's the process in a nutshell. This is how I usually write JavaScript modules, but it's also how I approach designing product changes at npm, from small tweaks to entire new product lines.

Before anything else, write down what the thing is in README.md. I use markdown, but any portable plain-text format will do. This takes three parts:

The name of the thing.

The 1-sentence description of what the thing is.

If necessary, a few paragraphs describing the thing.

Example code demonstrating how to use the thing.

Note: the thing doesn't exist yet. You do this before it exists.

Copy the example code into a test file. (Basically just do const t = require('tap') and throw some asserts around it.) For things bigger than a single JavaScript module, you can also copy the various points of the description into user story issues for QA or whatever process you use to verify that the thing you built is actually what the docs say it is.

Now you are ready to write the code that does that thing. Whenever it seems like the usage example was wrong, update it. As it becomes more clear what it is and how it works, keep the docs in sync with the code and the tests. The goal is to maintain 100% test coverage and 100% documentation coverage.

When you want to make a change to the thing, first update the docs and then write the code and the test. It doesn't matter if the test comes along with the code or if you write the code and then the test, but the docs should absolutely be written before any implementation or test code.

If you ever find yourself thinking "It's done, I just need to update the docs", then you've violated this process by even having that thought. Not only is it not done, but you might've done the wrong thing.

It's fine to write some code to try to figure out what the thing should be. But that is experimental scaffolding, and should not be thought of as anything other than useful garbage, like a sketch on a napkin. Maybe it can be turned into something enduring, but that starts with docs.

Writing clear prose about software is an unreasonably effective way to increase productivity by reducing both the frequency and cost of mistakes. Just as with good tests, it helps ensure that your program does what you think it does, and that you've thought it all the way through.

This unreasonable effectiveness of writing things down comes from the fact that a human brain's working memory size is absurdly small. While it seems that we can hold a lot of things in our mind at once, it's only by constantly swapping to long-term storage and ignoring increasing numbers of trees in order to see the forest. A written document does not have that problem. We can fully off-load significant chunks of thought with hardly any data-loss, allowing us to think slower and more carefully while still covering a huge semantic surface.

I don't like using inline comments to generate interface documentation, for the simple reason that my sole focus must be on the user mental model when writing docs, and that should lead the creation of the thing. If I'm writing documentation and implementation at the same time (or even in the same format), I'm naturally inclined to create a "nice implementation", which may or may not be an interface that meshes well with the user's mental model.