#mutation-testing

Here is a great description of different forms of code coverage. Thanks to Florian for the recap!!

Statement Coverage:

how many portions of the CFG are executed (nodes & edges)

#executed / #total

-> still possible to miss bugs

Branch Coverage:

test all possible branches in control flow (edges)

complete branch coverage implies complete statement coverage

-> still possible to miss bugs

Path Coverage:

test all possible paths (sequence of branches)

complete path coverage implies complete branch coverage

-> not feasible with loops (arbitrary # of paths)

Loop Coverage:

for each loop, test 0, 1, and 1+ iterations

coverage = #loops with 0,1,1+ iterations / #loops * 3s

Data Flow Coverage:

evaluated with DU pairs

coverage = #DU-pairs / used DU-pairs

 Glossary

CFG = control-flow graph

nodes with statements immediately and always executed after each other

nodes are connected by edges if reachable

for example, an if condition statement produces three nodes A (statements before if), B (statements if condition true),C (statements if condition false) and edges between A->B and A->C

DU: definition use pair

defines where variable used at the moment was defined at; each variable has typically many definition use pairs

for example, (2,3) means the variable a used at line 3 was assigned to (hence defined) at line 2

SQLite has written up amazing documentation on how they test their code base. An interesting nugget there was how mutation testing could fail even with 100% branch test coverage. Check it out!