#devoqa

This problem illustrates how common, every day Web functionality presents the test designer with a staggeringly large number of execution paths.

Imagine you have a shopping cart which can contain up to 100 different items in varying quantities. There are about 150 different items to choose from.

Now imagine you have been asked to design an automated test that exercises all possible combinations of items, in all valid quantities including zero.

How many variables do we need to think about here? There are 3 variables:

capacity of the cart,

number of different items, and

how many of each item is available

How would you describe the complexity of this test? The number of paths that need be excercised exhibits factorial or O(n!) growth as we put more items into the cart.

How long would you expect this test to take to run? There are over 10^45 ways (that's the sum of 150 choose i as i grows from 1 to 100) to fill the cart with just one of each item. That's just one of each item, we haven't even considered different quantities of items yet.

Assuming one test per microsecond, runtime exceeds the age of the universe (only about 4 * 10^17 seconds). And that still is just a fraction of the number of paths that would be encountered in the the full test case as described above; where not just all possible combinations of items, but also all possible combinations in all valid quantities from 0 to 100, would be tested. I didn't bother to do the math on that one, but it's a lot of paths :)

This problem illustrates how a test scenario that originally was simple, can "explode" into numerous cases through the addition of "just a little bit more" functionality to the application under test!

Imagine a form with two drop-down menus. The first menu has two options and the second menu has three options. You must choose one option from each menu in order to submit the form.

How much time would you expect that it would take you to write and execute a test plan for this form?

2 * 3 = 6

How does the answer change when we add a third drop-down menu that has four options?

2 * 3 * 4 = 24

How does it change if I add two more menus with five and six options, respectively?

2 * 3 * 4 * 5 * 6 = 720

How would your expectations about a manual test plan change when I add another drop-down menu with 7 options?

2 * 3 * 4 * 5 * 6 * 7 = 5040

This test will not scale well because it exhibits factorial growth. Additionally the test's design is bad: why do we need to test every single item on every single menu? Is every item handled by a different subroutine?

Divide and conquer the system by extracting parameters.

Note that I use the term "parameters" here very loosely to mean "any input that the system will accept, in whatever form." A binary file might be considered a "parameter" for testing purposes. Especially if the system you want to test can only consume binary files and has no programmatic interface!

For each parameter so extracted, it should be possible to write several meaningful tests around the interesting values that can be assigned to the the newly extracted parameter.

Software is arbitrarily decomposable so in practice it simply takes time and smart engineers to locate appropriate inputs and outputs for meaningful testing.

This approach is described in gorey detail in Working Effectively With Legacy Code. tl;dr: locate the meaningful inputs and outputs first and only then begin to design specific tests. Because the available inputs and outputs are a hard constraint on test design.

Do Whatever It Takes(tm) to get the system under test.

All systems have inputs and outputs. These can include:

accepting and sending messages over various TCP protocols

making database queries

writing and reading files from the filesystem

And all systems have domain languages including:

API specifications (documented or not)

"obviously intuitive" parameter orderings

expected file formats

Although these hidden languages are rarely elegant, identifying and using them makes the difference between safely adding tests to a system, and not.

In this talk — "Advanced PostMortem Fu and Human Error 101" — from the O'Reilly Velocity Conference, John Allspaw describes (in terms of software systems) the modern view of risk and responsibility. Allspaw draws upon recent research in the life-critical industries, particularly aerospace and medicine.

What has been happening in the "life-critical" fields is a movement away from the idea of "operator error" with irresponsible or malevolent human actors as the eventual "root cause" of a catastrophic failure. Instead, modern researchers propose that there is no root cause when a system fails. Complex systems (such as Web sites) are by definition so complex that their behavior cannot be reliably predicted. Failure is therefore a normal, inevitable consequence of doing business.

In my humble opinion, the field of Software Quality Assurance has a lot to learn from Allspaw's work, particularly the way he has connected Web application architecture with Sidney Dekker's idea of "drift into failure" and with Erik Hollnagel's concept of "efficiency-to-thoroughness tradeoffs."

Part 1 of the talk is embedded above, the rest is available from Safari books

Money quotes below :)

> I don't care how simple your Web site is, it is still a complex, > dynamic system.

> What happens during [a protracted response to an incident] can > infuriate, enthrall, fatigue, mystify or energize the emotions of an > entire company [and] can have lasting effects on your culture and > on your technology. This is what makes scar tissue. This is what > makes (in a very small sense) some sort of post-traumatic stress > disorder.

> Root cause analysis tends to make things look simple… But in > reality there was a lot going on. There were multiple things going > on, people's mindsets were in different places. We work in these > complex systems where there are interactions that can't be told by a > linear narrative.

> Safety is easy: just move the dominoes farther apart. That way they > won't hit each other if one falls over.

> We have to start talking about contributors not causes. There is > no root cause. There is no root cause of your failure… > Finding a root cause of a failure is like finding a root cause of a > success.

> Labeling [an incident] as "human error" is no good… What's > the remediation item? Be more careful? Be more vigilant? Don't do > that again? Where's the graph for carefulness? Human error isn't a > cause. It's an effect. It's an effect of how you've built your > organization.

> Failures are successes gone wrong… Why don't we fail all the > time? Having an answer to that is going to get you to a better place > where human error is concerned.

> We don't build systems that need protection from humans. If you > think that then you've probably watched too much Star Trek. Human > error is an inevitable by-product of strained complex > systems. Because humans vary in their capability, from Sullenberger > all the way to the guy who just caused the site outage.

> There is this idea that the amount of negligence is commensurate > with the severity of the outage… But there's no way that you > can take the severity of the outcome and map it to how good or bad > the action that helped contribute to it. But we still do because we > have this need for accountability.

> When the site goes down, nobody dies.

> Punishment as a deterrent is a losing proposition. Firing people, > docking their pay, benching them… only produce anxiety and > stress which all but guarantee that you are not going to get good > information about failures in the future. And if you don't get good > information about failure in the future, you're screwed.

> —John Allspaw

Liz Keogh on systems thinking.

Money quotes:

You can take apart a bicycle and put it back together again and it will work fine. But you can't do that with a frog. A frog is complex.

An "adaptive complex system" — which is most human systems — is one where as soon as people start looking at it, they start changing the behavior of the system as well. So the system is changing all the time.

Einstein said that the definition of madness is doing the same thing again and again and expecting a different result. In the complex space, doing the same thing again and again and expecting the same result, is mad.

— Liz Keogh

Other points of note:

I recently stumbled across this list of interview questions from back when I was hiring Software Engineers In Test for a team I was leading.

"Technical QA" means a lot of different things to a lot of different people. I found that no matter how I wrote my job postings, I always wound up interviewing candidates with an astoundingly wide range of skills and experience ranging somewhere along the gamut from senior engineer to human-computer interaction researcher.

This is a list of ideas. I'd never ask all of these questions in an interview. I'm also the kind of person who likes to skip around and I wrote this list with that in mind.

However the questions are ordered roughly according to level of technical knowledge required to answer, starting with the least technical.

Here are 41 interview questions to ask when hiring QA Automation Engineers.

Tell me about the first time you ever used a computer?

What was the first code you ever wrote?

When did you become interested in QA? ("Everyone has a different answer." —Stephen Donner)

Why are you interested in QA now?

What blogs do you read to keep up with the QA industry?

Describe the role of QA in the software life cycle: what does QA do and when does it happen?

Why does software have so many bugs?

Would it be a good idea to automate all the tests and dispense entirely with QA?

How does a file system work? what is a file? how does a directory "contain" files?

What happens when you empty the trash?

Imagine you are testing the validation of a new user sign up form for a web site. what kinds of names should the validation reject? what about email addresses?

How do Web sites protect credit cards? from being intercepted in transit? from being stolen off the server?

What is daylight saving time?

How do blind people use the Web?

How good are you at figuring out how to do things using Google?

What's an example of a sophisticated thing you've taught yourself to do using only resources you found on the internet? eg play ukulele, knit a sock, provision an ec2 instance

How good are you at communicating with other people? with engineers?

How good at command line are you?

How would you find a file on a Mac? using only the shell?

How good of a programmer are you?

What programming languages are you comfortable with?

What is html? what is valid html? what is a table-based layout?

What is css? the cascade? the box model?

What is JavaScript?

How can you tell if there is a JavaScript error on a web page?

Imagine there is a JavaScript error — what can you learn about it, how much detail can you provide?

Describe what happens after I type a URL in my browser and hit return. Explain how the browser and server get from that point to a fully loaded web page.

Have you used Charles and/or the Net tab in Firebug/Chrome Inspector?

What are "boundary conditions" aka "edge cases"?

Why do people use "boundary" and "edge" to describe these types of problems? edge of what?

What is a regular expression? what is a search pattern? a wildcard?

How would you make the same change to two different files? to 2000 files?

Have you used an SCM (for instance: git, svn, cvs, vss, ClearCase, Mercurial, Perforce)?

Imagine you found two copies of the same file on your machine. how do you tell if the copies really are the same, or if one has some changes that didn't make it into the other copy?

How do engineers typically use source control branches?

What does it mean to patch a web application?

How does Google search work?

How do password reset emails work?

What are some ways that passwords get stolen? how would you advise me to keep my users' passwords safe?

What is a botnet? what is malware?

How does malware get onto computers?

@noahsussman I don't really understand why you'd ask someone when they first started using a computer. Feels kind of classist.

— Camille Fournier (@skamille) October 24, 2014

@skamille you’re right. A better question would be “why do you love computers.”

— Noah Sussman (@noahsussman) October 24, 2014