#understanding science

Saying “There is no evidence that children transmit the virus or suffer the same mortality as adults” is NOT the same as saying “There is evidence that children do not transmit the virus and do not suffer the same mortality as adults.”

Science Literacy Lesson #2

Here’s the thing about science: it’s trying to prove itself wrong. All the time. And if it fails to prove a hypothesis wrong, then it concludes it must be getting closer to the truth.

Not that it has discovered the truth, just that it is closer. This is why important concepts in science are termed Theories, they are inviting you to test them, to challenge them, to prove them wrong. Calling something a Theory does not mean it’s unproven. The Theory of Evolution is on equal footing as the Theory of Gravity, but you don’t see many people jumping off rooftops because ‘Gravity is just a Theory!’

(Compare this to dogma that actively discourages questioning or challenging it)

But how we gather that evidence is subject to a hierarchy. Not all evidence is equal, not all ‘proof’ is valid and this information gets ranked. This is infuriating for science people talking to non-science people who don’t know why these things are ranked as they are.

So since it’s topical, and in some discussions it’s always topical, here’s one basic Hierarchy of Evidence in ascending order, with examples below the cut:

An anecdote (doesn’t really count tbh)

Expert Opinion

Case Study

Series of Case Studies

Case Control Studies

Randomized Controlled Trials

Double blind randomized controlled trials

Systemic reviews of those controlled trials

Details below the cut

Science Literacy Lesson #4

Let’s say you have two dice, a pair of d20s. 

The Null Hypothesis says there is no (zero) correlation between the numbers rolled by each dice, that they are both completely independent of each other.

You roll them together and they both come up with 20! Super lucky!

Or is it? How many times would you need to roll those dice to prove that they are linked to each other somehow?

The Null Hypothesis is the default assumption in science that two variables (things) just don’t affect each other. It’s the science equivalent of ‘innocent until proven guilty’. You need to prove a link between two things, you don’t assume it’s there.

So in this case we are assuming that the number rolled on the first dice will have no effect whatsoever on the number rolled on the second dice.

Ah! But both dice rolled a 20. So what does that mean?

It means your sample size was small.

You can’t reasonably make a conclusion about these dice based on a single roll. You can’t make a reasonable conclusion based on 5 rolls. The number of times you roll the dice is your sample size (often just called N when we’re discussing the experiment. So five subjects is N=5). If your sample size is unreasonably small, then it weakens your experiment and makes it harder to justify conclusions.

The sample size you require to prove your point varies depending on what you’re testing, but generally larger numbers are always better. An experiment with a sample size of 10,000 is significantly more meaningful than an experiment with a sample size of 12. This is quite simply a numbers game.

(That said, context is also important. If you were publishing a study about the Norther White Rhino, for example, and your sample size was only 2 that may still be significant as you do your best with what you’ve got.)

Now you know, as a reasonable human being or corvid-in-a-trenchcoat, that rolling the dice once doesn’t prove they’re linked somehow, because you know how dice work. A single incident is not any kind of useful evidence. Yet people use single incidents as anecdotes as ‘proof’ all the time! Can you see how frustrating that is?

You rolled the dice once and thought maybe the two dice were linked. You roll the dice 100 times and you will probably see those dice are not linked. The strength of your evidence will only grow as your sample size does.

But what if, hypothetically, you rolled this pair of dice 5 times, and each time you did so they came up with the same number?

Statistics in scientific experiments get complicated and you will completely fall asleep if I try to explain it, but experiments should be looking for a significant difference which is an indication of how likely or unlikely you would be to get the same result if the Null Hypothesis is true.

In this case, rolling your two d20 and them coming up with the same number each time for all 5 times in a row is something like 64 million to 1, if it’s just random chance. That’s a pretty significant difference.

But if you rolled the dice a thousand times and only 5 of those times came up with the same number? Not so significant. Another example of why you need to include your negative results.

Most results report their significant difference to a confidence value of 95 or 99. That means they’re either 95% or 99% sure that their results could not have been achieved by random chance alone. You can’t get to 100 of course, random chance is like that, but the closer you get, the stronger the evidence to reject the Null Hypothesis.

And, look, I’m more of a visual person, so have a graph. If your data produces a confidence value of 95, it means that there is a 95% chance the data didn’t just come from random luck. That the data wouldn’t fall under the blue bit of this curve if the Null Hypothesis is true.

And Confession: I don’t find these numbers fun. If I was running an experiment I’d totally pay a lovely and clever statistician to tell me what sort of sample sizes I would need to achieve my desired confidence interval. I’m not going to run down how you calculate them, just what these numbers are and why they matter.

The larger your sample size, the easier to prove a significant difference with a sigh confidence value. The weaker any of these factors are, the weaker the experiment in general.

(Also why we sometimes don’t use them and why that can be valid)

Science Literacy Lesson #3

Let’s say you have a brand new product that you think is wonderful (a new pet diet, an essential oil treatment, a new human antiviral, etc) and you want to do an experiment to prove it to the world.

You give it to a bunch of test subjects and 60% of them improve in whatever you’re measuring (pet health, anxiety, and patient survival in these examples). That’s great right? Proves it works?

Nope! Completely meaningless without a control group to compare it to.

You might have a hypothesis that your variable (the new food, treatment, medicine, etc) is effective, but you need to prove it against a population without that variable.

The saying in medicine is that about 2/3 of patients will recover no matter what you do. So it’s really, really important to know that a treatment you are recommending, or the variable you are testing in this experiment, performs better than ‘literally nothing’.

That’s where your control group comes in. If you gave 50 participants your experimental variable (food, essential oil, antiviral etc) and 30 improved in whatever you were measuring, but you also have a group of 50 participants that you didn’t give it to and 30 of them also improved, then your product is not as good as it initially looked.

So what might constitute a ‘control’ group?

An ideal control group receives no treatment. For some types of tests that’s not completely feasible as subjects are aware of whether they’re taking a pill or not, so that’s where a placebo is used. Both subjects are taking a pill, but only one is real and the other is known to produce no measurable results.

Using a placebo for the control group is ideal for something like the essential oils example, because you are testing an active product against nothing. You want to know whether that essential oil has any better results than the placebo.

This gets a little harder when testing a diet, because you can’t feed your control group literally nothing. Obviously the group being fed literally nothing for 12 months will be worse off than the group being fed the experimental diet! You have to feed them something, and this would either be a ‘market leader’ diet or an equivalent diet without the one variable you are trying to test (protein level, specific ingredient, etc).

Testing medicines, especially in humans, can get tricky again, especially if you’re testing a treatment for something critical where patients who are untreated are likely to die or suffer. So for these types of experiments the control group will be given the current standard treatment to see how that compares to the experimental one, instead of no treatment at all. You can’t ethically not treat your cancer/pain patients because they were randomly assigned to a control group.

And specifically in medical experimental trials, some will have a type of escape clause where if the tested medicine is proving amazingly, super effective, to the point where it would be definitely cruel to deny the control group that treatment, the control group is sort of disbanded and given the treatment anyway. This is one of the few situations where it’s acceptable to not have a control group. You had one, but they were doing so much worse than the experimental group that it wasn’t fair for them to continue.

Science Literacy Lesson #5

I hope you’re all enjoying these introductions to science literacy, especially if you’ve never come across these terms before or haven’t had a scientific education.I also hope they’re easy enough to understand, though the statistics one is a bit of a PITA.

This one is a lot more conceptual though, so no numbers as such to think about.

You’ve all heard of Placebos before, right?

Placebos are things (a treatment, diet, ritual, etc) that are inactive, meaning they should do basically nothing in regards to the effect you are measuring. Common placebos used in medicine are starch pills or saline injections. Any perceived improvement or result caused by these placebos is termed the Placebo Effect,

The Placebo effect is an important phenomenon in experiments because if you don’t account for it, you may well end up falsely concluding that sugar pills provide pain relief or that homeopathy works!

So, many types of experiments will include a placebo as a type of control group, and you might still see some results from that group, but your experimental group must demonstrate a significant difference from the placebo to be meaningful.

Now, what happens when the people in your experiment know they are taking the experimental treatment? If they know, that can produce a type of placebo effect too, because the subject expect results. People only need to believe they are taking something to report it has effects.

As a side note this can still occur in animal studies, as owners of those animals expect and report on a result subject to the placebo effect.

We attempt to counteract the placebo effect by Blinding.

Blinding means that experimental subjects have no idea whether they are in the placebo/control group, or the experimental group. They don’t get to know whether the pill they’re taking is a placebo or the real deal.

Double Blinding is even better, as it removes the placebo effect from the experimenters. In a double blinded study, the participants do not know whether they are getting the real treatment or the placebo, and neither does the experimenter. The only person who knows is a helpful pharmacist who has a list of participant numbers and who got what, revealed after the study data is collected.

Yes, the pesky placebo effect can influence the experimenters too, especially when the measures results are subjective (eg brighter, happier, shinier coat, clearer skin) and not numbers (eg weight, blood counts).