Maybe encouraging the publication of null results isn't enough--maybe we need a journal devoted to publishing results the study authors find personally annoying.

Replication Crisis [Explained]

Transcript Under the Cut

The paperback edition of The Lost Cause, my nationally bestselling, hopeful solarpunk novel is out this week!

My latest column for Locus Magazine is "Marshmallow Longtermism"; it's a reflection on how conservatives self-mythologize as the standards-bearers for deferred gratification and making hard trade-offs, but are utterly lacking in these traits when it comes to climate change and inequality:

https://locusmag.com/2024/09/cory-doctorow-marshmallow-longtermism/

Conservatives often root our societal ills in a childish impatience, and cast themselves as wise adults who understand that "you can't get something for nothing." Think here of the memes about lazy kids who would rather spend on avocado toast and fancy third-wave coffee rather than paying off their student loans. In this framing, poverty is a consequence of immaturity. To be a functional adult is to be sober in all things: not only does a grownup limit their intoxicant intake to head off hangovers, they also go to the gym to prevent future health problems, they save their discretionary income to cover a down-payment and student loans.

This isn't asceticism, though: it's a mature decision to delay gratification. Avocado toast is a reward for a life well-lived: once you've paid off your mortgage and put your kid through college, then you can have that oat-milk latte. This is just "sound reasoning": every day you fail to pay off your student loan represents another day of compounding interest. Pay off the loan first, and you'll save many avo toasts' worth of interest and your net toast consumption can go way, way up.

Cleaving the world into the patient (the mature, the adult, the wise) and the impatient (the childish, the foolish, the feckless) does important political work. It transforms every societal ill into a personal failing: the prisoner in the dock who stole to survive can be recast as a deficient whose partying on study-nights led to their failure to achieve the grades needed for a merit scholarship, a first-class degree, and a high-paying job.

Dividing the human race into "the wise" and "the foolish" forms an ethical basis for hierarchy. If some of us are born (or raised) for wisdom, then naturally those people should be in charge. Moreover, putting the innately foolish in charge is a recipe for disaster. The political scientist Corey Robin identifies this as the unifying belief common to every kind of conservativism: that some are born to rule, others are born to be ruled over:

https://pluralistic.net/2020/08/01/set-healthy-boundaries/#healthy-populism

This is why conservatives are so affronted by affirmative action, whose premise is that the absence of minorities in the halls of power stems from systemic bias. For conservatives, the fact that people like themselves are running things is evidence of their own virtue and suitability for rule. In conservative canon, the act of shunting aside members of dominant groups to make space for members of disfavored minorities isn't justice, it's dangerous "virtue signaling" that puts the childish and unfit in positions of authority.

Again, this does important political work. If you are ideologically committed to deregulation, and then a giant, deregulated sea-freighter crashes into a bridge, you can avoid any discussion of re-regulating the industry by insisting that we are living in a corrupted age where the unfit are unjustly elevated to positions of authority. That bridge wasn't killed by deregulation – it's demise is the fault of the DEI hire who captained the ship:

https://www.axios.com/local/salt-lake-city/2024/03/26/baltimore-bridge-dei-utah-lawmaker-phil-lyman-misinformation

The idea of a society made up of the patient and wise and the impatient and foolish is as old as Aesop's "The Ant and the Grasshopper," but it acquired a sheen of scientific legitimacy in 1970, with Walter Mischel's legendary "Stanford Marshmallow Experiment":

https://en.wikipedia.org/wiki/Stanford_marshmallow_experiment

In this experiment, kids were left alone in a locked room with a single marshmallow, after being told that they would get two marshmallows in 15 minutes, but only if they waited until them to eat the marshmallow before them. Mischel followed these kids for decades, finding that the kids who delayed gratification and got that second marshmallow did better on every axis – educational attainment, employment, and income. Adult brain-scans of these subjects revealed structural differences between the patient and the impatient.

For many years, the Stanford Marshmallow experiment has been used to validate the cleavage of humanity in the patient and wise and impatient and foolish. Those brain scans were said to reveal the biological basis for thinking of humanity's innate rulers as a superior subspecies, hidden in plain sight, destined to rule.

Then came the "replication crisis," in which numerous bedrock psychological studies from the mid 20th century were re-run by scientists whose fresh vigor disproved and/or complicated the career-defining findings of the giants of behavioral "science." When researchers re-ran Mischel's tests, they discovered an important gloss to his findings. By questioning the kids who ate the marshmallows right away, rather than waiting to get two marshmallows, they discovered that these kids weren't impatient, they were rational.

I know this is a meme, but it got me thinking.

Why is this on the men for not wanting to go to therapy and not on the therapists for making men not want to go to therapy?

Like, I know it is a "haha, men would rather X than go to therapy" thing, but if men would rather die than come see you, maybe, just possibly, you are doing something wrong?

Since my angry post about how science journals tend to work (with a few nice exceptions as I've found out via some lovely Tumblr users) has gotten widespread, I want to clarify something related to mistrust of science:

Almost all individual scientists are trustworthy. I've worked with a bunch of them, including a guy known as "One of the Four Horseman of Chemistry" at my old university (for being a pain to work with), and I've gotten along with them excellently. They have never asked me to p-hack, they have never asked me to manipulate data. Bad scientists who are constantly majorly manipulating their data are a rarity. The two I can think of right off hand are that guy at Standford who got exposed by a student journalist this year and some dude in the fly science area who keeps publishing (and vehemently defending) a bunch of stuff that no one can replicate. Also the good news here is that they do tend to get caught, if not by a student journalist, by their peers, and this is usually because bad scientists who are motivated solely by ego and puffing up the importance of their careers tend to eventually publish something that seems so ground-breaking that other scientists get incredulous, and that experiment bypasses the replication crisis [the phenomenon where experiments aren't repeated by different groups of scientists because they've already been published and won't get republished or contribute to anyone's career]. My science professor friend told me about this happening with a Korean lab that swears they can make superconducting happen at room temperature. That's a big claim. Other scientists were immediately like "Hold up," tried to repeat their methods, and no one can make it work, so that lab has lost a lot of credit in the science community.

Individual scientists, the scientific process, scientific methods, and peer review are generally very trustworthy. Science as a whole is generally very trustworthy.

What you have the right to be suspicious of is 1) Lack of published research on a given thing, 2) Studies with important results that you cannot find any follow-up studies on, especially if they've been widely spread around as "proof" of something.

And THOSE are the problems that come from the fact that very few journals and pretty much no highly prestigious journals publish negative or null results (i.e. "We found nothing interesting") or repeated trials of experiments that have already been published. This disincentivizes reseachers repeating the work of other research groups (the replication crisis). This is not a moral failing on the part of the scientists; they just know that even if they do do repeats of other people's work, it will likely never see the light of day, so there is literally no point in spending years doing that when funding and resources are limited (and that's pretty much always the case in research). Some labs have taken on the responsibility of repeating other people's experiments, but most of that gets shared internally among science groups. This helps improve science from the inside, but because journals won't publish these studies, the public is denied these same confirmations. Scientists aren't hiding things from you; the journal publishing industry is forcing information underground. Then not replicating experiments can result in a single study finding something "interesting" by chance rather than because there's actually a causatory relationship and then becoming accepted science when it's just a fluke: Like the study showing differences in men's vs. women's brains. Not true, just got a statistical difference by chance that one time, which then took a very long time to realize. This situation is once again, not a result of malice or manipulation by the scientists - it's just something that happened by accident.

And since very few people publish null results, scientists probably frequently spend a lot of time accidentally replicating experiments that turned out "boring" and were never published thinking it has simply never been done (which is a waste of time because they won't be published, no one will ever know it's been repeated a billion times, and if it HAS been repeated a billion times, we really can just move on and direct resources to other projects). The professor I know says that scientists generally assume that if the experiment would be easy to set up and has nothing published on it means that it's been tried and found to be uninteresting... which adds the additional problem of experiments never being done because people assume they're too easy.

Once again, notice that at every stage of the game here, the issue is the publishing conventions, not the morals or intentions of the scientists.

Now when you combine these conventions with limited funding that is often allocated based on how prestigious the lab and its work are, which is frequently determined by how many publications and citations it has, WHICH IS IN TURN affected by whether or not you can get your work into high impact-factor journals, there starts to be an incentive to overstate mildly significant results or even mess with the statistics in pursuit of trying to keep finding interesting things to publish, so that you will keep getting funding, so that you can continue your research.

And here I would also like to lay way more blame on the publishing conventions than the scientists (a lot of people quit academia rather than exaggerate results). If the literal survival of a lab was not directly tied to how much and where they publish, there would be no incentive to exaggerate. Scientists that give in to that are also not necessarily horrible and immoral either - they may be banking on their line of research being really important, so they feel the need to overstate initial results so that people will keep giving them money to keep on going. These people also have livelihoods, and "publish publish publish or lose your job" is a really tough spot to put people in then expect them to behave perfectly. Scientists are human, and humans put under immense pressure will break down and make decisions in their own self-interest or in the interest of what their fallible human judgments think is the greater good. That's not malice, that's not bad people trying to fool you, that's not people using science to push an agenda. That is a direct result of the strangle-hold the publishing industry has on science.

Oh and also: There are predatory journals that will publish anything if you pay them a huge amount of money, so now people also have to look up THE JOURNAL to see if it's legit - that's a big thing scientists are doing a good job of warning young scientists about. The legit journals, which sit behind a paywall, also do not pay anything to the scientists they publish. That paywall goes 100% to the publisher and not the authors or their lab.

My good people of tumblr,

We do need to talk about science. Even though I fear a post like this will once again fail to break containment. Still, I do see from time to time posts go around citing scienctific papers - but doing it either in bad faith or not understanding a specific issue.

This issue is that science can be wrong. Even if it was super influential.

I hope that most people on this (usually fairly left leaning) website will generally agree that manmade climate change is real, and that vaccination is indeed something that helps to prevent the spread of disease. This is generally something that the vast majority of science agrees on by now. However, you will be able to find papers - even recent papers in at least medium sized journals - that seek to disprove this.

In these cases we all know the main reason for it: most people trying to disprove manmade climate change are paid off by the fossil lobby. Meanwhile, most people trying to disprove vaxination are some "alternate healthcare" nutjobs far down the crunchy pipeline. If you look at their studies, you usually find a lot of issues with the way they are set up.

Positively we have to say: yes, the majority of science does agree in both cases. Climate Change is real, and vaccination is good for many people. Great. Hooray.

However, here is the thing: just because a lot of scientists agree on something does not make it right. Sadly, there is this phenomen in science where sometimes something is assumed that just makes somewhat makes intuitive sense. And then there is the phenomen where things are argued in bad faith that when argued make inuitive sense.

Several good examples of this are brought in by Naomi Klein in Merchants of Doubt, where she goes through examples of companies investing in science to get specific outcomes and how it shaped the idea of doubting science.

But there are also other examples. One of my favorite is the concept of IQ. The IQ test was not invented with bad intentions (it was a pre-school test in France), but it was wider dispersed as an "intelligence test" by an eugenicist, and then people just ran with it being a test for intelligence until recently. Till this day A LOT OF PSYCH STUDIES are constructed around IQ tests. But that does not make it anymore true that the IQ test tells you a whole lot about intelligence. It might be able to tell you some limited how much talent and training a person has in specific areas (math, logical reasoning, spatial reasoning, memory, and some specific linguistic skills), but that is just a tiny part of what makes intelligence.

Either way: sometimes it just turns out that for one reason or another there is this one thing in science that EVERYONE THOUGHT was a given and that eventually turns out to be wrong - with this meaning that a ton of research that might have build on it might be wrong or at least not fully right or needing to be reconsider. (For example: some studies build on IQ still have interesting observations outside of corellation to this IQ - but still need to be reevaluated in this regard.)

Another good example is the entire thing with fatness. You know, for a long while it was common sense that "obesity causes diabetes". But then fairly recently we have learned "actually, people with the genetic predisposion to develop diabetes are more likely to be fat due to their metabolism working different".

Then there is also the entire replication issue, the Bernoulli fallacy, the automtic bias of many studies (due to WEIRD subjects) and so on and so forth.

Why am I talking about this?

Because at times I see both people I agree with, and people I disagree with bring up studies that have clear issues - and go unchallenged.

You all know the examples where I get annoyed about this: People sharing studies that assume heteronormativity, studies that assume biological monogamy in humans, studies that very clearly leave out alternate explanations, studies that assume that correlation is causation and so on.

And also some of them do rely on research that has been debunked.

Yes, generally there is things we can assume is real: Climate change, vaccination, smoking is bad and so on and so forth. Oh, trans people exist, is also one of those.

This was a machine learning model – the job of the data scientist was, put everything in, see what's significant, of that discard everything that's discriminatory, the rest is your model. Ultimately with twelve astrological signs it's over 50/50 that one will come out significant at 95%. I thought it was elegant. "Astrological signs? Do you believe that?" my boss said. I said it wasn't a question of belief, I was a statistician and was going to follow the numbers rather than letting anyone's preexisting theories about the stars and planets influence the data science. I think he believed that meant I'd agreed to take it out.

Like, the guy literally said "We're very likely to have a false positive here by chance, but since we got one we have to take it seriously. I'm a statistician."

He's fully aware that he's p-hacking and garden-pathing. He's fully aware of the multiple comparisons problem. And then he endorses the conclusion anyway!

Recommendations to meditate, exercise, and pursue nature for happiness rely on weak evidence