One weak link paper -- Andrew Wakefield's fraudulent research that falsely claimed a link between vaccination and autism -- had catastrophic effects. Preventing publication could have prevented the fallout. Science publication is a weak link problem (with some strong link aspects).
It's science funding that's mostly a strong link problem.
There are two separate problems here: understanding the world (strong link) and making sure that everyone believes the right things (weak link). I think a good word for the first one is "science", and the second one is maybe "science communication".
I agree these are separate problems, but saying that the dissemination of research isn't part of scence is kind of a "guns don't kill people" argument...and not even one I think you really believe, since you blew up thanks to an essay calling for the end of journals and peer review. Which, I'm sorry to say, is a perfect example of applying a strong-link solution (deregulation) to a weak-link problem.
I know this is less fun, but I'd argue that figuring out how to keep bad quality work out of publication and the news is much more pressing right now than speeding up the next major scientific breakthrough.
If a science publication publishes an article that gets a lot of attention from journalists, that publication is now involved in science communication to the masses.
If the research can be sensationalized, it will be.
And if the information is false and sensationalized, it may lead to poor outcomes, such as we see today with vaccination. It's why we see a measles outbreak in Texas now.
Posted a reply to the essay and then realized that you already nailed it. We need much more freedom when it comes to doing research (though a little gatekeeping to prevent things like giving radioactive cocktails to expectant mothers is still in order), and a lot more rigor around keeping bad science out of publication.
I'm not surprised the idea comes from Gladwell. It's nuanced, compelling, and almost certainly wrong. It is (and you join in) much much too optimistic about bad science. Such science can have massively bad effects before (if!) it's disproved, with racial science and eugenics just being the most immediate example. Entire societies get organized around these scientific conclusions and people suffer and die as a result. Second, the assumption that outliers are where scientific innovation come from seems attractive but doesn't hold up to scrutiny. For athletic endeavors (a strong link problem) does encouraging weirdness really create better Olympic athletes? (And I see the hidden issue: "outliers" is being used both for "better" and for "weirder." Those are not the same things). Third, I know it's commonplace to talk about scientific stagnation, but I seem to remember that the world managed to create an entirely new set of vaccines in record time within the last half decade. That doesn't seem to me to be particularly stagnant. The world is in the process of creating multiple reusable space vehicles after fifty years of stagnation in crewed space exploration. That doesn't seem stagnant to me.
Perhaps it's about the fields where innovation is happening, hmm?
"The world is (not) in the process of creating multiple reusable space vehicles," but rather, a very small group of individuals, who fall decidedly on the extreme risk end of the spectrum, have set this into motion. These individuals are decidedly "outliers," whose weirdness is at least somewhat analogous to their risk-taking / innovativeness. And by virtue of this enterprise existing at all, it's fair to call them "better."
As for RNA vaccines, I don't know enough about the lineage of RNA and vaccine research (do you?) to make the claim that, again, you credit to the "world," when it is more likely that a select few individuals took on the lion(ess)'s share of the risk / work, whether they were publicly funded, or not.
In the realm of elite athleticism, absolutely "weirdness" makes better olympic athletes on net. You're talking about glory won by the slimmest of margins, and those margins are more often than not gained by risk takers who are constantly tweaking and seeking out new nutrition programs, training regimens, and even equipment upgrades (see "super shoes" in marathoning).
Empirically, the data on declining innovation has been stacking up for decades.
---
Adam, I appreciated your weak / strong link framing. It pairs well with my risk averse / risk normalized paradigm. The world of education is a lot like the world of science.
"a very small group of individuals, who fall decidedly on the extreme risk end of the spectrum, have set this into motion. These individuals are decidedly "outliers," whose weirdness is at least somewhat analogous to their risk-taking / innovativenes"
Jeff Bezos is not an outlier. Elon Musk is not an outlier. Boeing is not an outlier.
" I don't know enough about the lineage of RNA and vaccine research (do you?) to make the claim"
Moderna is not an outlier; Astra-Zeneca is not an outlier; the University of Oxford is not an outlier.
"see "super shoes" in marathoning"
Super shoes were first made by Nike, definitely not an outlier.
"Empirically, the data on declining innovation has been stacking up for decades."
So had the data on the racial inferiority of non-whites. How'd that turn out?
I think there may be a number of things that are not strong or weak link problems. Or even something like education where it is both strong and weak. You want to fix some of the underlying issues to provide a base education, but then you also have many flavors of genius that should be encouraged. Programs like no child left behind may drag on the genius, and gifted student programs often only care about particular types of genius. Maybe for something like that we need to not treat it as 1 problem across all locations, all students, all teachers, all subjects...
Agreed: many things that seem like one problem are in fact many overlapping problems that have different––even opposite!––dynamics. The right solution to one might be the wrong solution to another.
An interesting mental framework with possible applications. It's unfortunate that you immediately use it for a bad example.
Drek science has consequences; how much bad institutional policy has come out of low numbers of scientific papers on subjects like psychological priming, which turn out to be unreplicable (and thus almost certainly not true)? Not to mention that the desire for novel research means that bad research actually drives out good.
(If you're reposting this article, I'm going to feel free to repost my comment on it!)
Deeply ingrained and problematic behaviors and beliefs in the medical and scientific communities are deep-link problems to the extent that they lack an acceptable solution that could easily be developed by those who have no conflict of interest, but those who have the means to make meaningful change have both conflicts and biased that prevent an outside solution from ever being heard, much less taken seriously by those who are in a position to change the system for those who cannot.
Not so fast. Musical composition is a strong-link problem; musical reproduction (sound systems) is a weak-link problem (try getting good sound with a great receiver and rotten speakers).
2-Science is a strong-link system because of the great discoveries. But that does not mean replication can be discarded - it is part of the scientific method, and many "small" results will find their way into textbooks, which cannot be allowed to be filled with junk.
3-I heard of the strong-link idea here in an earlier column, and it is truly brilliant and important.
Re #1: this is a great example of multiple overlapping problems with different (or opposite) dynamics. "Music" isn't one thing, and when you separate it into component parts you'll find that some are strong-link problems and some are weak-link problems.
The timing of this repost lining up with OpenAI’s latest “Deep Research” attention grab is instructive, I think. By promising to automate the mundane and bureaucratic bits of science - the summaries, the citation hunting, the elucidation of where your research fits into fine grain of the literature, OpenAI is implicitly treating science as a weak-link problem. These things can only ever improve scientific literature that has little novel to offer, because by definition they work with the stuff that’s already been published!
Science doesn’t advance by saturating already-overworked journals with automated mediocrity; it advances by observing new things about the world- something a language model simply can’t do.
I wonder if "teaching quality" in education is a strong link or weak link problem. It seems like both - you need to remove the bad teachers but also support/facilitate the great ones. Could this be why it is such a difficult thing to manage?
I'm not sure what the implied solution is. As others have already pointed out, not-so-great science is reducing the amount of funding available to great science much more than in the music business analogy. The system worked fine as long as it was growing - because there were few scientists and the pressure on their career was low. Now that we have humongous number of people hoping for a career in science and that the costs of doing research are much higher at least in most fields, the system is hardly growing any more and risk aversion is not a surprising adaptation to such a context. Do we cut the number of professors ten-fold, so that we can start all over again? Or the number of PhD students?
I really enjoy your work and it is a must-read in my inbox. At the end of this article, I feel that we should both not feel bad about bad science, it is a strong-link system after all, but that this strong link system doesn’t provide for actual strong-link science to occur. This type of dissonance isn’t resolved in the article, which after a bunch of good points kind of left me aimless. And that is because this weak-link/stong-link frame can be used to illuminate the implicit parts of the society of science and make science policy much more weak-link issues.
What you don’t really mention is that the very social structure of current academia is a strong-link system. Massive amounts of resources are provided to those at the very top assuming that the decision-making elite in our society can already identify who/what those strong links are. That they are all too often wrong, which your article provides ample proof of, implies that we should be treating this like a weak-link problem as we don’t know who the strong links are until after their work is done. Therefore anybody in the system could provide some valuable insight.
Our current system implies that social competition is how you find the best science. I use social competition because it is more accurate to reduce these decisions to their base social character, as retraction and replication issues in current science diminish these competitions' claim to some higher value. The raw numbers of professors vs postdocs vs grad students requires this strong-link competitive system. Keep in mind that such drastic competition leaves plenty of plausible deniability, if not excuses, for harassment and discrimination of the people lower down in the system. Modifying the system away from this means producing some big and blunt changes which you may not have wanted to delve into.
Furthermore, some science is more on the weak-link, or at least medium-link spectrum. Taxonomy for example, beyond issues of dividers vs uniters, people doing bad research or just making things up creates real problems when organizing what are different species and their ensuing taxonomy, phylogeny, and potential neat-things about them. Taxonomy relies on a compendium of work going back to Linnaeus. Your idea of science really relies upon that science that can take advantage of ‘ah-ha’ moments, and though psychology may be in need of these ‘ah-ha’ moments plenty of other sciences are not necessarily that way. Like the number of species out there and how to identify them really kickstarts the capacity for people to do empirical research on them. Some parts of science are simply complex, meaning there is not some kind of ‘ah-ha’ simplicity that is going to make the actually useful parts of the knowledge of a grand collection of species cognizable into a single 3 minute idea.
Finally, your own previous work directly states that we should do some large high quality studies on things like flossing and sunscreen. I think it is a great point. What you argue for isn’t some kind of strong-link approach which would spit a bunch of small studies at the problem and see which ones have an ‘ah-ha’ moment, but to run one big study, which is actually done well and completely. This is because there is no ‘ah-ha’ moment to be found, save for the insights of simply a large and therefore complex to implement study. The very point of your proposal here is to have the type of weak-link care for detail across the whole study so that the final results can be trusted and not passed off as ‘some additional study of dubious impact.’
Imagine the government being in charge of deciding whether every single song ever recorded was in tune. I think that’s a non-problem because there’s a zero chance of it happening. I really enjoy your analysis and perhaps my approach is my weak link problem: I tend to believe that more problems than not, and overwhelmingly so, are weak-link problems. Humans actually need very little for survival and beyond that, most problems are weak-link issues. Once air, reasonably potable water, and shelter with some form of self defense are established, the other “problems” that we either know are inevitable, are reasonably anticipated, or should be anticipated be by all people of average or higher intelligence (more than half of human beings), are weak-link, and thus solvable and surmountable.
One weak link paper -- Andrew Wakefield's fraudulent research that falsely claimed a link between vaccination and autism -- had catastrophic effects. Preventing publication could have prevented the fallout. Science publication is a weak link problem (with some strong link aspects).
It's science funding that's mostly a strong link problem.
There are two separate problems here: understanding the world (strong link) and making sure that everyone believes the right things (weak link). I think a good word for the first one is "science", and the second one is maybe "science communication".
I agree these are separate problems, but saying that the dissemination of research isn't part of scence is kind of a "guns don't kill people" argument...and not even one I think you really believe, since you blew up thanks to an essay calling for the end of journals and peer review. Which, I'm sorry to say, is a perfect example of applying a strong-link solution (deregulation) to a weak-link problem.
I know this is less fun, but I'd argue that figuring out how to keep bad quality work out of publication and the news is much more pressing right now than speeding up the next major scientific breakthrough.
I think this is correct if you characterize peer reviewing publications as science communication.
If a science publication publishes an article that gets a lot of attention from journalists, that publication is now involved in science communication to the masses.
If the research can be sensationalized, it will be.
And if the information is false and sensationalized, it may lead to poor outcomes, such as we see today with vaccination. It's why we see a measles outbreak in Texas now.
Which means replication is essential - part of the scientific method.
Posted a reply to the essay and then realized that you already nailed it. We need much more freedom when it comes to doing research (though a little gatekeeping to prevent things like giving radioactive cocktails to expectant mothers is still in order), and a lot more rigor around keeping bad science out of publication.
i thought...
the only two kinds of problems in the world are Diploma Problem vs Toothbrushing problem (https://www.experimental-history.com/p/so-you-wanna-de-bog-yourself)
or they are the Well-defined problems vs Poorly defined problems (https://www.experimental-history.com/p/why-arent-smart-people-happier)
man you are a legend Adam.
There are only three kinds of two kinds of problems, or possibly more
Of couse more. More pls.
I'm not surprised the idea comes from Gladwell. It's nuanced, compelling, and almost certainly wrong. It is (and you join in) much much too optimistic about bad science. Such science can have massively bad effects before (if!) it's disproved, with racial science and eugenics just being the most immediate example. Entire societies get organized around these scientific conclusions and people suffer and die as a result. Second, the assumption that outliers are where scientific innovation come from seems attractive but doesn't hold up to scrutiny. For athletic endeavors (a strong link problem) does encouraging weirdness really create better Olympic athletes? (And I see the hidden issue: "outliers" is being used both for "better" and for "weirder." Those are not the same things). Third, I know it's commonplace to talk about scientific stagnation, but I seem to remember that the world managed to create an entirely new set of vaccines in record time within the last half decade. That doesn't seem to me to be particularly stagnant. The world is in the process of creating multiple reusable space vehicles after fifty years of stagnation in crewed space exploration. That doesn't seem stagnant to me.
Perhaps it's about the fields where innovation is happening, hmm?
I think you're a bit off on this one, David.
"The world is (not) in the process of creating multiple reusable space vehicles," but rather, a very small group of individuals, who fall decidedly on the extreme risk end of the spectrum, have set this into motion. These individuals are decidedly "outliers," whose weirdness is at least somewhat analogous to their risk-taking / innovativeness. And by virtue of this enterprise existing at all, it's fair to call them "better."
As for RNA vaccines, I don't know enough about the lineage of RNA and vaccine research (do you?) to make the claim that, again, you credit to the "world," when it is more likely that a select few individuals took on the lion(ess)'s share of the risk / work, whether they were publicly funded, or not.
In the realm of elite athleticism, absolutely "weirdness" makes better olympic athletes on net. You're talking about glory won by the slimmest of margins, and those margins are more often than not gained by risk takers who are constantly tweaking and seeking out new nutrition programs, training regimens, and even equipment upgrades (see "super shoes" in marathoning).
Empirically, the data on declining innovation has been stacking up for decades.
---
Adam, I appreciated your weak / strong link framing. It pairs well with my risk averse / risk normalized paradigm. The world of education is a lot like the world of science.
"a very small group of individuals, who fall decidedly on the extreme risk end of the spectrum, have set this into motion. These individuals are decidedly "outliers," whose weirdness is at least somewhat analogous to their risk-taking / innovativenes"
Jeff Bezos is not an outlier. Elon Musk is not an outlier. Boeing is not an outlier.
" I don't know enough about the lineage of RNA and vaccine research (do you?) to make the claim"
Moderna is not an outlier; Astra-Zeneca is not an outlier; the University of Oxford is not an outlier.
"see "super shoes" in marathoning"
Super shoes were first made by Nike, definitely not an outlier.
"Empirically, the data on declining innovation has been stacking up for decades."
So had the data on the racial inferiority of non-whites. How'd that turn out?
This was the first thing I read from substack btw
Is improv a strong-link or weak-link problem?
Weak-link when you're starting out, strong-link when you're experienced
Another compelling and useful concept!
Needs better name though.
How about “NO vs GO” problem?
(“Never” Outlier vs “Great” Outlier)
Split Seam vs A-team?
I think there may be a number of things that are not strong or weak link problems. Or even something like education where it is both strong and weak. You want to fix some of the underlying issues to provide a base education, but then you also have many flavors of genius that should be encouraged. Programs like no child left behind may drag on the genius, and gifted student programs often only care about particular types of genius. Maybe for something like that we need to not treat it as 1 problem across all locations, all students, all teachers, all subjects...
Agreed: many things that seem like one problem are in fact many overlapping problems that have different––even opposite!––dynamics. The right solution to one might be the wrong solution to another.
These were my thoughts as I was reading as well. The binary thinking of this kind of theory doesn't apply.
An interesting mental framework with possible applications. It's unfortunate that you immediately use it for a bad example.
Drek science has consequences; how much bad institutional policy has come out of low numbers of scientific papers on subjects like psychological priming, which turn out to be unreplicable (and thus almost certainly not true)? Not to mention that the desire for novel research means that bad research actually drives out good.
(If you're reposting this article, I'm going to feel free to repost my comment on it!)
Great stimulus for thought, but I disagree in one particular. I think for science as it becomes applied the problem is both strong and weak.
You have pointed out the strong link issues
Weak link issues: not just fraud like the autism vaxx thing as mentioned, but the whole issue of eg medical reversal. https://pmc.ncbi.nlm.nih.gov/articles/PMC3238324/
Strong link dominance only exists if the weak stuff is ignored but it often is not!
Deeply ingrained and problematic behaviors and beliefs in the medical and scientific communities are deep-link problems to the extent that they lack an acceptable solution that could easily be developed by those who have no conflict of interest, but those who have the means to make meaningful change have both conflicts and biased that prevent an outside solution from ever being heard, much less taken seriously by those who are in a position to change the system for those who cannot.
1-"Because music is a strong-link problem"
Not so fast. Musical composition is a strong-link problem; musical reproduction (sound systems) is a weak-link problem (try getting good sound with a great receiver and rotten speakers).
2-Science is a strong-link system because of the great discoveries. But that does not mean replication can be discarded - it is part of the scientific method, and many "small" results will find their way into textbooks, which cannot be allowed to be filled with junk.
3-I heard of the strong-link idea here in an earlier column, and it is truly brilliant and important.
Re #1: this is a great example of multiple overlapping problems with different (or opposite) dynamics. "Music" isn't one thing, and when you separate it into component parts you'll find that some are strong-link problems and some are weak-link problems.
Seems like random "useless" government studies could be positive.
The timing of this repost lining up with OpenAI’s latest “Deep Research” attention grab is instructive, I think. By promising to automate the mundane and bureaucratic bits of science - the summaries, the citation hunting, the elucidation of where your research fits into fine grain of the literature, OpenAI is implicitly treating science as a weak-link problem. These things can only ever improve scientific literature that has little novel to offer, because by definition they work with the stuff that’s already been published!
Science doesn’t advance by saturating already-overworked journals with automated mediocrity; it advances by observing new things about the world- something a language model simply can’t do.
This is a great framework - thanks!
I wonder if "teaching quality" in education is a strong link or weak link problem. It seems like both - you need to remove the bad teachers but also support/facilitate the great ones. Could this be why it is such a difficult thing to manage?
I'm not sure what the implied solution is. As others have already pointed out, not-so-great science is reducing the amount of funding available to great science much more than in the music business analogy. The system worked fine as long as it was growing - because there were few scientists and the pressure on their career was low. Now that we have humongous number of people hoping for a career in science and that the costs of doing research are much higher at least in most fields, the system is hardly growing any more and risk aversion is not a surprising adaptation to such a context. Do we cut the number of professors ten-fold, so that we can start all over again? Or the number of PhD students?
I really enjoy your work and it is a must-read in my inbox. At the end of this article, I feel that we should both not feel bad about bad science, it is a strong-link system after all, but that this strong link system doesn’t provide for actual strong-link science to occur. This type of dissonance isn’t resolved in the article, which after a bunch of good points kind of left me aimless. And that is because this weak-link/stong-link frame can be used to illuminate the implicit parts of the society of science and make science policy much more weak-link issues.
What you don’t really mention is that the very social structure of current academia is a strong-link system. Massive amounts of resources are provided to those at the very top assuming that the decision-making elite in our society can already identify who/what those strong links are. That they are all too often wrong, which your article provides ample proof of, implies that we should be treating this like a weak-link problem as we don’t know who the strong links are until after their work is done. Therefore anybody in the system could provide some valuable insight.
Our current system implies that social competition is how you find the best science. I use social competition because it is more accurate to reduce these decisions to their base social character, as retraction and replication issues in current science diminish these competitions' claim to some higher value. The raw numbers of professors vs postdocs vs grad students requires this strong-link competitive system. Keep in mind that such drastic competition leaves plenty of plausible deniability, if not excuses, for harassment and discrimination of the people lower down in the system. Modifying the system away from this means producing some big and blunt changes which you may not have wanted to delve into.
Furthermore, some science is more on the weak-link, or at least medium-link spectrum. Taxonomy for example, beyond issues of dividers vs uniters, people doing bad research or just making things up creates real problems when organizing what are different species and their ensuing taxonomy, phylogeny, and potential neat-things about them. Taxonomy relies on a compendium of work going back to Linnaeus. Your idea of science really relies upon that science that can take advantage of ‘ah-ha’ moments, and though psychology may be in need of these ‘ah-ha’ moments plenty of other sciences are not necessarily that way. Like the number of species out there and how to identify them really kickstarts the capacity for people to do empirical research on them. Some parts of science are simply complex, meaning there is not some kind of ‘ah-ha’ simplicity that is going to make the actually useful parts of the knowledge of a grand collection of species cognizable into a single 3 minute idea.
Finally, your own previous work directly states that we should do some large high quality studies on things like flossing and sunscreen. I think it is a great point. What you argue for isn’t some kind of strong-link approach which would spit a bunch of small studies at the problem and see which ones have an ‘ah-ha’ moment, but to run one big study, which is actually done well and completely. This is because there is no ‘ah-ha’ moment to be found, save for the insights of simply a large and therefore complex to implement study. The very point of your proposal here is to have the type of weak-link care for detail across the whole study so that the final results can be trusted and not passed off as ‘some additional study of dubious impact.’
Imagine the government being in charge of deciding whether every single song ever recorded was in tune. I think that’s a non-problem because there’s a zero chance of it happening. I really enjoy your analysis and perhaps my approach is my weak link problem: I tend to believe that more problems than not, and overwhelmingly so, are weak-link problems. Humans actually need very little for survival and beyond that, most problems are weak-link issues. Once air, reasonably potable water, and shelter with some form of self defense are established, the other “problems” that we either know are inevitable, are reasonably anticipated, or should be anticipated be by all people of average or higher intelligence (more than half of human beings), are weak-link, and thus solvable and surmountable.