1. I wouldn't say heliocentrism, germ theory, or CTs are obvious in retrospect. I actually just wrote a whole blog post on how germ theory would be impossible without microscopes (https://trevorklee.substack.com/p/a-medical-thought-experiment), and I don't think anyone reading this blog would be able to prove heliocentrism without Googling how to. Meanwhile, corporate marketing still hasn't figured out RCTs, even though there's a direct financial incentive for them. Also, the concepts (and math) embedded in E=MC^2 are, again, beyond the ken of almost anyone reading this blog.
2. There's a difference between "we're running out of ideas" (which is silly) and "all the low-hanging fruit has already been picked".
3. There's a difference between discoveries in physics, which are supposed to be true for everywhere at all times, and discoveries in psychology, which can't make that assumption (i.e. I'm not sure cavemen abided by the same conversational conventions we do).
4. I think bringing in Thomas Kuhn earlier would have helped. There's always a flood of "easy" papers at the beginning of a paradigm shift, as the basic elements of the paradigm get fleshed out and improved. This also is often true when new instruments get developed.
1. I think this is beyond the level that people get to when they think ideas are getting harder to find. There are two illusions here. One is that people probably think they understand these ideas better than they do, and even if they know they'd have to do some research, they underestimate just how much. The other is that they underestimate how hard it was for the original thinkers to come up with this stuff. Germs seem obvious because we learn about them from the time we're children, and it seems silly that anybody ever doubted their existence. It's just hard for us to put ourselves in a world where the most educated people could think really hard and still be totally incredulous that tiny organisms could cause disease.
2. They're similar if you believe the latter strongly enough. If ideas really do get harder to find, eventually it becomes unlikely that anybody ever discovers one, so practically it feels like running out.
3. I think this seems truer than it is because physics has developed much farther than psychology. Physicists had to figure out variations across contexts as well, like why very small systems seemed to act differently. Psychologists might have a harder problem, and I have no idea how they'll solve it, but I'm optimistic about it. For now, we are still cavemen trying to explain why a giant fireball appears in the sky every day.
4. It's interesting that this hasn't happened in psychology. We've piled up a bunch of replication failures, but nobody seems to be stepping up to provide an alternative. I think this is because we don't really have a strong paradigm; a very weak paradigm can handle lots of problems without failing, because there's nothing really to argue against.
The article "ideas are harder to find" is based not on ideas but on the final economic implementation of ideas that impact the economy (where they get the total factor productivity numbers). That is Ideas + economic feasibility + resource availability ($$$) + permissions (regulatory) + defeating political attacks from those who fear or are threatened by the innovation. Any one of these series steps can be the rate-limiting step. Even if you have an infinite number of scientific ideas, the rate of economic growth and apparent innovation will be limited by one of the other factors.
The authors also misuse R&D spending by companies to obtain progress as a metric for saying that R&D productivity is declining without noting that the actual areas being researched are changing over time. In an evolving technology area like chips or GMO, the researchers borrow from other areas of technology, which at that moment, are more advance than their new area. The cost of the R&D in that spill-in area of technology is on someone else's books. For example, initial semi-conductor manufacturers used clean rooms from hospital suppliers, vaccum systems from the space program, everything from saws to grinding/polishing systems from metallurgy, etc. When the feature size of chips evolved to the point where a virus looms as a mountain of impurity, they were compelled to do R&D on clean rooms. That technology advance then spilled-back to hospital clean room technology. The required R&D was then on the Chip companies books, being counted as declining R&D productivity.
The spill-in and spill-out of R&D benefits for modern multi-factor innovations is very complex and results in areas like biotech obtaining a huge spill-in of technology from computer/chip area. For analyzing DNA by looking at a color of light for each letter being added (only 4 letters) onto the sample DNA they needed holes in the 70 nm range that utilized a spill-in from semi-conductor technology. Some of these biotechnology areas are growing even faster than Moore's Law as they adapt the R&D from both semi-conductors and computers.
Meanwhile, the new semi-conductors with 2 nm line widths need very hard UV light close to soft X ray light sources. Light sources use a small drop of tin hit with a laser beam to a high temperaure plasma state, but this technology probably has a massive R&D spill-in from the billions spent on inertial confinement fusion, where a small pellet containing hydrogen isotopes is compressed to temperatures of our sun for fusion, also using converging laser beans. A few billion in R&D spill-in is good.
The "low hanging fruit" mental model or even the "tree of knowledge" model don't work very well. The very low hanging centuries old fruit of Maxwell's equations, General Relativity, and the Schrödinger equation has long gone to seed and produced trees whose branches meld with every branch of the founding tree of human knowledge they cross. When every point of crossing branches effectively starts a new tree, the tree model ceases making sense, and network models more useful images. Your i-phone utilizes Maxwell's equations, General Realtivity and Quantium mechanics to make it work. Those are not elements of Apple's research budgets.
As the overall "knowledge network" grows, more possible ideas and technologies are created at a rate much faster than the economies in the deleveloped world economies (probably up around 10%/yr). Clearly, we can conclude that the slow step controlling economic growth is not lack of "Ideas". Science has a meaning of the word "idea" that excludes economic/political factors outside the idea itself.
The regulatory and permission areas are expanding rapidly and getting slower. Imagine, if Apple had to build the i-Phone in California and needed 100,000 employees in a new facility utilizing hazardous materials, highways, etc. This is a state where trying to get permission for desalinization of seawater, in water-short Southern California, has proved impossible after decades of trying and tens of millions of dollars spent in the effort. Meanwhile Israel now obtains 50% of their water supply from desalization using RO technology, that was developed at UCLA in the '50s. Good ideas abound. It is not a lack of ideas, but obtaining permission to make use of the ideas which is the sadly limiting factor which is preventing the United States from fulfilling its potential, to the detriment not only of our country, but of the world.
To me, the whole "idea shortage" concept is a convenient excuse to blame innovation stagnation on something other than the true "rate controlling" step, which has been put into place by government regulation. With the web of knowledge now being so large, people like the authors of this article (Bloom, Jones, Van Reenen, and Webb) may not have the basic knowledge required to see the real complexity and only see what is in the sub-network of economics. Just viewing the economics (accounting numbers), we can get accounting artifacts that say R&D productivity is decreasing when the R&D is just covering more breadth.
The article "ideas are harder to find" is based not on ideas but on the final economic implementation of ideas that impact the economy (where they get the total factor productivity numbers). That is Ideas + economic feasibility + resource availability ($$$) + permissions (regulatory) + defeating political attacks from those who fear or are threatened by the innovation. Any one of these series steps can be the rate-limiting step. Even if you have an infinite number of scientific ideas, the rate of economic growth and apparent innovation will be limited by one of the other factors.
The authors also misuse R&D spending by companies to obtain progress as a metric for saying that R&D productivity is declining without noting that the actual areas being researched are changing over time. In an evolving technology area like chips or GMO, the researchers borrow from other areas of technology, which at that moment, are more advance than their new area. The cost of the R&D in that spill-in area of technology is on someone else's books. For example, initial semi-conductor manufacturers used clean rooms from hospital suppliers, vaccum systems from the space program, everything from saws to grinding/polishing systems from metallurgy, etc. When the feature size of chips evolved to the point where a virus looms as a mountain of impurity, they were compelled to do R&D on clean rooms. That technology advance then spilled-back to hospital clean room technology. The required R&D was then on the Chip companies books, being counted as declining R&D productivity.
The spill-in and spill-out of R&D benefits for modern multi-factor innovations is very complex and results in areas like biotech obtaining a huge spill-in of technology from computer/chip area. For analyzing DNA by looking at a color of light for each letter being added (only 4 letters) onto the sample DNA they needed holes in the 70 µm range that utilized a spill-in from semi-conductor technology. Some of these biotechnology areas are growing even faster than Moore's Law as they adapt the R&D from both semi-conductors and computers.
Meanwhile, the new semi-conductors with 2µ line widths need very hard UV light close to soft X ray light sources. Light sources use a small drop of tin hit with a laser beam to a high temperaure plasma state, but this technology probably has a massive R&D spill-in from the billions spent on inertial confinement fusion, where a small pellet containing hydrogen isotopes is compressed to temperatures of our sun for fusion, also using converging laser beans. A few billion in R&D spill-in is good.
The "low hanging fruit" mental model or even the "tree of knowledge" model don't work very well. The very low hanging centuries old fruit of Maxwell's equations, General Relativity, and the Schrödinger equation has long gone to seed and produced trees whose branches meld with every branch of the founding tree of human knowledge they cross. When every point of crossing branches effectively starts a new tree, the tree model ceases making sense, and network models more useful images. Your i-phone utilizes Maxwell's equations, General Realtivity and Quantium mechanics to make it work. Those are not elements of Apple's research budgets.
As the overall "knowledge network" grows, more possible ideas and technologies are created at a rate much faster than the economies in the developed world economies (probably up around 10%/yr). Clearly, we can conclude that the slow step controlling economic growth is not lack of "Ideas". Science has a meaning of the word "idea" that excludes economic/political factors outside the idea itself.
The regulatory and permission areas are expanding rapidly and getting slower. Imagine, if Apple had to build the i-Phone in California and needed 100,000 employees in a new facility utilizing hazardous materials, highways, etc. This is a state where trying to get permission for desalinization of seawater, in water-short Southern California, has proved impossible after decades of trying and tens of millions of dollars spent in the effort. Meanwhile Israel now obtains 50% of their water supply from desalinization using RO technology, that was developed at UCLA in the '50s. Good ideas abound. It is not a lack of ideas, but obtaining permission to make use of the ideas which is the sadly limiting factor which is preventing the United States from fulfilling its potential, to the detriment not only of our country, but of the world.
To me, the whole "idea shortage" concept is a convenient excuse to blame innovation stagnation on something other than the true "rate controlling" step, which has been put into place by government regulation. With the web of knowledge now being so large, people like the authors of this article (Bloom, Jones, Van Reenen, and Webb) may not have the basic knowledge required to see the real complexity and only see what is in the sub-network of economics. Just viewing the economics (accounting numbers), we can get accounting artifacts that say R&D productivity is decreasing when the R&D is just covering more breadth.
Thanks for this great comment! I agree that, even if whatever measures of scientific output were indeed slowing down, it doesn't have to be that ideas are getting harder to find, but people seem to leap very quickly to that assumption.
I screwed up on editing and ended up with µm instead of nm = nanometer but needs for n. That 2 nm line width is in the 20 atoms size range and that 70 nanometers is about a factor of 100 smaller that I could do back in the '60s using state of the art hardware.
In the hard sciences, you do have to stand on the shoulders of giants to even see where you are or where the boundaries of the web of human knowledge are at the present time.
> The hard part of these ideas wasn’t coming up with them; it was picking them out of a bunch of worse ideas
The crux of the post is here. Our values, our beliefs about what is good, determine so much of what happens in our lives. Our values determine what choices we will make. And if you don't think 'correct values' _means anything_, you end up with no way of sorting through a sea of noise.
Believe that there is goodness out there, shiny things waiting patiently for our wits to grow stronger, is what it takes to sort through all the bad ideas to find the few good ones. But sorting takes effort, and we are incapable of exerting effort if we don't expect there to be a reward.
Speaking from my own experience, I think that your "Pop Culture Has Become an Oligopoly" speaks to part of the problem. I was in graduate school in Artificial Intelligence in the late 1970s, with one paper published in 1978 on what is now called Machine Learning. I had a thesis defense about 10 years later that did not go well. Professor: "Your theorem shows that it is possible to learn the complements of recursively enumerable sets. So you that proved that you learn what you cannot compute. In what fields does that happen?" Hapless grad student bluts out "Well, it happens in religion all the time". Stunned silence. And then questioning gets more hostile.
I have a good career in computers, including a stint at Bell Labs/AT&T Labs. But I wanted to get something published in Computational Learning Theory, even though I was, without the PhD, not hireable for a research position. Nothing. Rejection. Everything gets turned down.
So in 2013, at the ripe old age of 60, I say: I am done with trying to get published in Computer Science. I turn to philosophy, and in 2019 get something published in a supplement to the International Journal of Quantum Foundations. My paper is on the relationship between Aristotelian Metaphysics and Quantum Mechanics. Only 41 years between publications. I need to pick up the pace.
Striking physics off my list, I get published the next year in the Journal of Theoretical Biology with the claim that the supposed parasitic retroviruses (transposons) help detemine body part structure. Being theoretical speculation, it needs to be proved or disproved in the lab, though. In 2022 I get a followup paper published in Biosystems on a new, information-based modality for treating cancer. Again, needs to be tested, experimentally. All done in my spare time.
Why, all of a sudden am I successful? I have wondered about that, and your thoughts may in this essay and the pop culture essay may have something to say about it. First off, doing theory and not being part of the insitutions of science gives me freedom. Like Hobbes of Calvin and Hobbes says: "It's one of the perks of being feral." But that speaks to the oligopoly of science. Instead of the follow-on sequels of movies, we get a few ideas where people pile on and the other ideas are ignored and whither away. Currently, the big fad in cancer is immunotherapy. In AI, it is neural nets, which I have always considered a fad that, like too many things in computers, is an overuse of a particular metaphor. Second, the downisde of not being part of the institutions also means that nobody knows my work. One side-effect of proliferation is that only a small fraction of the ideas reach critical mass (go viral). And the rest of us, regardless of quality don't get read. This is especially true if you are not part of the culture. I can only afford to present a poster on my work at one conference. Anything more sucks up too much of the family vacation budget. But that limits the chance of exposing new ideas that are not in the mainstream, even though they may be good. In the last year, I have seen a bunch of concerts, including Wolf Alice in the local clubs, and Taylor Swift at the local stadium. Both Taylor and Ellie Rowsell (Wolf Alice's lyricist) are very talented. - I love them both. But there appears to be only a limited space for superstars at any given time.
> Above a pretty low threshold, we should expect per capita productivity to drop whenever we add more people.
It like not just that inefficiency by "too many cooks" are a problem (Amdahl's Law and the Mythical Man-Month), but an asymmetric increase in "posers" and underperformers when expanding in size (Parkinson's Law and Gervais Principle). This is also why there is an increasing anger against "elite overproduction" as wasted human muscle sucked into the cerebral system.
The social sciences are sufficiently fragmented and the human experience is sufficiently broad enough, that there will always be low-hanging fruit. I think of social sciences often build upon the work of others as an after-thought.
That said, consider fields like astronomy where it used to be possible to make significant contribution by just recording movement of stars as a hobbyist. No way a part-time hobbyist could make a contribution like that anymore! The at-home natural science casual stuff has already been picked! I think this is because the natural sciences are both much more linear than the social sciences (where the elevator analogy makes more sense for them than it does the social sciences) but also they better defined problems (which is why they need elevators in the first place!).
Hi John-Henry! Sorry for the delay in responding. I don't know much about astronomy myself, so I reached out to my friend Josie Peters, who is an astrophysicist (https://www.josieapeters.com/). Here's what she said [lightly edited]:
"Some of the most amazing images we have are down to amateurs. Astronomers have to put in detailed proposals for telescope time (and not a minute more!), so we can only observe enough to achieve our science goals. Whereas an amateur can observe the night sky for a whole year if they want. Here's an example: https://earthsky.org/space/milky-way-panorama-metsavainio-astrophotography/
Also, citizen science has led to big discoveries. A friend of mine, Professor Chris Lintott, created the Zooniverse. There are tones of citizen science projects and loads of big papers that come out of there, ranging from galaxy classification (the discovery of Hanny's Wooverp) to penguin counting.
And machine learning is not accurate enough. They have papers about how the combo of both is the most accurate! So citizen scientists make a huge difference."
Great essay. I wish I’d read this ten years ago. Well no more excuses and I’ll get back to my dreams of solving important problems like consciousness, teleportation and why it’s ten times quicker to write a book if it doesn’t have your name on it.
Although serious question. Now we have AI and tools like deep research it feels like we’re undergoing a tooling shift for a whole new level of low-hanging fruit. What would you suggest to someone who is genuinely interested in tackling more esoteric ideas and not overjoyed by the formalities of a PhD?
I got to work with a few PhD’s for my thesis and they were so uninspired and depressed it seemed like a terrible life choice…
I feel like playing a bit the cowardly pessimistic devil’s advocate.
> But if we want a better world,
Wanting a better world signals that you’re not happy with this one. Is that what you want to signal? If you’re tough and confident in your toughness, you won’t care how awful the world is to all those losers around you. Caring signals you’re afraid you might be one of them. Truth fears no trial.
> we have to believe it’s possible to create one.
Believing without evidence?
> And that takes courage, because if you truly believe in a better world, you have to do something about it.
It also takes courage to say, “Screw you, losers; got mine. If you want some, come take it from my cold dead hands”.
> You don’t get to smugly smirk as the ship sinks; you have to start pumping the water out. Smirking seems easier than pumping at first, but pumping turns out to be really fun.
A real Sociopath (in the Gervais-principle sense) wouldn’t settle for that. First off, they’d subtly degrade and obstruct the first pumper (the bravest person?, except noöne will acknowledge them as such after seeing how the Sociopath treats them) in any way they can without getting their hands dirty. There’s probably some important task to do at the moment and no time to try everyone’s patience toying with that silly pump. There’s likely some authority figure to dutifully report that flagrant misuse of time and equipment to, and you’re savvy enough to be thanked by everyone for your helpful intervention, rather than earning everyone’s wholehearted contempt as a vile snitch, as would happen to that pathetic Loser—or is it a Clueless?—with the pump.
Soon, everyone (see what I’m doing?; the Loser/Clueless isn’t part of “everyone”) will join in on the smirking, the booing, the beaning and the “helpfully”, small-cee cluelessly interfering with the operation of the pump. The Loser/Clueless will be forgotten about, the conversation will evolve ever so cromulently and someone, quite possibly the original Sociopath, will shrewdly figure out that someone needs to pump out the water. The Loser/Clueless will be angrily cast overboard for getting in the way. There’ll be a frantic turmoil and the Sociopath, fed up with so much incompetence, will unfold their portable wings, like the scorpion did after stinging the frog halfway across the river, and fly to greener pastures as all the small-ell losers are left to drown—or, more accurately, to fight one another and then drown.
> You start to feel useful. You make friends with the other people trying to keep the boat afloat. You stop caring about all the people who say what you’re doing will never work. Ultimately, pumping is way easier than smirking, and it feels better too. Optimism cures pain; pessimism, like painkillers, merely dulls it.
To any scientist out there looking for some truly low-hanging fruit: figure out how cats purr.
Yes, that's right. Despite being the world's most popular pet, literally no one knows HOW cats do that funny little vibration thing with their bodies that (generally) means they're happy snuggling right next to your face when you're trying to sleep.
This is a great article! The spirit of this article is infectious and relentlessly positive, and makes some excellent points.
That said, I entirely disagree with your conclusion. My background is accelerator physics, which is maybe the best possible example of it getting harder to do experiments to find new stuff. Old-style accelerators are simple and some of them you can literally build in your backyard (do not attempt this you will die). The frontiers of research in the accelerator communities now are all novel, weird concepts that are really really tricky to make work and require massive engineering infrastructure to even attempt - plasma accelerators, laser plasma wakefield, terahertz guns, DLAs (my project), are all examples of extremely delicate engineering balancing acts that barely work, all trying to get around the fundamental scaling law of accelerators: Energy = gradient * length.
Since particle physics mainly works in orders of magnitude (new physics needs 100x more energy, not 50% more), either we invent new technology to increase gradient (very very hard), or we build a 100x bigger accelerator. Except the LHC is the size of a city.
Anyway I guess my point is that my field has very complex engineering challenges that makes it MUCH harder to build and test an idea than it was in the 1930s.
Ever possible idea about the nature of Reality and what we are as human beings that has been posited in all times and places is now freely available on the internet.
Confronted by all of that how does one even begin to understand the nature of Truth and Reality.
Check out this website provides a comprehensive all inclusive tool for understanding our situation - it currently has 2000 pages:
Good thoughts overall! A few issues I had:
1. I wouldn't say heliocentrism, germ theory, or CTs are obvious in retrospect. I actually just wrote a whole blog post on how germ theory would be impossible without microscopes (https://trevorklee.substack.com/p/a-medical-thought-experiment), and I don't think anyone reading this blog would be able to prove heliocentrism without Googling how to. Meanwhile, corporate marketing still hasn't figured out RCTs, even though there's a direct financial incentive for them. Also, the concepts (and math) embedded in E=MC^2 are, again, beyond the ken of almost anyone reading this blog.
2. There's a difference between "we're running out of ideas" (which is silly) and "all the low-hanging fruit has already been picked".
3. There's a difference between discoveries in physics, which are supposed to be true for everywhere at all times, and discoveries in psychology, which can't make that assumption (i.e. I'm not sure cavemen abided by the same conversational conventions we do).
4. I think bringing in Thomas Kuhn earlier would have helped. There's always a flood of "easy" papers at the beginning of a paradigm shift, as the basic elements of the paradigm get fleshed out and improved. This also is often true when new instruments get developed.
1. I think this is beyond the level that people get to when they think ideas are getting harder to find. There are two illusions here. One is that people probably think they understand these ideas better than they do, and even if they know they'd have to do some research, they underestimate just how much. The other is that they underestimate how hard it was for the original thinkers to come up with this stuff. Germs seem obvious because we learn about them from the time we're children, and it seems silly that anybody ever doubted their existence. It's just hard for us to put ourselves in a world where the most educated people could think really hard and still be totally incredulous that tiny organisms could cause disease.
2. They're similar if you believe the latter strongly enough. If ideas really do get harder to find, eventually it becomes unlikely that anybody ever discovers one, so practically it feels like running out.
3. I think this seems truer than it is because physics has developed much farther than psychology. Physicists had to figure out variations across contexts as well, like why very small systems seemed to act differently. Psychologists might have a harder problem, and I have no idea how they'll solve it, but I'm optimistic about it. For now, we are still cavemen trying to explain why a giant fireball appears in the sky every day.
4. It's interesting that this hasn't happened in psychology. We've piled up a bunch of replication failures, but nobody seems to be stepping up to provide an alternative. I think this is because we don't really have a strong paradigm; a very weak paradigm can handle lots of problems without failing, because there's nothing really to argue against.
The article "ideas are harder to find" is based not on ideas but on the final economic implementation of ideas that impact the economy (where they get the total factor productivity numbers). That is Ideas + economic feasibility + resource availability ($$$) + permissions (regulatory) + defeating political attacks from those who fear or are threatened by the innovation. Any one of these series steps can be the rate-limiting step. Even if you have an infinite number of scientific ideas, the rate of economic growth and apparent innovation will be limited by one of the other factors.
The authors also misuse R&D spending by companies to obtain progress as a metric for saying that R&D productivity is declining without noting that the actual areas being researched are changing over time. In an evolving technology area like chips or GMO, the researchers borrow from other areas of technology, which at that moment, are more advance than their new area. The cost of the R&D in that spill-in area of technology is on someone else's books. For example, initial semi-conductor manufacturers used clean rooms from hospital suppliers, vaccum systems from the space program, everything from saws to grinding/polishing systems from metallurgy, etc. When the feature size of chips evolved to the point where a virus looms as a mountain of impurity, they were compelled to do R&D on clean rooms. That technology advance then spilled-back to hospital clean room technology. The required R&D was then on the Chip companies books, being counted as declining R&D productivity.
The spill-in and spill-out of R&D benefits for modern multi-factor innovations is very complex and results in areas like biotech obtaining a huge spill-in of technology from computer/chip area. For analyzing DNA by looking at a color of light for each letter being added (only 4 letters) onto the sample DNA they needed holes in the 70 nm range that utilized a spill-in from semi-conductor technology. Some of these biotechnology areas are growing even faster than Moore's Law as they adapt the R&D from both semi-conductors and computers.
Meanwhile, the new semi-conductors with 2 nm line widths need very hard UV light close to soft X ray light sources. Light sources use a small drop of tin hit with a laser beam to a high temperaure plasma state, but this technology probably has a massive R&D spill-in from the billions spent on inertial confinement fusion, where a small pellet containing hydrogen isotopes is compressed to temperatures of our sun for fusion, also using converging laser beans. A few billion in R&D spill-in is good.
The "low hanging fruit" mental model or even the "tree of knowledge" model don't work very well. The very low hanging centuries old fruit of Maxwell's equations, General Relativity, and the Schrödinger equation has long gone to seed and produced trees whose branches meld with every branch of the founding tree of human knowledge they cross. When every point of crossing branches effectively starts a new tree, the tree model ceases making sense, and network models more useful images. Your i-phone utilizes Maxwell's equations, General Realtivity and Quantium mechanics to make it work. Those are not elements of Apple's research budgets.
As the overall "knowledge network" grows, more possible ideas and technologies are created at a rate much faster than the economies in the deleveloped world economies (probably up around 10%/yr). Clearly, we can conclude that the slow step controlling economic growth is not lack of "Ideas". Science has a meaning of the word "idea" that excludes economic/political factors outside the idea itself.
The regulatory and permission areas are expanding rapidly and getting slower. Imagine, if Apple had to build the i-Phone in California and needed 100,000 employees in a new facility utilizing hazardous materials, highways, etc. This is a state where trying to get permission for desalinization of seawater, in water-short Southern California, has proved impossible after decades of trying and tens of millions of dollars spent in the effort. Meanwhile Israel now obtains 50% of their water supply from desalization using RO technology, that was developed at UCLA in the '50s. Good ideas abound. It is not a lack of ideas, but obtaining permission to make use of the ideas which is the sadly limiting factor which is preventing the United States from fulfilling its potential, to the detriment not only of our country, but of the world.
To me, the whole "idea shortage" concept is a convenient excuse to blame innovation stagnation on something other than the true "rate controlling" step, which has been put into place by government regulation. With the web of knowledge now being so large, people like the authors of this article (Bloom, Jones, Van Reenen, and Webb) may not have the basic knowledge required to see the real complexity and only see what is in the sub-network of economics. Just viewing the economics (accounting numbers), we can get accounting artifacts that say R&D productivity is decreasing when the R&D is just covering more breadth.
The article "ideas are harder to find" is based not on ideas but on the final economic implementation of ideas that impact the economy (where they get the total factor productivity numbers). That is Ideas + economic feasibility + resource availability ($$$) + permissions (regulatory) + defeating political attacks from those who fear or are threatened by the innovation. Any one of these series steps can be the rate-limiting step. Even if you have an infinite number of scientific ideas, the rate of economic growth and apparent innovation will be limited by one of the other factors.
The authors also misuse R&D spending by companies to obtain progress as a metric for saying that R&D productivity is declining without noting that the actual areas being researched are changing over time. In an evolving technology area like chips or GMO, the researchers borrow from other areas of technology, which at that moment, are more advance than their new area. The cost of the R&D in that spill-in area of technology is on someone else's books. For example, initial semi-conductor manufacturers used clean rooms from hospital suppliers, vaccum systems from the space program, everything from saws to grinding/polishing systems from metallurgy, etc. When the feature size of chips evolved to the point where a virus looms as a mountain of impurity, they were compelled to do R&D on clean rooms. That technology advance then spilled-back to hospital clean room technology. The required R&D was then on the Chip companies books, being counted as declining R&D productivity.
The spill-in and spill-out of R&D benefits for modern multi-factor innovations is very complex and results in areas like biotech obtaining a huge spill-in of technology from computer/chip area. For analyzing DNA by looking at a color of light for each letter being added (only 4 letters) onto the sample DNA they needed holes in the 70 µm range that utilized a spill-in from semi-conductor technology. Some of these biotechnology areas are growing even faster than Moore's Law as they adapt the R&D from both semi-conductors and computers.
Meanwhile, the new semi-conductors with 2µ line widths need very hard UV light close to soft X ray light sources. Light sources use a small drop of tin hit with a laser beam to a high temperaure plasma state, but this technology probably has a massive R&D spill-in from the billions spent on inertial confinement fusion, where a small pellet containing hydrogen isotopes is compressed to temperatures of our sun for fusion, also using converging laser beans. A few billion in R&D spill-in is good.
The "low hanging fruit" mental model or even the "tree of knowledge" model don't work very well. The very low hanging centuries old fruit of Maxwell's equations, General Relativity, and the Schrödinger equation has long gone to seed and produced trees whose branches meld with every branch of the founding tree of human knowledge they cross. When every point of crossing branches effectively starts a new tree, the tree model ceases making sense, and network models more useful images. Your i-phone utilizes Maxwell's equations, General Realtivity and Quantium mechanics to make it work. Those are not elements of Apple's research budgets.
As the overall "knowledge network" grows, more possible ideas and technologies are created at a rate much faster than the economies in the developed world economies (probably up around 10%/yr). Clearly, we can conclude that the slow step controlling economic growth is not lack of "Ideas". Science has a meaning of the word "idea" that excludes economic/political factors outside the idea itself.
The regulatory and permission areas are expanding rapidly and getting slower. Imagine, if Apple had to build the i-Phone in California and needed 100,000 employees in a new facility utilizing hazardous materials, highways, etc. This is a state where trying to get permission for desalinization of seawater, in water-short Southern California, has proved impossible after decades of trying and tens of millions of dollars spent in the effort. Meanwhile Israel now obtains 50% of their water supply from desalinization using RO technology, that was developed at UCLA in the '50s. Good ideas abound. It is not a lack of ideas, but obtaining permission to make use of the ideas which is the sadly limiting factor which is preventing the United States from fulfilling its potential, to the detriment not only of our country, but of the world.
To me, the whole "idea shortage" concept is a convenient excuse to blame innovation stagnation on something other than the true "rate controlling" step, which has been put into place by government regulation. With the web of knowledge now being so large, people like the authors of this article (Bloom, Jones, Van Reenen, and Webb) may not have the basic knowledge required to see the real complexity and only see what is in the sub-network of economics. Just viewing the economics (accounting numbers), we can get accounting artifacts that say R&D productivity is decreasing when the R&D is just covering more breadth.
Thanks for this great comment! I agree that, even if whatever measures of scientific output were indeed slowing down, it doesn't have to be that ideas are getting harder to find, but people seem to leap very quickly to that assumption.
Adam,
I screwed up on editing and ended up with µm instead of nm = nanometer but needs for n. That 2 nm line width is in the 20 atoms size range and that 70 nanometers is about a factor of 100 smaller that I could do back in the '60s using state of the art hardware.
In the hard sciences, you do have to stand on the shoulders of giants to even see where you are or where the boundaries of the web of human knowledge are at the present time.
> The hard part of these ideas wasn’t coming up with them; it was picking them out of a bunch of worse ideas
The crux of the post is here. Our values, our beliefs about what is good, determine so much of what happens in our lives. Our values determine what choices we will make. And if you don't think 'correct values' _means anything_, you end up with no way of sorting through a sea of noise.
Believe that there is goodness out there, shiny things waiting patiently for our wits to grow stronger, is what it takes to sort through all the bad ideas to find the few good ones. But sorting takes effort, and we are incapable of exerting effort if we don't expect there to be a reward.
Excellent points, Adam!
We need news ideas more than ever. Here’s to the high-hanging fruit.
Let’s fight the cowards together.
Speaking from my own experience, I think that your "Pop Culture Has Become an Oligopoly" speaks to part of the problem. I was in graduate school in Artificial Intelligence in the late 1970s, with one paper published in 1978 on what is now called Machine Learning. I had a thesis defense about 10 years later that did not go well. Professor: "Your theorem shows that it is possible to learn the complements of recursively enumerable sets. So you that proved that you learn what you cannot compute. In what fields does that happen?" Hapless grad student bluts out "Well, it happens in religion all the time". Stunned silence. And then questioning gets more hostile.
I have a good career in computers, including a stint at Bell Labs/AT&T Labs. But I wanted to get something published in Computational Learning Theory, even though I was, without the PhD, not hireable for a research position. Nothing. Rejection. Everything gets turned down.
So in 2013, at the ripe old age of 60, I say: I am done with trying to get published in Computer Science. I turn to philosophy, and in 2019 get something published in a supplement to the International Journal of Quantum Foundations. My paper is on the relationship between Aristotelian Metaphysics and Quantum Mechanics. Only 41 years between publications. I need to pick up the pace.
Striking physics off my list, I get published the next year in the Journal of Theoretical Biology with the claim that the supposed parasitic retroviruses (transposons) help detemine body part structure. Being theoretical speculation, it needs to be proved or disproved in the lab, though. In 2022 I get a followup paper published in Biosystems on a new, information-based modality for treating cancer. Again, needs to be tested, experimentally. All done in my spare time.
Why, all of a sudden am I successful? I have wondered about that, and your thoughts may in this essay and the pop culture essay may have something to say about it. First off, doing theory and not being part of the insitutions of science gives me freedom. Like Hobbes of Calvin and Hobbes says: "It's one of the perks of being feral." But that speaks to the oligopoly of science. Instead of the follow-on sequels of movies, we get a few ideas where people pile on and the other ideas are ignored and whither away. Currently, the big fad in cancer is immunotherapy. In AI, it is neural nets, which I have always considered a fad that, like too many things in computers, is an overuse of a particular metaphor. Second, the downisde of not being part of the institutions also means that nobody knows my work. One side-effect of proliferation is that only a small fraction of the ideas reach critical mass (go viral). And the rest of us, regardless of quality don't get read. This is especially true if you are not part of the culture. I can only afford to present a poster on my work at one conference. Anything more sucks up too much of the family vacation budget. But that limits the chance of exposing new ideas that are not in the mainstream, even though they may be good. In the last year, I have seen a bunch of concerts, including Wolf Alice in the local clubs, and Taylor Swift at the local stadium. Both Taylor and Ellie Rowsell (Wolf Alice's lyricist) are very talented. - I love them both. But there appears to be only a limited space for superstars at any given time.
> Above a pretty low threshold, we should expect per capita productivity to drop whenever we add more people.
It like not just that inefficiency by "too many cooks" are a problem (Amdahl's Law and the Mythical Man-Month), but an asymmetric increase in "posers" and underperformers when expanding in size (Parkinson's Law and Gervais Principle). This is also why there is an increasing anger against "elite overproduction" as wasted human muscle sucked into the cerebral system.
I think this is true and not true.
The social sciences are sufficiently fragmented and the human experience is sufficiently broad enough, that there will always be low-hanging fruit. I think of social sciences often build upon the work of others as an after-thought.
That said, consider fields like astronomy where it used to be possible to make significant contribution by just recording movement of stars as a hobbyist. No way a part-time hobbyist could make a contribution like that anymore! The at-home natural science casual stuff has already been picked! I think this is because the natural sciences are both much more linear than the social sciences (where the elevator analogy makes more sense for them than it does the social sciences) but also they better defined problems (which is why they need elevators in the first place!).
Hi John-Henry! Sorry for the delay in responding. I don't know much about astronomy myself, so I reached out to my friend Josie Peters, who is an astrophysicist (https://www.josieapeters.com/). Here's what she said [lightly edited]:
"Some of the most amazing images we have are down to amateurs. Astronomers have to put in detailed proposals for telescope time (and not a minute more!), so we can only observe enough to achieve our science goals. Whereas an amateur can observe the night sky for a whole year if they want. Here's an example: https://earthsky.org/space/milky-way-panorama-metsavainio-astrophotography/
Also, citizen science has led to big discoveries. A friend of mine, Professor Chris Lintott, created the Zooniverse. There are tones of citizen science projects and loads of big papers that come out of there, ranging from galaxy classification (the discovery of Hanny's Wooverp) to penguin counting.
And machine learning is not accurate enough. They have papers about how the combo of both is the most accurate! So citizen scientists make a huge difference."
Great essay. I wish I’d read this ten years ago. Well no more excuses and I’ll get back to my dreams of solving important problems like consciousness, teleportation and why it’s ten times quicker to write a book if it doesn’t have your name on it.
Although serious question. Now we have AI and tools like deep research it feels like we’re undergoing a tooling shift for a whole new level of low-hanging fruit. What would you suggest to someone who is genuinely interested in tackling more esoteric ideas and not overjoyed by the formalities of a PhD?
I got to work with a few PhD’s for my thesis and they were so uninspired and depressed it seemed like a terrible life choice…
I feel like playing a bit the cowardly pessimistic devil’s advocate.
> But if we want a better world,
Wanting a better world signals that you’re not happy with this one. Is that what you want to signal? If you’re tough and confident in your toughness, you won’t care how awful the world is to all those losers around you. Caring signals you’re afraid you might be one of them. Truth fears no trial.
> we have to believe it’s possible to create one.
Believing without evidence?
> And that takes courage, because if you truly believe in a better world, you have to do something about it.
It also takes courage to say, “Screw you, losers; got mine. If you want some, come take it from my cold dead hands”.
> You don’t get to smugly smirk as the ship sinks; you have to start pumping the water out. Smirking seems easier than pumping at first, but pumping turns out to be really fun.
A real Sociopath (in the Gervais-principle sense) wouldn’t settle for that. First off, they’d subtly degrade and obstruct the first pumper (the bravest person?, except noöne will acknowledge them as such after seeing how the Sociopath treats them) in any way they can without getting their hands dirty. There’s probably some important task to do at the moment and no time to try everyone’s patience toying with that silly pump. There’s likely some authority figure to dutifully report that flagrant misuse of time and equipment to, and you’re savvy enough to be thanked by everyone for your helpful intervention, rather than earning everyone’s wholehearted contempt as a vile snitch, as would happen to that pathetic Loser—or is it a Clueless?—with the pump.
Soon, everyone (see what I’m doing?; the Loser/Clueless isn’t part of “everyone”) will join in on the smirking, the booing, the beaning and the “helpfully”, small-cee cluelessly interfering with the operation of the pump. The Loser/Clueless will be forgotten about, the conversation will evolve ever so cromulently and someone, quite possibly the original Sociopath, will shrewdly figure out that someone needs to pump out the water. The Loser/Clueless will be angrily cast overboard for getting in the way. There’ll be a frantic turmoil and the Sociopath, fed up with so much incompetence, will unfold their portable wings, like the scorpion did after stinging the frog halfway across the river, and fly to greener pastures as all the small-ell losers are left to drown—or, more accurately, to fight one another and then drown.
> You start to feel useful. You make friends with the other people trying to keep the boat afloat. You stop caring about all the people who say what you’re doing will never work. Ultimately, pumping is way easier than smirking, and it feels better too. Optimism cures pain; pessimism, like painkillers, merely dulls it.
It’s the Sociopath’s job to reverse all that.
Adam, this article was nothing short of amazing.
THANK YOU for taking the time to write this.
To any scientist out there looking for some truly low-hanging fruit: figure out how cats purr.
Yes, that's right. Despite being the world's most popular pet, literally no one knows HOW cats do that funny little vibration thing with their bodies that (generally) means they're happy snuggling right next to your face when you're trying to sleep.
This is a great article! The spirit of this article is infectious and relentlessly positive, and makes some excellent points.
That said, I entirely disagree with your conclusion. My background is accelerator physics, which is maybe the best possible example of it getting harder to do experiments to find new stuff. Old-style accelerators are simple and some of them you can literally build in your backyard (do not attempt this you will die). The frontiers of research in the accelerator communities now are all novel, weird concepts that are really really tricky to make work and require massive engineering infrastructure to even attempt - plasma accelerators, laser plasma wakefield, terahertz guns, DLAs (my project), are all examples of extremely delicate engineering balancing acts that barely work, all trying to get around the fundamental scaling law of accelerators: Energy = gradient * length.
Since particle physics mainly works in orders of magnitude (new physics needs 100x more energy, not 50% more), either we invent new technology to increase gradient (very very hard), or we build a 100x bigger accelerator. Except the LHC is the size of a city.
Anyway I guess my point is that my field has very complex engineering challenges that makes it MUCH harder to build and test an idea than it was in the 1930s.
"Above a pretty low threshold, we should expect per capita productivity to drop whenever we add more people."?
Simple answer: https://www.youtube.com/watch?v=LOJbM0aXZp0
Ever possible idea about the nature of Reality and what we are as human beings that has been posited in all times and places is now freely available on the internet.
Confronted by all of that how does one even begin to understand the nature of Truth and Reality.
Check out this website provides a comprehensive all inclusive tool for understanding our situation - it currently has 2000 pages:
www.beezone.com/whiteandorangeproject/index-2.html
Plus this reference www.beezone.com/beezones-main-stack/gtbiblio.html
The book/project featured on the site directly above now features 80 thousand items
Four related essays beginning with this one www.intergalworld.net/reynolds16.html