Love your work, Adam, but have to take issue with 2 things here - one big & one small (I’m not an academic psychologist; I’m an academic in another field, so I’m not defending my turf).
1. Solving psychiatric illness: Real, important, replicable progress is being made every year. Tens, maybe hundreds, of thousands of people just in the US can live independent and meaningful lives because they take psychiatric medication for serious psychiatric illnesses. This includes major and treatment-resistant depression, treated with TMS or neurostimulators. I recognize that you are writing about psychology and not psychiatry, but: (a) you said that nothing in the DMS has been “cured”; yet people with serious DMS diagnoses *can* be fully treated; and (b) making the argument that psychology is no better now than it was 50 years ago leads to treatment nihilism, which can seriously harm people who then miss out on effective treatments.
2. “Big 5.” It may be that the big five are no more predictive than hocus pocus, but you can’t tell that by looking at whether somebody buys a house in their lifetime. It’s the wrong endpoint. I could predict, on an average basis, who will or won’t buy a house in their lifetime and it doesn’t have anything to do with psychology. I would ask: 1. Are they White? 2. Have they received an inheritance or other chunk of change from a family member? Being white and receiving intergenerational wealth (two variables that are also correlated) are the strongest predictors of home ownership. We should not expect psychology to predict unlike types of social facts.
Thanks again for the great article. Please keep up the great work!
1. I'm glad we're getting better at these things! I think it's important for patients entering psychotherapy to have a realistic idea of what it can do for them, and that comes from being transparent about its abilities as well as its limitations. The more damaging kind of nihilism comes from the practitioners themselves––one of those meta-analyses begins with the author claiming that we shouldn't try to come up with new treatments for depression, because they all work equally well. That's pretty scary to me!
2. I think the response from the Big Five people would be something like: the random luck of being born privileged will affect your outcomes, but personality should matter as well. Some people who are white and inherit money don't end up owning a house, and lots of people who aren't white and don't inherit money *do* end up owning a house, and those differences should be partially explained by their personalities (conscientiousness in particular, I'd imagine). I don't think the Big Five does spectacularly well at predicting that, but I would bet that you'd make better predictions using the Big Five than using race + inheritance alone.
I found that many experts claim that talking therapy does help. One source I found states that:
> Treatment options such as medication and talking therapy play an important role in helping many people with depression and can significantly improve people’s lives.
Regarding the second point, I agree with Amanda that the examples are not perfect in this context. While the Big 5 don't perform much better than the MBTI in the given context, I'd argue it is because they are at not necessarily are supposed to predict these given outcomes (Maybe at a facet level they would do better? I am not sure). There are however other things the Big 5 are much better at predicting. For example, conscientiousness is a great predictor of academic achievement, rivaling intelligence in certain age groups (e.g. Poropat, 2009). I am not aware of any studies indicating that any of the MBTI score are a predictive of academic achievement.
Overall, thanks for writing this article. You verbalised many things that left me frustrated with psychology during my PhD and still bother me today. This has helped me pin-point more precisely where exactly my frustration stems from.
Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2), 322–338. https://doi.org/10.1037/a0014996
Also, the psychology of addiction and the bio-psychology of early-life adverse events and their relationship to future addiction (among other things) represent demonstrable advances over folk psychology, with important, actionable outcomes for treatment, prevention, and policy.
Friendly challenge: Take a look into that subfield, see if you agree, and, if so, see what’s the same or different about systems, questions, funding or mentorship structures, or other factors.
And yet, I can find no demonstrable improvement in our ability to prevent or cure addiction. I'm not convinced any of this is real. If you can find a real application explicitly based on this research, that has consistently worked, please point me in that direction.
Considering *gestures at the encampments outside*, I think you’d do well to try to justify *your * claim that we’ve figured something out about addiction rather than sending readers off to dig in a no-doubt vast literature to which you’ve given not even a hint of guidance.
Hi, Jerome. My comment is to and for Adam, an academic psychologist. He should be familiar enough with the field to know how to begin, particularly since he’s saying here that he has reviewed the entire field of empirical psychology and found little evidence it has produced any contributions. If you’re genuinely curious, I recommend you start with the work of Bruce McEwen, from Rockefeller University. Though McEwen was a biological psychiatrist, his work on early life stress and allostatic load (both in relation to addiction and beyond) has profoundly shaped work in psychology, psychiatry, and policy.
Thank you for the suggestion, but the conversation is about psychology’s real-world impacts. If it’s necessary to dig into the primary literature to ferret out unspecified impacts of 30-year-old work, which has thus had an entire generation to flesh itself out in treatments, then the claim is self-defeating. At the heart of the post here is that mature sciences can demonstrate impacts legible to non-participants; it seems clear that you can’t do that along this line, not to mention the fact that it’s very far from clear that other “academic psychologists” find your argument any more convincing than I do.
Regarding allostatic load, I find: “Interventions can include encouraging sleep quality and quantity, social support, self-esteem and wellbeing, improving diet, avoiding alcohol or drug consumption, and participating in physical activity.” I hope you’ll forgive my lack of astonishment at the level of impact on psychiatric practice reflected here.
I was interested to see you list Anki as a success at “cramming it into a box,” because I’ve spent about ten years developing, supporting, and writing about spaced-repetition systems in some capacity (including Anki), and I’ve often written about how using memory tools well is actually very complicated, noting that if only we could find a prescriptive set of simple rules you could follow (like you could encode into software) that would consistently work, it could finally take off!
Anki does do a good job at packaging up the insight of spaced repetition itself. Unfortunately, that is only a tiny part of the problem of learning/remembering things, and solving this restricted problem leaves most of the real problem unsolved. Just a few common and important problems it doesn't solve:
* Lack of motivation / habit – people forget to study, don't want to study, don't study for a few days and then come back and have a huge pile of cards and give up, etc. (This is moderately tractable – I think RemNote, the startup I’m currently involved in, does much better than Anki does here – but habits are just hard, and there’s only so much you can do to help people remember things if they don’t review them!)
* Learning something that's useful / you care about knowing – it's super common for people to use Anki to memorize, say, all the symbols on the periodic table, and then if you ask them why they wanted to know that, they have no idea.
* Writing cards that are unclear – oftentimes when you write a card (or write one for someone else to import), you think you know what it’s asking, but when you actually see it out of context 3 months later it is inscrutable, or makes you think you were supposed to give some other answer instead. In the best case, this is frustrating and messes up the spacing algorithm because you can’t accurately evaluate how well you remembered the content once you fix it. More commonly, people who don’t have a lot of experience don’t even notice why they didn’t know the answer, and keep forgetting the card over and over again without ever doing anything to fix it.
* Learning the thing the way you need to remember it – it’s really easy to memorize a card based on, say, some unusual wording on the front of it, and find yourself unable to recall the information when given a real-life cue. Or you might just ask the question in a way that doesn’t match how you’ll be asked to recall it, and find you just can’t quite make the connection when you are. (Memory is much more sensitive to tiny shifts in context than most people realize.)
* Keeping cards relevant / understanding why you’re struggling – it’s common for cards to become stale because you no longer care about the information on them, or you don’t understand what they were supposed to mean, so you struggle to remember them, and you just keep repeating them over and over again until you get frustrated.
Maybe the core issue is that the apparently simple problem of "remembering question/answer pairs" is deeply tangled up with all sorts of other complicated human stuff that prevents you from actually effectively using the tool unless you also solve those other problems (or at least partially address them). I appreciate the claim that the “people are just complex” argument is unimaginative, but I suspect this may be where psychology really does diverge from, say, physics. Sure you're ignoring all kinds of factors when you, say, model the trajectory of a ball you throw. But with just a couple of equations, you still get answers that are accurate enough to do amazing things to a pretty decent degree of accuracy – you can really forget about a lot of the adjacent questions (e.g., relativity, or often even air resistance). Whereas if you just hand someone Anki, it’s still better than them not having it, but the vast majority of people get resoundingly mediocre results, because of all these adjacent issues, and there's not an obvious way to cram solutions to those issues into the box, because they deeply depend on parts of your internal mental state that can't be reliably introspected. This is frustrating, so most people give up, even though the spaced-repetition tool itself is extremely powerful.
Now I guess you could argue something similar about the ICE. Like of course having a great ICE doesn’t mean people will become good drivers, that's a separate problem to solve, and that shouldn’t diminish the beauty of putting combustion in a box. But there’s some way in which dismissing it like this feels unsatisfying to me – with the ICE that seems so obvious as to be absurd, while with spaced repetition it really feels to me like the fundamental problem is basically unsolved, and we’ve only handled some tiny restricted subproblem. Is it just that we think about psychology differently? Or that the path to making cars safer is clearer than the path to understanding what cards will change someone's mental state into being a person they prefer to be? I’m not sure.
(I have a bunch of blog posts about addressing some of these issues with memory tools – by developing personal skills, not by improving the tools – for anyone curious. You can start at https://controlaltbackspace.org/precise/.)
Super interesting, thank you! I agree that we haven't solved the problem of remembering things and boxed it in Anki––we've figured out two things (spaced repetition and testing are betting than nothing) and boxed those. I think the reason we haven't figured out the rest is because those problems are harder to see in the first place. Nobody booting up Anki thinks to themselves, "I probably need help understanding why I want to memorize these things in the first place." If we knew why people try to memorize things they don't actually care about, we might be able to build a solution into the box.
I live outside of Cincinnati. And maybe the issue here is, you’re thinking of Cincinnati as a destination, rather than a heading.
If the heading is looked at through the classical lens of, “how can I live excellently”, we might instead focus on the things that top performers do differently. Talk to, for example, any professional athletes about the importance of attitude. Or visualization. Or the dangers of resentment. Or the necessity of relaxation, or the power of routine. These people will do anything they can for an edge.
Instead of studying random psychologists, study people like Darryl Davis, a black man who converted numerous members of the KKK, to abandon the group, mainly by listening to them and earning their respect. See how successful FBI hostage negotiators think about conflict, and learn from them, instead of studying random people who for the most part aren’t super functional or happy.
You’ll find that people who thrive in highly competitive fields do things very differently from ordinary people, using what I’d say are toolkits developed from psychological knowledge, a combination of folk psychology and actual empirical evidence. Their success is evidence that they understand reality.
If you want to be really heterodox, really think outside the box, try floating the idea that some ways of thinking actually work better than others. It’s heresy in academia to say you think this is true of human metanarratives. Instead of setting “mere understanding” as the goal, make the goal pursuit of excellence and see what comes of it. Study the great teachers of the past and try to reverse engineer their logic. Taking seriously claims by numerous faiths, see if you can find their intersection and then understand why those things work. Study very successful people and see what makes their belief structures different. See if you can predict future success from high schoolers based on narratives like “the world is rigged against me” vs “if I try hard I will succeed” vs “if I learn a valuable skill I’ll be financially ok”
I predict your colleagues will disparage you for the audacity to believe some belief structures really are better than others, and this maybe explains why the field is stuck.
Cincinnati is named, by the way, after a Roman dictator who was asked by the senate to stick around in power, since he did a good job in an emergency. His response was, “no, I’d rather go tend my farm.” This response strikes most people today as alien, but I think the ethos pervades the city and its surroundings: family and community are more important than anything else. This is, I think, what makes it work.
Interesting ideas! Of course, we have to be ultra-careful here not to get spurious results due to survivorship bias. So, just by analyzing the techniques of successful people, you won't yet have a causal result (mainly since you don't know how many people were using the same techniques but failed in the end). If you try to run experiments with the knowledge gathered afterwards, then we're talking!
As for comparing success of high-schoolers: Getting causality would again be difficult here. If the students have these attitudes prior to the experiment, then there are reason for that - financial situation, racism, (lack of) luck, generalizing from personal experience, trauma, whatever. As long as the source isn't "I have randomly acquired this attitude", you will have confounders which may be very difficult to control for.
What you could try is to instill these attitudes - perhaps just the ones we expect to yield positive results so as not to farr unethically - with a control group receiving no such input. Maybe a motivational speech or something. I'd definitely be interested to read the results of such a study.
FBI hostage negotiators are using specialized tools that most psychologists don't have, called 'firearms'. Davis, on the other hand, is using a classic tool of all good psychological research known as "making the whole thing up".
I have an urge to write a smarter, more informed comment. But it is dinnertime and the site of Cincinnati Chili - and I grew up in Cincy - is instead triggering far more primal hunger urges. Darn right in Cincy you get to eat that chili, and it is the thing I miss most about no longer living there.
It seems to me that psychology is caught between two opposing forces:
1. There's ample evidence anywhere you care to look that deliberate or accidental intervention can dramatically alter human behavior: take for example cults, revolutionary movements, totalitarian governments, whatever it was that worked to mostly get people to stop smoking cigarettes.....
2. Attempts by the discipline to turn these examples into a set of systematic principles that can be applied according to some underlying theory have been at best underwhelming and at worst, catastrophically bad.
Maybe it's worth digging into how both these things can be true....?
I think it means that we haven't gotten the right principles, probably we have the wrong idea about how principles get generated. I'm planning to write more about this in the future.
Psychologists* like to think they've debunked loads of folk assumptions, but I really don't think they have. The Milgram study doesn't challenge anything we previously assumed about human behaviour - we've known for thousands of years that most people defer to authority figures, most human organisations are built on this fact. What Milgram challenged was more like our post-WW2 Liberal/individualist assumptions about ourselves. It shocks us because we think that *we* wouldn't keep pressing the button, and the results indicate that we probably would.
Folk wisdom has stood the test of time for a reason. If we want to improve psychology, I think we should start by trying to understand why.
*(I can make these sorts of generalisations, I'm one myself)
Daniel Lakens has suggested that social science research could benefit from greater coordination. Physics, for example, has quite a centralised model because of how expensive the equipment is — they have to meet up and agree "okay, US you look at these questions; Europe, look at these questions, etc".
Social science by contrast is very decentralised. Incentive is to pursue research that is likely to turn up a publishable result, even if it's unlikely to advance our understanding very much. If the field were more coordinated, perhaps that may allow psychologists to build up big theories and pursue risky studies that take decades but produce valuable insights.
I think coordination may be a symptom of progress rather than a cause. The advantage of a paradigm is that it brings some consensus about which problems are important to solve in the first place, so they can be divided and conquered. I don't think there is a justifiable consensus like that right now in psychology.
One point of critique: Your 5th criterion of the old psych theories not being considered absurd seems like a pretty unfair comparison to me. You compare the humors, a theory from about 2500 years ago[1] (which could only be disproven once science proper arrived, I.e. about 200 years ago) to the ego/id/superego theory from about 100 years ago[2].
I don't know about the (history of) psychology to provide a counterexample if one exists, perhaps someone else knows more? Though obviously it is tough to prove that NO psychological perspective/terminology has died out like the humor concepts did. Additionally, perhaps the id/ego/superego model remains in use *because* it captures enough of a real phenomenon to be of use. I'm sure there is some part of biological or medical understanding from 100 or even 1000 years ago that is still in active use today.
Overall, I'm having difficulty seeing strong support for psychology failing this criterion.
I mean the humors example to show how a paradigm shift can show up as a change in terminology, rather than to compare the amount of progress in medicine vs. psychology. I think the id/ego/superego example is illustrative because it's some of the earliest work that we would consider psychology, and it's still influential. And I think you're right about the reason for that: we haven't yet discovered something that's obviously better.
Setting aside the majority of the article, the ending about mysteries struck me the most. We psychologists are really good in digging up concepts and naming things but we are not really good - not at all - in explaining why these concepts come about and why and how do they hang together, and most especially - when and where do these even apply. Psychology puts, by definition too much emphasis on the individual. But "individuals" are actually entangled jumbles of other people, varied context and settings, their history, etc. As a result, you kind of end up throwing spaghetti on the wall - megastudy style - whenever you want to change or reinforce a "simple" behaviors such as gym attendance or vaccination. In other words, what psychology is missing are explanatory mechanisms and how these are reinforced or dampened by varied contexts.
Linked to that is that, implicitly, the aim of psychology is too damn high: under a positivist paradigm, where - to be called science - we have to use RCTs, pay attention to concepts like internal validity and replicability and generalizability, and use increasingly more and more arcane statistical techniques, psychology, as a science, is found wanting. But I think this stems partially from wanting too much: if we assume that any trial findings are context-bound (irrespective of whether you've calculated a sample size for your RCT to be representative of the underlying population), we can dismiss generalizability and replicability as concepts that don't apply to psychology. If we also agree that any finding found in the real world is much more valuable than one found in artificial settings (hello, MTurk), we put ecological validity over internal validity. By doing so, also, we acknowledge the wobbliness we're dealing with in psychology: important things such as emotions, meanings, consciousness-related stuff, don't easily - or at all - yield to measurement. If we abandon all that, I have a sense we would actually start making some progress and who knows, maybe one day a future psychologist will finally tell me why people enjoy watching other people cook.
The problem is that psychologists assume, rashly, that people can be described at scale. Now, in part this is a methodological artifact, the drunk's streetlight: for various reasons, the Big Five is way more tractable as an instrument, as a fundamental theory, etc., so that's the block you look on even if you lost the keys in the dark block a street over.
I think that in practice the Big Five might be better as the Big 300. I'm a million years away from being able to codify this, but my assuming it's true has led to some extremely interesting discoveries and successes in my work identifying customers who are WAY more likely to buy your thing. It's a Pareto, like everything else, only it's not a univariate factor making up that Pareto. It's a bunch. But even identifying one or two of them is bringing a machine gun to the marketing stick fight. And I'm not particularly interested in "proving it at scale." That's not the area I work in, and it's the wrong approach anyway.
Thing is, a lot of folk stuff works. A lot of success is being just a little bit ahead of other people, and "empirically verifiable" is usually not the name of the track you're running on. Now, if you have evidence it *is* empirically verifiable (vaccines), don't be a fool. But that's not usually relevant to the problems you're facing.
Interesting article. It takes a brave man to review all the accumulated wisdom gathered from his own discipline and conclude “we have not really learned much.”
Unfortunately, this seems to be true of most if not all social sciences. Not sure where that leaves us…
Economics has come a long way. I am not an academic, but I use econ all the time, and it works. With financial markets, it works beautifully. It would be incredible to see psychology obtain this level of sophistication and predictive ability.
I would be happy even to see sociology, as an example, simply use the tools of economics and policy analyses. That would help so many people, but they appear to viscerally hate such an idea, even when it would benefit pretty much everyone.
Psychology will be more difficult for obvious reasons.
Yes, I agree that economics is probably the most developed and useful of all the social science disciplines. If all the social sciences hit that level, that would be real progress.
The fact we still can't reach a consensus in the nature vs nurture debate with so much biological dna/genome progress is concerning. Strangely the unpredictability of human nature can be charming albeit dangerous,if psychology does advance and we can make accurate assessments of future behavior we might face a social crisis.
Actually we're making great strides in genetics on this topics, and a consensus is already emerging there around roughly 30-70% genetics vs everything else (which includes different kinds of environments, but also in some sense random chance) in the particular setting of western society. The current problems are as follows:
1)As alluded, the nature-vs-nurture framing is already very bad, which is why it's mostly been discontinued by serious researchers. There are many more environmental options beyond "nurture".
2) social scientists generally take little part in this research, since a substantial number of them take a hard ideological 0% genetics stance on many traits, which is simply implausible. Most geneticists have a reasonable stance that everything is some part nature, some part nurture, which is what we find again and again. A colleague of mine is currently working on certain known inborn disabilities in a collaboration project with social scientists, and their comments are mostly some variant of "we don't believe these disabilities have any genetic background" or "any research into the causes of disabilities is wasted & dangerous ableism, all of this should strictly go into how to support them". It risks toppling the entire collaboration.
3) Explained variance is actually a pretty bad measure. If you compare a person in Africa vs a person in Europe, environment will be so important as to wash everything out. If you compare two not genetically related persons raised in the same family, the differences will be mostly genetics + random chance. This is why there is a movement for sibling-based analysis to first establish facts for one reasonably well controlled case, which then can be tested and extended to further away settings such as maybe cousins at first, and so on.
4) Related to above, but many of the traits we know best are described as "environmentally mediated genetics". Case in point: lactose intolerance/non-persistence. The basic underlying trait, whether you persist in producing lactase and how much you produce, is in itself almost 100% genetics, with only rare exceptions concerning acute illness. However, whether you have any resulting negative repercussions is entirely due to whether you consume food products that include lactose. So if you would analyze downstream traits, such as rates of nausea, for a group of people who all have different degrees of lactose intolerance and who consume different quantities of lactose independent of their intolerance, you will get a result such as "nausea is X% nature and y% nurture" even assuming that you can magically strictly identify the nausea caused by lactose. As such, a much better model is to go beyond the simple nature-vs-nurture percentage points, and towards something like that biology sets up the basic rules and limits, and the environment decides where exactly you end up.
Yes, I agree that one of the fatal flaws in social science is the stubborn refusal to neglect genetics as a causal factor. I discuss this somewhat here and plan to write more on the topic in the future:
One issue in the social sciences generally and psychology in particular is that we're bound by ethics. Many nurture VS nature questions could be answered by withholding nurture from a randomly selected group of people/children. Or altering nature by gene editing, surgery or the like. But fortunately, you will have a hard time convincing the ethics board of permitting this. Thus, in practice, nature and nurture are always present in an interning led fashion to some degree and you have to rely on clever experiments and statistics to tease out results. That said, it seems to me like a lot *has* already been discovered on this topic. It's just that naturally, lots of phenomena are influenced by both aspects, so you don't get a clear winner most of he time.
Thanks for the post, Adam :)! A few thoughts for anyone curious about tackling the 'mysteries' of psychology, based on some ideas I’ve come across:
- Creative Incubation: You mentioned getting great ideas by “doing nothing.” or doing something else. This concept is actually quite well-documented in psychology under what's called 'creative incubation'. It refers to the way certain mental processes seem to be more effective when we’re not consciously focusing on them. Unconscious connections form, and solutions come to the surface when ready. There’s fascinating research on how this process might help in problem-solving.
- Perception of Causality: The study of causality in psychology traces back to Albert Michotte’s work in the 1930/50s. He explored how people intuitively determine when A leads to B—a process we engage in all the time. For example, check out this experiment: https://www.youtube.com/watch?v=VTNmLt7QX8E. Why do people assign causal interpretations to such scenarios? What drives this reasoning? We really have no effort to make up a story here, while these are only objects moving around.
- Psychometrics: What does it mean to measure a psychological variable accurately? The field of psychometrics delves into this, with pioneers like Krantz, Luce, and Tversky. Unfortunately, much of this important work has been sidelined in contemporary psychology. Yet, it’s essential for creating solid, mathematical foundations in the field.
- Beliefs and Input: How do we form beliefs about how the world works based on our surroundings? To what extent do we use logic or rely on heuristics? The work of Gigerenzer and Kahneman provides insightful perspectives, linking experiences to the beliefs they generate.
- Consciousness: When does a mental process become conscious? Some academics like Chalmers and Dehaene have made popular contributions, but this question goes back to psychology’s roots, with Wundt, for instance, touching on it. A definitive answer however still eludes us.
I have to disagree on one point of our text, which is imo critical to understand what is currently going wrong in psychology, but also social science in general: Folk psychology.
Back when physics was in its infancy, it started with quantifying things that people have already noticed. For example, gravity. People had already noticed that things tend to fall down, rather than up, or the movements of celestial bodies. The hard part was generating a specific formula on how it works in-detail. Only *after* considerable hard work we found that intuitions on at least some points were wrong, such as that the force of gravity is actually identical between differently-weighted objects. But many if not most intuitions turned out roughly correct! Likewise for many other things; The basic motivating question for many researchers was extremely often "why is this thing, which we all can perceive and all agree on, the way it is?". This does not preclude eventually overturning some intuitions, but that is generally much further down the line!
In social sciences however, I regularly see even honest students and researchers start from the assumption that all "folk theories" are wrong and they only try to work out how they are wrong. Upturning intuitions is their entire raison d'etre. If you don't get "surprising" results, you're at best boring and uncool or at worst vilified. Best example is imo stereotypes; If I ask friends who literally studied psychology, many will declare with confidence that all stereotypes are wrong and have been thoroughly debunked by psychological research. Yet stereotype accuracy is one of the most robust and replicable findings there is. But psychologists shouldn't stop there! We need to quantify how accurate they are, we need to find how different parameters influence accuracy, and so on. The good news is, this is at least being worked on by a small group of researchers; The bad news is, many others try their best to entirely ignore these findings.
To be frank, you can't do science this way. And your idea of shaking science up may sound lofty and fun, but does nothing to improve this problem. Worse, it falls into the exact same hole.
To make progress in any field, you need orthogonal metrics - a second way of measuring outcomes that isn't correlated with the first. As long as your only metric is surveys, you will be blind to the questions you don't ask, or the weird biases that pop up because of phrasing or inattention or phase of the moon. You'll never be sure that your result was causation, rather than correlation.
If you mix in surveys with an orthogonal metric like EEG or fMRI, when your hypothesis is validated in both metrics you probably have something. The weird biases in one method probably don't interact with the other.
For progress in science, measurement is everything. The weird stultification you see in psychology is because the field has made nearly all the progress that's possible with existing measurement systems. To fix psychology, build cheaper, more precise measurement systems, ideally ones that are independent of what's used today. Do this, and all of the cultural atrophy of the field will spontaneously resolve.
Hi, I did fMRI-based research in the late noughties. Please don't trust anything that comes from there. I stopped paying attention by 2014 or so so maybe there's something not completely worthless from after that?
If the experiment includes only fMRI, yeah, it has a high likelihood of being worthless. Part of the reason is that all those studies were n=6, and everyone doing similar size studies should be the first up against the wall when the revolution comes.
But also, these failures are not the fault of the fMRI itself. Those nuclear spins flip with pretty high reproducibility. It’s the analysis of the fMRI data that’s causing the issues. It’s the interpretation.
If someone shows me a second analysis, independent of the first, that agrees, I’m totally down for the fMRI results. Even if the second technique is also mediocre reliability like the fMRI analysis, if it really is independent, the odds of concordance by chance are low. Not impossible, of course - we still have to deal with bad incentives that lead to fabrication, for example. But man, the problem is smaller.
As a corollary, if you are developing a claim that can only be validated using a single method… maybe we should ignore you in that basis alone. Too many researchers understand the techniques of data collection but not the process of science. We need to layer on a few simple rules to enforce some science on people. My vote that “every interesting claim needs validation by an independent technique” would actually get most of the job done.
what do you think about Ian Hacking’s work? I’m sympathetic to the idea that we’re still in the early days of psychology. But psychology seems categorically different from early chemistry in that people will respond and act differently on the basis of classification - what Hacking calls the “looping effect of human kinds” - while chemical compounds don’t. It’s not simply that there are more variables- the variables are themselves part of the dynamic system we’re trying to understand.
I think this will remain an irreducible complicating factor no matter how much progress is eventually made in psychology.
It's funny you mention that––the rest of the Principe story I started in the post involves him realizing that the reaction is entirely dependent on using iron tools, which Valentine didn't know. The "sulfur of antimony" is ultimately more like the "sulfur of iron." That seems analogous to me: the tools we use to study something are sometimes reactive with the thing we're studying, and this makes studying them more complicated, but not impossible.
Love your work, Adam, but have to take issue with 2 things here - one big & one small (I’m not an academic psychologist; I’m an academic in another field, so I’m not defending my turf).
1. Solving psychiatric illness: Real, important, replicable progress is being made every year. Tens, maybe hundreds, of thousands of people just in the US can live independent and meaningful lives because they take psychiatric medication for serious psychiatric illnesses. This includes major and treatment-resistant depression, treated with TMS or neurostimulators. I recognize that you are writing about psychology and not psychiatry, but: (a) you said that nothing in the DMS has been “cured”; yet people with serious DMS diagnoses *can* be fully treated; and (b) making the argument that psychology is no better now than it was 50 years ago leads to treatment nihilism, which can seriously harm people who then miss out on effective treatments.
2. “Big 5.” It may be that the big five are no more predictive than hocus pocus, but you can’t tell that by looking at whether somebody buys a house in their lifetime. It’s the wrong endpoint. I could predict, on an average basis, who will or won’t buy a house in their lifetime and it doesn’t have anything to do with psychology. I would ask: 1. Are they White? 2. Have they received an inheritance or other chunk of change from a family member? Being white and receiving intergenerational wealth (two variables that are also correlated) are the strongest predictors of home ownership. We should not expect psychology to predict unlike types of social facts.
Thanks again for the great article. Please keep up the great work!
1. I'm glad we're getting better at these things! I think it's important for patients entering psychotherapy to have a realistic idea of what it can do for them, and that comes from being transparent about its abilities as well as its limitations. The more damaging kind of nihilism comes from the practitioners themselves––one of those meta-analyses begins with the author claiming that we shouldn't try to come up with new treatments for depression, because they all work equally well. That's pretty scary to me!
2. I think the response from the Big Five people would be something like: the random luck of being born privileged will affect your outcomes, but personality should matter as well. Some people who are white and inherit money don't end up owning a house, and lots of people who aren't white and don't inherit money *do* end up owning a house, and those differences should be partially explained by their personalities (conscientiousness in particular, I'd imagine). I don't think the Big Five does spectacularly well at predicting that, but I would bet that you'd make better predictions using the Big Five than using race + inheritance alone.
I found that many experts claim that talking therapy does help. One source I found states that:
> Treatment options such as medication and talking therapy play an important role in helping many people with depression and can significantly improve people’s lives.
(https://www.sciencemediacentre.org/expert-reaction-to-a-review-paper-on-the-serotonin-theory-of-depression/)
Why do various experts have totally different views on psychotherapy?
As for studies, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6998909/ found that psychotherapy is effective for patients with depression
Regarding the second point, I agree with Amanda that the examples are not perfect in this context. While the Big 5 don't perform much better than the MBTI in the given context, I'd argue it is because they are at not necessarily are supposed to predict these given outcomes (Maybe at a facet level they would do better? I am not sure). There are however other things the Big 5 are much better at predicting. For example, conscientiousness is a great predictor of academic achievement, rivaling intelligence in certain age groups (e.g. Poropat, 2009). I am not aware of any studies indicating that any of the MBTI score are a predictive of academic achievement.
Overall, thanks for writing this article. You verbalised many things that left me frustrated with psychology during my PhD and still bother me today. This has helped me pin-point more precisely where exactly my frustration stems from.
Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135(2), 322–338. https://doi.org/10.1037/a0014996
Also, the psychology of addiction and the bio-psychology of early-life adverse events and their relationship to future addiction (among other things) represent demonstrable advances over folk psychology, with important, actionable outcomes for treatment, prevention, and policy.
Friendly challenge: Take a look into that subfield, see if you agree, and, if so, see what’s the same or different about systems, questions, funding or mentorship structures, or other factors.
And yet, I can find no demonstrable improvement in our ability to prevent or cure addiction. I'm not convinced any of this is real. If you can find a real application explicitly based on this research, that has consistently worked, please point me in that direction.
Considering *gestures at the encampments outside*, I think you’d do well to try to justify *your * claim that we’ve figured something out about addiction rather than sending readers off to dig in a no-doubt vast literature to which you’ve given not even a hint of guidance.
Hi, Jerome. My comment is to and for Adam, an academic psychologist. He should be familiar enough with the field to know how to begin, particularly since he’s saying here that he has reviewed the entire field of empirical psychology and found little evidence it has produced any contributions. If you’re genuinely curious, I recommend you start with the work of Bruce McEwen, from Rockefeller University. Though McEwen was a biological psychiatrist, his work on early life stress and allostatic load (both in relation to addiction and beyond) has profoundly shaped work in psychology, psychiatry, and policy.
Thank you for the suggestion, but the conversation is about psychology’s real-world impacts. If it’s necessary to dig into the primary literature to ferret out unspecified impacts of 30-year-old work, which has thus had an entire generation to flesh itself out in treatments, then the claim is self-defeating. At the heart of the post here is that mature sciences can demonstrate impacts legible to non-participants; it seems clear that you can’t do that along this line, not to mention the fact that it’s very far from clear that other “academic psychologists” find your argument any more convincing than I do.
Regarding allostatic load, I find: “Interventions can include encouraging sleep quality and quantity, social support, self-esteem and wellbeing, improving diet, avoiding alcohol or drug consumption, and participating in physical activity.” I hope you’ll forgive my lack of astonishment at the level of impact on psychiatric practice reflected here.
Right, great then, it should be easy for you to find somewhere where this research has demonstrably improved people's lives.
I was interested to see you list Anki as a success at “cramming it into a box,” because I’ve spent about ten years developing, supporting, and writing about spaced-repetition systems in some capacity (including Anki), and I’ve often written about how using memory tools well is actually very complicated, noting that if only we could find a prescriptive set of simple rules you could follow (like you could encode into software) that would consistently work, it could finally take off!
Anki does do a good job at packaging up the insight of spaced repetition itself. Unfortunately, that is only a tiny part of the problem of learning/remembering things, and solving this restricted problem leaves most of the real problem unsolved. Just a few common and important problems it doesn't solve:
* Lack of motivation / habit – people forget to study, don't want to study, don't study for a few days and then come back and have a huge pile of cards and give up, etc. (This is moderately tractable – I think RemNote, the startup I’m currently involved in, does much better than Anki does here – but habits are just hard, and there’s only so much you can do to help people remember things if they don’t review them!)
* Learning something that's useful / you care about knowing – it's super common for people to use Anki to memorize, say, all the symbols on the periodic table, and then if you ask them why they wanted to know that, they have no idea.
* Writing cards that are unclear – oftentimes when you write a card (or write one for someone else to import), you think you know what it’s asking, but when you actually see it out of context 3 months later it is inscrutable, or makes you think you were supposed to give some other answer instead. In the best case, this is frustrating and messes up the spacing algorithm because you can’t accurately evaluate how well you remembered the content once you fix it. More commonly, people who don’t have a lot of experience don’t even notice why they didn’t know the answer, and keep forgetting the card over and over again without ever doing anything to fix it.
* Learning the thing the way you need to remember it – it’s really easy to memorize a card based on, say, some unusual wording on the front of it, and find yourself unable to recall the information when given a real-life cue. Or you might just ask the question in a way that doesn’t match how you’ll be asked to recall it, and find you just can’t quite make the connection when you are. (Memory is much more sensitive to tiny shifts in context than most people realize.)
* Keeping cards relevant / understanding why you’re struggling – it’s common for cards to become stale because you no longer care about the information on them, or you don’t understand what they were supposed to mean, so you struggle to remember them, and you just keep repeating them over and over again until you get frustrated.
Maybe the core issue is that the apparently simple problem of "remembering question/answer pairs" is deeply tangled up with all sorts of other complicated human stuff that prevents you from actually effectively using the tool unless you also solve those other problems (or at least partially address them). I appreciate the claim that the “people are just complex” argument is unimaginative, but I suspect this may be where psychology really does diverge from, say, physics. Sure you're ignoring all kinds of factors when you, say, model the trajectory of a ball you throw. But with just a couple of equations, you still get answers that are accurate enough to do amazing things to a pretty decent degree of accuracy – you can really forget about a lot of the adjacent questions (e.g., relativity, or often even air resistance). Whereas if you just hand someone Anki, it’s still better than them not having it, but the vast majority of people get resoundingly mediocre results, because of all these adjacent issues, and there's not an obvious way to cram solutions to those issues into the box, because they deeply depend on parts of your internal mental state that can't be reliably introspected. This is frustrating, so most people give up, even though the spaced-repetition tool itself is extremely powerful.
Now I guess you could argue something similar about the ICE. Like of course having a great ICE doesn’t mean people will become good drivers, that's a separate problem to solve, and that shouldn’t diminish the beauty of putting combustion in a box. But there’s some way in which dismissing it like this feels unsatisfying to me – with the ICE that seems so obvious as to be absurd, while with spaced repetition it really feels to me like the fundamental problem is basically unsolved, and we’ve only handled some tiny restricted subproblem. Is it just that we think about psychology differently? Or that the path to making cars safer is clearer than the path to understanding what cards will change someone's mental state into being a person they prefer to be? I’m not sure.
(I have a bunch of blog posts about addressing some of these issues with memory tools – by developing personal skills, not by improving the tools – for anyone curious. You can start at https://controlaltbackspace.org/precise/.)
Super interesting, thank you! I agree that we haven't solved the problem of remembering things and boxed it in Anki––we've figured out two things (spaced repetition and testing are betting than nothing) and boxed those. I think the reason we haven't figured out the rest is because those problems are harder to see in the first place. Nobody booting up Anki thinks to themselves, "I probably need help understanding why I want to memorize these things in the first place." If we knew why people try to memorize things they don't actually care about, we might be able to build a solution into the box.
I think your link got messed up a little, but this works: https://controlaltbackspace.org/precise/
Hmm, it looks normal and works for me, but thanks!
Huh, somehow it's fine on the website. But in the mobile app, the link URL goes up to the closing parenthesis.
I live outside of Cincinnati. And maybe the issue here is, you’re thinking of Cincinnati as a destination, rather than a heading.
If the heading is looked at through the classical lens of, “how can I live excellently”, we might instead focus on the things that top performers do differently. Talk to, for example, any professional athletes about the importance of attitude. Or visualization. Or the dangers of resentment. Or the necessity of relaxation, or the power of routine. These people will do anything they can for an edge.
Instead of studying random psychologists, study people like Darryl Davis, a black man who converted numerous members of the KKK, to abandon the group, mainly by listening to them and earning their respect. See how successful FBI hostage negotiators think about conflict, and learn from them, instead of studying random people who for the most part aren’t super functional or happy.
You’ll find that people who thrive in highly competitive fields do things very differently from ordinary people, using what I’d say are toolkits developed from psychological knowledge, a combination of folk psychology and actual empirical evidence. Their success is evidence that they understand reality.
If you want to be really heterodox, really think outside the box, try floating the idea that some ways of thinking actually work better than others. It’s heresy in academia to say you think this is true of human metanarratives. Instead of setting “mere understanding” as the goal, make the goal pursuit of excellence and see what comes of it. Study the great teachers of the past and try to reverse engineer their logic. Taking seriously claims by numerous faiths, see if you can find their intersection and then understand why those things work. Study very successful people and see what makes their belief structures different. See if you can predict future success from high schoolers based on narratives like “the world is rigged against me” vs “if I try hard I will succeed” vs “if I learn a valuable skill I’ll be financially ok”
I predict your colleagues will disparage you for the audacity to believe some belief structures really are better than others, and this maybe explains why the field is stuck.
Cincinnati is named, by the way, after a Roman dictator who was asked by the senate to stick around in power, since he did a good job in an emergency. His response was, “no, I’d rather go tend my farm.” This response strikes most people today as alien, but I think the ethos pervades the city and its surroundings: family and community are more important than anything else. This is, I think, what makes it work.
Interesting ideas! Of course, we have to be ultra-careful here not to get spurious results due to survivorship bias. So, just by analyzing the techniques of successful people, you won't yet have a causal result (mainly since you don't know how many people were using the same techniques but failed in the end). If you try to run experiments with the knowledge gathered afterwards, then we're talking!
As for comparing success of high-schoolers: Getting causality would again be difficult here. If the students have these attitudes prior to the experiment, then there are reason for that - financial situation, racism, (lack of) luck, generalizing from personal experience, trauma, whatever. As long as the source isn't "I have randomly acquired this attitude", you will have confounders which may be very difficult to control for.
What you could try is to instill these attitudes - perhaps just the ones we expect to yield positive results so as not to farr unethically - with a control group receiving no such input. Maybe a motivational speech or something. I'd definitely be interested to read the results of such a study.
FBI hostage negotiators are using specialized tools that most psychologists don't have, called 'firearms'. Davis, on the other hand, is using a classic tool of all good psychological research known as "making the whole thing up".
I have an urge to write a smarter, more informed comment. But it is dinnertime and the site of Cincinnati Chili - and I grew up in Cincy - is instead triggering far more primal hunger urges. Darn right in Cincy you get to eat that chili, and it is the thing I miss most about no longer living there.
I desire profundity; I find hunger. Such is life.
It seems to me that psychology is caught between two opposing forces:
1. There's ample evidence anywhere you care to look that deliberate or accidental intervention can dramatically alter human behavior: take for example cults, revolutionary movements, totalitarian governments, whatever it was that worked to mostly get people to stop smoking cigarettes.....
2. Attempts by the discipline to turn these examples into a set of systematic principles that can be applied according to some underlying theory have been at best underwhelming and at worst, catastrophically bad.
Maybe it's worth digging into how both these things can be true....?
I think it means that we haven't gotten the right principles, probably we have the wrong idea about how principles get generated. I'm planning to write more about this in the future.
Psychologists* like to think they've debunked loads of folk assumptions, but I really don't think they have. The Milgram study doesn't challenge anything we previously assumed about human behaviour - we've known for thousands of years that most people defer to authority figures, most human organisations are built on this fact. What Milgram challenged was more like our post-WW2 Liberal/individualist assumptions about ourselves. It shocks us because we think that *we* wouldn't keep pressing the button, and the results indicate that we probably would.
Folk wisdom has stood the test of time for a reason. If we want to improve psychology, I think we should start by trying to understand why.
*(I can make these sorts of generalisations, I'm one myself)
Daniel Lakens has suggested that social science research could benefit from greater coordination. Physics, for example, has quite a centralised model because of how expensive the equipment is — they have to meet up and agree "okay, US you look at these questions; Europe, look at these questions, etc".
Social science by contrast is very decentralised. Incentive is to pursue research that is likely to turn up a publishable result, even if it's unlikely to advance our understanding very much. If the field were more coordinated, perhaps that may allow psychologists to build up big theories and pursue risky studies that take decades but produce valuable insights.
What do you think of this argument?
I think coordination may be a symptom of progress rather than a cause. The advantage of a paradigm is that it brings some consensus about which problems are important to solve in the first place, so they can be divided and conquered. I don't think there is a justifiable consensus like that right now in psychology.
Fascinating article!
One point of critique: Your 5th criterion of the old psych theories not being considered absurd seems like a pretty unfair comparison to me. You compare the humors, a theory from about 2500 years ago[1] (which could only be disproven once science proper arrived, I.e. about 200 years ago) to the ego/id/superego theory from about 100 years ago[2].
I don't know about the (history of) psychology to provide a counterexample if one exists, perhaps someone else knows more? Though obviously it is tough to prove that NO psychological perspective/terminology has died out like the humor concepts did. Additionally, perhaps the id/ego/superego model remains in use *because* it captures enough of a real phenomenon to be of use. I'm sure there is some part of biological or medical understanding from 100 or even 1000 years ago that is still in active use today.
Overall, I'm having difficulty seeing strong support for psychology failing this criterion.
[1] https://en.m.wikipedia.org/wiki/Humorism
[2] https://en.m.wikipedia.org/wiki/Id,_ego_and_superego
I mean the humors example to show how a paradigm shift can show up as a change in terminology, rather than to compare the amount of progress in medicine vs. psychology. I think the id/ego/superego example is illustrative because it's some of the earliest work that we would consider psychology, and it's still influential. And I think you're right about the reason for that: we haven't yet discovered something that's obviously better.
jesus, this needs a content warning for the skyline chili jumpscare
Setting aside the majority of the article, the ending about mysteries struck me the most. We psychologists are really good in digging up concepts and naming things but we are not really good - not at all - in explaining why these concepts come about and why and how do they hang together, and most especially - when and where do these even apply. Psychology puts, by definition too much emphasis on the individual. But "individuals" are actually entangled jumbles of other people, varied context and settings, their history, etc. As a result, you kind of end up throwing spaghetti on the wall - megastudy style - whenever you want to change or reinforce a "simple" behaviors such as gym attendance or vaccination. In other words, what psychology is missing are explanatory mechanisms and how these are reinforced or dampened by varied contexts.
Linked to that is that, implicitly, the aim of psychology is too damn high: under a positivist paradigm, where - to be called science - we have to use RCTs, pay attention to concepts like internal validity and replicability and generalizability, and use increasingly more and more arcane statistical techniques, psychology, as a science, is found wanting. But I think this stems partially from wanting too much: if we assume that any trial findings are context-bound (irrespective of whether you've calculated a sample size for your RCT to be representative of the underlying population), we can dismiss generalizability and replicability as concepts that don't apply to psychology. If we also agree that any finding found in the real world is much more valuable than one found in artificial settings (hello, MTurk), we put ecological validity over internal validity. By doing so, also, we acknowledge the wobbliness we're dealing with in psychology: important things such as emotions, meanings, consciousness-related stuff, don't easily - or at all - yield to measurement. If we abandon all that, I have a sense we would actually start making some progress and who knows, maybe one day a future psychologist will finally tell me why people enjoy watching other people cook.
Here's a folk explanation for you.
The problem is that psychologists assume, rashly, that people can be described at scale. Now, in part this is a methodological artifact, the drunk's streetlight: for various reasons, the Big Five is way more tractable as an instrument, as a fundamental theory, etc., so that's the block you look on even if you lost the keys in the dark block a street over.
I think that in practice the Big Five might be better as the Big 300. I'm a million years away from being able to codify this, but my assuming it's true has led to some extremely interesting discoveries and successes in my work identifying customers who are WAY more likely to buy your thing. It's a Pareto, like everything else, only it's not a univariate factor making up that Pareto. It's a bunch. But even identifying one or two of them is bringing a machine gun to the marketing stick fight. And I'm not particularly interested in "proving it at scale." That's not the area I work in, and it's the wrong approach anyway.
Thing is, a lot of folk stuff works. A lot of success is being just a little bit ahead of other people, and "empirically verifiable" is usually not the name of the track you're running on. Now, if you have evidence it *is* empirically verifiable (vaccines), don't be a fool. But that's not usually relevant to the problems you're facing.
Interesting article. It takes a brave man to review all the accumulated wisdom gathered from his own discipline and conclude “we have not really learned much.”
Unfortunately, this seems to be true of most if not all social sciences. Not sure where that leaves us…
Economics has come a long way. I am not an academic, but I use econ all the time, and it works. With financial markets, it works beautifully. It would be incredible to see psychology obtain this level of sophistication and predictive ability.
I would be happy even to see sociology, as an example, simply use the tools of economics and policy analyses. That would help so many people, but they appear to viscerally hate such an idea, even when it would benefit pretty much everyone.
Psychology will be more difficult for obvious reasons.
Yes, I agree that economics is probably the most developed and useful of all the social science disciplines. If all the social sciences hit that level, that would be real progress.
The fact we still can't reach a consensus in the nature vs nurture debate with so much biological dna/genome progress is concerning. Strangely the unpredictability of human nature can be charming albeit dangerous,if psychology does advance and we can make accurate assessments of future behavior we might face a social crisis.
Actually we're making great strides in genetics on this topics, and a consensus is already emerging there around roughly 30-70% genetics vs everything else (which includes different kinds of environments, but also in some sense random chance) in the particular setting of western society. The current problems are as follows:
1)As alluded, the nature-vs-nurture framing is already very bad, which is why it's mostly been discontinued by serious researchers. There are many more environmental options beyond "nurture".
2) social scientists generally take little part in this research, since a substantial number of them take a hard ideological 0% genetics stance on many traits, which is simply implausible. Most geneticists have a reasonable stance that everything is some part nature, some part nurture, which is what we find again and again. A colleague of mine is currently working on certain known inborn disabilities in a collaboration project with social scientists, and their comments are mostly some variant of "we don't believe these disabilities have any genetic background" or "any research into the causes of disabilities is wasted & dangerous ableism, all of this should strictly go into how to support them". It risks toppling the entire collaboration.
3) Explained variance is actually a pretty bad measure. If you compare a person in Africa vs a person in Europe, environment will be so important as to wash everything out. If you compare two not genetically related persons raised in the same family, the differences will be mostly genetics + random chance. This is why there is a movement for sibling-based analysis to first establish facts for one reasonably well controlled case, which then can be tested and extended to further away settings such as maybe cousins at first, and so on.
4) Related to above, but many of the traits we know best are described as "environmentally mediated genetics". Case in point: lactose intolerance/non-persistence. The basic underlying trait, whether you persist in producing lactase and how much you produce, is in itself almost 100% genetics, with only rare exceptions concerning acute illness. However, whether you have any resulting negative repercussions is entirely due to whether you consume food products that include lactose. So if you would analyze downstream traits, such as rates of nausea, for a group of people who all have different degrees of lactose intolerance and who consume different quantities of lactose independent of their intolerance, you will get a result such as "nausea is X% nature and y% nurture" even assuming that you can magically strictly identify the nausea caused by lactose. As such, a much better model is to go beyond the simple nature-vs-nurture percentage points, and towards something like that biology sets up the basic rules and limits, and the environment decides where exactly you end up.
Yes, I agree that one of the fatal flaws in social science is the stubborn refusal to neglect genetics as a causal factor. I discuss this somewhat here and plan to write more on the topic in the future:
https://frompovertytoprogress.substack.com/p/social-mobility-vs-upward-mobility
One issue in the social sciences generally and psychology in particular is that we're bound by ethics. Many nurture VS nature questions could be answered by withholding nurture from a randomly selected group of people/children. Or altering nature by gene editing, surgery or the like. But fortunately, you will have a hard time convincing the ethics board of permitting this. Thus, in practice, nature and nurture are always present in an interning led fashion to some degree and you have to rely on clever experiments and statistics to tease out results. That said, it seems to me like a lot *has* already been discovered on this topic. It's just that naturally, lots of phenomena are influenced by both aspects, so you don't get a clear winner most of he time.
Thanks for the post, Adam :)! A few thoughts for anyone curious about tackling the 'mysteries' of psychology, based on some ideas I’ve come across:
- Creative Incubation: You mentioned getting great ideas by “doing nothing.” or doing something else. This concept is actually quite well-documented in psychology under what's called 'creative incubation'. It refers to the way certain mental processes seem to be more effective when we’re not consciously focusing on them. Unconscious connections form, and solutions come to the surface when ready. There’s fascinating research on how this process might help in problem-solving.
- Perception of Causality: The study of causality in psychology traces back to Albert Michotte’s work in the 1930/50s. He explored how people intuitively determine when A leads to B—a process we engage in all the time. For example, check out this experiment: https://www.youtube.com/watch?v=VTNmLt7QX8E. Why do people assign causal interpretations to such scenarios? What drives this reasoning? We really have no effort to make up a story here, while these are only objects moving around.
- Psychometrics: What does it mean to measure a psychological variable accurately? The field of psychometrics delves into this, with pioneers like Krantz, Luce, and Tversky. Unfortunately, much of this important work has been sidelined in contemporary psychology. Yet, it’s essential for creating solid, mathematical foundations in the field.
- Beliefs and Input: How do we form beliefs about how the world works based on our surroundings? To what extent do we use logic or rely on heuristics? The work of Gigerenzer and Kahneman provides insightful perspectives, linking experiences to the beliefs they generate.
- Consciousness: When does a mental process become conscious? Some academics like Chalmers and Dehaene have made popular contributions, but this question goes back to psychology’s roots, with Wundt, for instance, touching on it. A definitive answer however still eludes us.
I have to disagree on one point of our text, which is imo critical to understand what is currently going wrong in psychology, but also social science in general: Folk psychology.
Back when physics was in its infancy, it started with quantifying things that people have already noticed. For example, gravity. People had already noticed that things tend to fall down, rather than up, or the movements of celestial bodies. The hard part was generating a specific formula on how it works in-detail. Only *after* considerable hard work we found that intuitions on at least some points were wrong, such as that the force of gravity is actually identical between differently-weighted objects. But many if not most intuitions turned out roughly correct! Likewise for many other things; The basic motivating question for many researchers was extremely often "why is this thing, which we all can perceive and all agree on, the way it is?". This does not preclude eventually overturning some intuitions, but that is generally much further down the line!
In social sciences however, I regularly see even honest students and researchers start from the assumption that all "folk theories" are wrong and they only try to work out how they are wrong. Upturning intuitions is their entire raison d'etre. If you don't get "surprising" results, you're at best boring and uncool or at worst vilified. Best example is imo stereotypes; If I ask friends who literally studied psychology, many will declare with confidence that all stereotypes are wrong and have been thoroughly debunked by psychological research. Yet stereotype accuracy is one of the most robust and replicable findings there is. But psychologists shouldn't stop there! We need to quantify how accurate they are, we need to find how different parameters influence accuracy, and so on. The good news is, this is at least being worked on by a small group of researchers; The bad news is, many others try their best to entirely ignore these findings.
To be frank, you can't do science this way. And your idea of shaking science up may sound lofty and fun, but does nothing to improve this problem. Worse, it falls into the exact same hole.
To make progress in any field, you need orthogonal metrics - a second way of measuring outcomes that isn't correlated with the first. As long as your only metric is surveys, you will be blind to the questions you don't ask, or the weird biases that pop up because of phrasing or inattention or phase of the moon. You'll never be sure that your result was causation, rather than correlation.
If you mix in surveys with an orthogonal metric like EEG or fMRI, when your hypothesis is validated in both metrics you probably have something. The weird biases in one method probably don't interact with the other.
For progress in science, measurement is everything. The weird stultification you see in psychology is because the field has made nearly all the progress that's possible with existing measurement systems. To fix psychology, build cheaper, more precise measurement systems, ideally ones that are independent of what's used today. Do this, and all of the cultural atrophy of the field will spontaneously resolve.
Hi, I did fMRI-based research in the late noughties. Please don't trust anything that comes from there. I stopped paying attention by 2014 or so so maybe there's something not completely worthless from after that?
If the experiment includes only fMRI, yeah, it has a high likelihood of being worthless. Part of the reason is that all those studies were n=6, and everyone doing similar size studies should be the first up against the wall when the revolution comes.
But also, these failures are not the fault of the fMRI itself. Those nuclear spins flip with pretty high reproducibility. It’s the analysis of the fMRI data that’s causing the issues. It’s the interpretation.
If someone shows me a second analysis, independent of the first, that agrees, I’m totally down for the fMRI results. Even if the second technique is also mediocre reliability like the fMRI analysis, if it really is independent, the odds of concordance by chance are low. Not impossible, of course - we still have to deal with bad incentives that lead to fabrication, for example. But man, the problem is smaller.
As a corollary, if you are developing a claim that can only be validated using a single method… maybe we should ignore you in that basis alone. Too many researchers understand the techniques of data collection but not the process of science. We need to layer on a few simple rules to enforce some science on people. My vote that “every interesting claim needs validation by an independent technique” would actually get most of the job done.
what do you think about Ian Hacking’s work? I’m sympathetic to the idea that we’re still in the early days of psychology. But psychology seems categorically different from early chemistry in that people will respond and act differently on the basis of classification - what Hacking calls the “looping effect of human kinds” - while chemical compounds don’t. It’s not simply that there are more variables- the variables are themselves part of the dynamic system we’re trying to understand.
I think this will remain an irreducible complicating factor no matter how much progress is eventually made in psychology.
It's funny you mention that––the rest of the Principe story I started in the post involves him realizing that the reaction is entirely dependent on using iron tools, which Valentine didn't know. The "sulfur of antimony" is ultimately more like the "sulfur of iron." That seems analogous to me: the tools we use to study something are sometimes reactive with the thing we're studying, and this makes studying them more complicated, but not impossible.