A landmark study that claimed men enjoy an unfair advantage in scientific careers has been seriously challenged: a nearly identical rerun of the experiment has found that the opposite is true.
The results are all the more striking because a leading scientific journal had refused to repeat the original experiment, raising concerns that some researchers are reluctant to scrutinise results that align with their views.
The original study, published in 2012, has been cited 4,600 times — an enormous figure in a field where most papers attract fewer than 20 citations. It involved science professors being sent a fictional CV for a lab manager job.
In half of CVs, the applicant was named “John”, while for the other half the applicant was named “Jennifer”. Everything else, from the candidate’s academic grades to work experience, was identical.
The professors who saw the male name rated him as more competent, more hireable and more deserving of mentoring and a higher salary. The finding is often mentioned in debates about the under-representation of women in science, technology, engineering and mathematics (Stem) careers.
The new paper repeated the test on a far larger scale. Whereas the original had asked about 130 academics at six universities, the authors of the new paper — led by Nathan Honeycutt and Lee Jussim of Rutgers University in New Jersey — contacted nearly 1,300 professors from more than 50 American research institutions, using the same application materials and the same measures of perceived competence, hireability, likeability and salary recommendations.
The aim was to see whether the result of the original paper held when the experiment was repeated. It did not. The female applicant was seen as marginally more capable and appealing to work with and the more hireable of the pair. She was also seen as worth a bigger salary — $35,550 versus $34,150 for the man. The differences were small, but consistent. The widely cited bias against women failed to reappear; it now tilted the other way.
According to the authors of the new study, the resistance they encountered when they suggested repeating the experiment may be as telling as the result. Honeycutt said that he and his co-authors were taken aback when they proposed a rerun to Nature Human Behaviour, a leading journal.
When Honeycutt and his colleagues submitted a registered replication report (RRR) — a formal proposal to repeat the study — it was rejected by a panel of peer reviewers acting for the journal.
Honeycutt said his impression was that the reviewers approved of the original findings and feared they might not hold up. “We can’t know for certain — but [that is our suspicion] given just the nature of their feedback and pushback,” he said.
Writing on the Substack website, Jussim added: “If reviewers like [a study] and its results and they anticipate it won’t replicate, they can concoct serious scientific-sounding reasons to reject the RRR. Which … is exactly what happened at [Nature Human Behaviour].” He argued that the process reflected “a profoundly antiscientific attitude”.
One objection raised by the reviewers was that professors might already know about the original 2012 paper and would therefore give “tainted” responses. Honeycutt and Jussim proposed a solution: they would ask participants if they were familiar with the earlier study and analyse the data accordingly. The suggestion was dismissed.
Jussim later wrote that the reviewers’ objections in effect meant the original study “could never be replicated”, implying its conclusions “must stand intact, for ever”.
His team took the project elsewhere, preregistered their methods and ran the work regardless. Most professors, it turned out, had not heard of the 2012 paper. Their results have now been accepted by the journal Meta-Psychology.
How far the new results reflect the wider state of equality in science is debatable. Women earn about half of all doctorates in the life sciences and a growing share in fields such as chemistry and engineering. However, they remain in the minority in senior academic posts. In Britain, only about a quarter of professors in Stem are women. The study looked only at one type of relatively junior job, that of a laboratory manager.
• ‘Labs aren’t designed for people like me’
Erika Pastrana, vice-president of the Nature Research Journals portfolio, said: “Decisions by our editors to accept or reject replication studies are based solely on whether the research meets our editorial criteria, including standards for methodological rigour. They are not influenced by whether the original study’s conclusions may or may not be replicated. Our decisions are not driven by a preferred narrative.”
She added: “We do not agree that a study becomes immune to scrutiny if exact replication is difficult or not pursued. Scientific scrutiny occurs through many mechanisms, including replication, critical commentary, meta-analyses, methodological debate, and new research that builds on or challenges prior work.”