The Challenge of Replicability in Social Science Research
Your editorial on social science research highlights the persistent issue of poor replicability in results, a problem often exploited by critics to dismiss the entire field. In the complex realm of human behavior, factors such as flawed methodology, misapplied statistics, and sample variations contribute to this replicability crisis. However, a deeper, less-discussed factor is the lack of systematic observation of human behavior in natural, everyday environments, akin to how scientists study other species to understand their actions.
The Problem with Cultural Terms in Scientific Inquiry
This observational gap is understandable, given that human cultures possess a rich tapestry of knowledge about behavior, actions, and descriptive language, which has been crucial for the success of Homo sapiens. Social sciences frequently rely on these cultural terms, but they are ill-suited for scientific purposes. Unlike the natural sciences, which develop their own terminology from scratch because their subjects lack inherent language, social sciences often use terms evolved for "living forward" to "understand backwards," as Søren Kierkegaard noted in 1843.
This reliance on the "active insider" perspective of cultural participants, rather than developing objective "outsider" terms through observation, undermines scientific rigor. Cultural terms are inherently subjective, change over time, and vary across societies, making research couched in such slippery, action-oriented language prone to replication failures. Dr. John Richer from Oxford emphasizes that this approach reduces motivation for the detailed observation needed for true scientific understanding.
Call for Better Tools and Data in Social Sciences
Your editorial rightly advocates for weighing individual studies against broader evidence bases, with serious policymaking increasingly depending on systematic reviews that synthesize all relevant evidence transparently. There is optimism that human behavior and societies represent the last great frontier of discovery, potentially more rewarding than studying stars, oceans, or the human body. However, current social science tools are likened to Galileo's telescope compared to modern space observatories that generate trillions of daily observations.
Data is identified as the essential fuel for progress, not only in natural sciences but also in social sciences and artificial intelligence. Language models must be complemented by world models and people models to fulfill their potential. For scientific advancement and effective governance, there is a pressing need to invest in public data with vastly improved coverage, speed, volume, and detail. Will Moy, Chief Executive of the Campbell Collaboration, argues that such data will serve as the raw material for both scientific breakthroughs and better government policies.
Incentivizing Peer Review to Enhance Research Robustness
Thank you for your editorial, which brings nuance and context to the replicability issue, noting how political groups might misuse low replication rates to delegitimize scientific evidence. While shifting incentives to test existing results is key, there is optimism about practical solutions. Currently, scientists are hired and promoted based on authored works, with peer review contributions largely ignored.
Recognizing researchers' peer review activity could incentivize better practices. Tools like Web of Science researcher profiles and ORCID already track reviewing efforts; updating this system to award bonus points for excellent reviews and penalize poor ones would encourage researchers to invest time in spotting specious results before publication. This would also aid editors in finding suitable reviewers, strengthening the double-blind peer review system as humanity's best method for identifying replicable truths, as noted by Prof. David Comerford, Programme Director of MSc Behavioral Science at the University of Stirling.
Have an opinion on this topic? Share your thoughts for potential publication in our letters section.
