Position Paper: Replication Crisis in Human-Centered Security Research—Are We There Yet?

Summary

Replicability is critical for meta-science, verifying scientific knowledge, and advancing science as a whole. However, the broader security community and the human-centered security (HCS) research community, in particular, struggle with making studies replicable, and only a few replication studies have been published so far. In this position paper, we analyze Call for Papers (CfPs) and existing replication studies to identify pain points that hinder replication in HCS research, use those insights to discuss whether the community is already in a replication crisis, and assess the extent to which this contributes to or hinders meta-science in the field. We conclude with calls to action to improve reporting transparency and data sharing, better recognize replication studies as valuable scientific contributions, and develop community-driven understandings of how replications should be conducted for HCS research.