The security of our secrets: A history of privacy and confidentiality in law and statistical practice

United States statistical programs and practices are among the best in the world. Lurking underneath this success, however, is a riddlegiven the potential for abuse and harm, why do Americans willingly hand over their personal information to government data collection programs? In this article, we seek an answer to this riddle by examining the evolution of United States legal and statistical programs, with a particular focus on the United States Census of Population. In so doing, we explore the statistical programs, policies, regulations, and codes of ethics that have evolved in the United States over the past two centuries. We conclude that the willingness of individuals to disclose their personal information is not linked to programs of legal coercion or to simple cost/benefit analyses. Instead, we note that the intent of United States statistical programs has been to increase the level of trust and confidence that individuals have that their information will be kept strictly confidential. Various legal frameworks and the promulgation of statistical society codes of ethics buttress our basic conclusion that trust is an essential characteristic of a successful and efficient modern statistical program. We conclude by noting some recent developments that may threaten this trust program, including post 9/11 national security efforts, the rise of new data-gathering and analysis technologies, and the in creasing use of private data collectors for government statistical programs.