Auditing Algorithms: Understanding Algorithmic Systems from the Outside In
By Danaë Metaxa, Stanford University, USA, metaxa@cs.stanford.edu | Joon Sung Park, Stanford University, USA, joonspk@stanford.edu | Ronald E. Robertson, Northeastern University, USA, robertson.ron@northeastern.edu | Karrie Karahalios, University of Illinois at Urbana-Champaign, USA, kkarahal@illinois.edu | Christo Wilson, Northeastern University, USA, cbw@ccs.neu.edu | Jeff Hancock, Stanford University, USA, hancockj@stanford.edu | Christian Sandvig, University of Michigan, USA, csandvig@umich.edu
Abstract
Algorithms are ubiquitous and critical sources of information online, increasingly acting as gatekeepers for users accessing or sharing information about virtually any topic, including their personal lives and those of friends and family, news and politics, entertainment, and even information about health and well-being. As a result, algorithmically-curated content is drawing increased attention and scrutiny from users, the media, and lawmakers alike. However, studying such content poses considerable challenges, as it is both dynamic and ephemeral: these algorithms are constantly changing, and frequently changing silently, with no record of the content to which users have been exposed over time. One strategy that has proven effective is the algorithm audit: a method of repeatedly querying an algorithm and observing its output in order to draw conclusions about the algorithm’s opaque inner workings and possible external impact. In this work, we present an overview of the algorithm audit methodology, including the history of audit studies in the social sciences from which this method is derived; a summary of key algorithm audits over the last two decades in a variety of domains, including health, politics, discrimination, and others; and a set of best practices for conducting algorithm audits today, contextualizing these practices using search engine audits as a case study. Finally, we conclude by discussing the social, ethical, and political dimensions of auditing algorithms, and propose normative standards for the use of this method.
Auditing Algorithms: Understanding Algorithmic Systems from the Outside In
Algorithms are ubiquitous and critical sources of information online and increasingly act as gatekeepers for users accessing or sharing information about virtually any topic. This includes information about their personal lives and those of friends and family, news and politics, entertainment, and even health and well-being. As a result, algorithmically-curated content is drawing increased attention and scrutiny from users, the media, and lawmakers alike. Studying such content poses considerable challenges as it is both dynamic and ephemeral. One strategy that has proven effective is the algorithm audit: a method of repeatedly querying an algorithm and observing its output in order to draw conclusions about the algorithm’s opaque inner workings and possible external impact.
In this work, the authors present an overview of the algorithm audit methodology. They include the history of audit studies in the social sciences from which this method is derived; a summary of key algorithm audits over the last two decades in a variety of domains such as health, politics, and discrimination; and a set of best practices for conducting algorithm audits today. The authors conclude by discussing the social, ethical, and political dimensions of auditing algorithms, and propose normative standards for the use of this method.
