Replication Code for: Measuring Racial Discrimination in Algorithms

Principal Investigator(s): View help for Principal Investigator(s) David Arnold, UCSD; Will Dobbie, Harvard Kennedy School; Peter Hull, University of Chicago

Version: View help for Version V1

Name	File Type	Size	Last Modified
replication			02/01/2021 03:22:PM

Project Citation:

Arnold, David, Dobbie, Will, and Hull, Peter. Replication Code for: Measuring Racial Discrimination in Algorithms. Nashville, TN: American Economic Association [publisher], 2021. Ann Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 2021-04-28. https://doi.org/10.3886/E131362V1

Project Description

Summary: This Stata and R code replicates the analysis in Arnold, Dobbie, and Hull (2021). The data for this paper contains confidential information about criminal defendants in New York City and so is restricted-use. Accessing the data can be done by entering a data sharing agreement with the New York State Division of Criminal Justice Services and Office of Court Administration. Inquiries can be sent to:

DCJS Research Request Team

Office of Justice Research and Performance, New York State Division of Criminal Justice Services 80 South Swan St., Albany, NY 12210

DCJS.ResearchRequests@dcjs.ny.gov

www.criminaljustice.ny.gov

An abstract of the project follows

There is growing concern that the rise of algorithmic decision-making can lead to discrimination against legally protected groups, but measuring such algorithmic discrimination is often hampered by a fundamental selection challenge. We develop new quasi-experimental tools to overcome this challenge and measure algorithmic discrimination in the setting of pretrial bail decisions. We first show that the selection challenge reduces to the challenge of measuring four moments: the mean latent qualification of white and Black individuals and the race-specific covariance between qualification and the algorithm’s treatment recommendation. We then show how these four moments can be estimated by extrapolating quasi-experimental variation across as-good-as-randomly assigned decision-makers. Estimates from New York City show that a sophisticated machine learning algorithm discriminates against Black defendants, even though defendant race and ethnicity are not included in the training data. The algorithm recommends releasing white defendants before trial at an 8 percentage point (11 percent) higher rate than Black defendants with identical potential for pretrial misconduct, with this unwarranted disparity explaining 77 percent of the observed racial disparity in algorithmic recommendations. We find a similar level of algorithmic discrimination with regression-based recommendations, using a model inspired by a widely used pretrial risk assessment tool.

Scope of Project

Subject Terms: Algorithmic discrimination; pretrial detention; racial discrimination; quasi-experiment

JEL Classification:
      C26 Single Equation Models: Single Variables: Instrumental Variables (IV) Estimation
      J15 Economics of Minorities, Races, Indigenous Peoples, and Immigrants; Non-labor Discrimination
      K42 Illegal Behavior and the Enforcement of Law

Geographic Coverage: New York City

Time Period(s): 11/1/2008 – 11/1/2013

Related Publications

Download this project

Published Versions

V1 [2021-04-28]

Export Metadata

Dublin Core

DDI 2.5

Report a Problem

Found a serious problem with the data, such as disclosure risk or copyrighted content? Let us know.

This material is distributed exactly as it arrived from the data depositor. ICPSR has not checked or processed this material. Users should consult the investigator(s) if further information is desired.