Stuck in a Pattern

Early evidence on "predictive policing" and civil rights

August 2016

A report from Upturn

David Robinson & Logan Koepke

Executive Summary

The term “predictive policing” refers to computer systems that use data to forecast where crime will happen or who will be involved. Some tools produce maps of anticipated crime “hot spots,” while others score and flag people deemed most likely to be involved in crime or violence.

Though these systems are rolling out in police departments nationwide, our research found pervasive, fundamental gaps in what’s publicly known about them.

How these tools work and make predictions, how they define and measure their performance and how police departments actually use these systems day-to-day, are all unclear. Further, vendors routinely claim that the inner working of their technology is proprietary, keeping their methods a closely-held trade secret, even from the departments themselves. And early research findings suggest that these systems may not actually make people safer — and that they may lead to even more aggressive enforcement in communities that are already heavily policed.

Predictive policing systems typically rely, at a minimum, on historical data held by the police — records of crimes reported by the community, and of those identified by police on patrol, for example. Some systems seek to enhance their predictions by considering other factors, like the weather or a location’s proximity to liquor stores. However, criminologists have long emphasized that crime reports, and other statistics gathered by the police, are not an accurate record of all the crime that occurs in a community; instead, they are partly a record of law enforcement’s responses to what happens in a community. This means that predictive systems that rely on historical crime data risk fueling a cycle of distorted enforcement.

Predictions that come from computers may be trusted too much by police, the courts, and the public. People who lack technical expertise have a natural and well-documented tendency to overestimate the accuracy, objectivity, and reliability of information that comes from a computer, including from a predictive policing system. As one RAND study aptly put it, “[p]redictive policing has been so hyped that the reality cannot live up to the hyperbole. There is an underlying, erroneous assumption that advanced mathematical and computational power is both necessary and sufficient to reduce crime [but in fact] the predictions are only as good as the data used to make them.”1

The fact that we even call these systems “predictive” is itself a telling sign of excessive confidence in the systems. The systems really make general forecasts, not specific predictions. A more responsible term — and one more accurately evocative of the uncertainty inherent in these systems, would be “forecasting.”

The systems we found also appear not to track details about enforcement practices or community needs, which means that departments are missing potentially powerful opportunities to assess their performance more holistically and to avoid problems within their ranks.

In an overwhelming majority of cases, departments operate predictive systems with no apparent governing policies, and open public discussion about the adoption of these systems seems to be the exception to the rule. Though federal and state grant money has helped fuel the adoption of these systems, that money comes with few real strings in terms of transparency, accountability, and meaningfully involving the public.

In our survey of the nation’s 50 largest police forces, we found that at least 20 of them have used a predictive policing system, with at least an additional 11 actively exploring options to do so. Yet some sources indicate that 150 or more departments may be moving toward these systems with pilots, tests, or new deployments.

Our study finds a number of key risks in predictive policing, and a trend of rapid, poorly informed adoption in which those risks are often not considered. We believe that conscientious application of data has the potential to improve police practices in the future. But we found little evidence that today’s systems live up to their claims, and significant reason to fear that they may reinforce disproportionate and discriminatory policing practices.

  1. See Walter L. Perry, Brian McInnis, Carter C. Price, Susan C. Smith, John S. Hollywood, Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations, RAND Corporation, 2013, available at https://www.rand.org/content/dam/rand/pubs/research_reports/RR200/RR233/RAND_RR233.pdf.

Table of Contents

About This Report

Over the last several months, in partnership with a broad national coalition of major civil rights, civil liberties, and technology policy organizations, Upturn has been researching the emerging technologies that vendors are selling, and police have begun adopting, under the moniker of “predictive policing.” Our initial questions were simple: As best we can tell right now, what are these technologies, how do they work, and what might they mean for civil rights?

To answer those questions, we:

We found that key stakeholders often struggle to make well-informed choices about these new technologies. Not only advocates, journalists, and community members, but also crime analysts, police executives, city councils, and other stakeholders are hard pressed to critically evaluate the claims made by technology vendors. A shortage of technical expertise, rapid change in how the technologies work, and the veil of secrecy that often surrounds these tools all make this harder.

Language compounds the challenge: Though some technologies are widely agreed to fall into the category of “predictive policing,” there’s no consensus on that term’s boundaries. Given that many of these technologies build on longstanding police methods, it can be hard to draw a bright line between what is old and what is genuinely new in this space. Nonetheless, as this report explains, we believe that there is indeed something genuinely new happening here, and that it needs attention.

We prepared this report to share what we’ve learned so far, in the hope that it will empower and motivate key stakeholders to ask hard questions and get clear answers. It’s designed as a factual and analytical resource for everyone involved in the debate, and as background and context for the statement of civil right concerns released today by The Leadership Conference on Civil and Human Rights and sixteen other groups. We assess the landscape as we see it today and point to several core challenges, but do not offer ultimate conclusions about what any one community or stakeholder should decide or how they should act. We do think everyone involved needs a clear, factual understanding as to what is and isn’t happening today in predictive policing.

This report is a first step.

Upturn is a team of technologists and policy analysts based in Washington DC. We work with civil rights organizations and other social sector groups, acting as translators, guides, and allies in the fast-changing world of digital technology.

This report is just the beginning of our work on data-driven decisionmaking in criminal justice. We are committed to rigorously understanding and exploring the risks, promise, and actual impact of new applications of data. We welcome corrections or additions to these initial findings — and appreciate feedback on any aspect of this report. You can reach our team at hello@teamupturn.org.

What We've Learned So Far

Here’s what you need to know first.

Predictive policing is a marketing term — popularized by vendors in the public safety industry — for computer systems that use data to automatically forecast where crime will happen or who will be involved.

Mapping recent crime “hotspots” and looking for trends in crime data have long been part of police work. But these new systems go beyond simply helping human crime analysts digest data. They aim to automatically predict future crime to inform police decisions. To do so, these systems make judgments about what the data means — flagging people most likely to be involved in violent crime in the future, for instance, or weighing hundreds or thousands of factors in ways that no human analyst can fully grasp. These extra functions are motivated by hopes for greater efficiency and safety, but those benefits remain largely unproven — and these tools create major new risks.

Most predictive policing systems make predictions about where and when future crime will happen. These are called “place-based” tools. There’s also an emerging category of “person-based” tools that focus on the who of future crimes: who may commit them, or who may be a victim.2

Place-based: Crime mapping with a new twist.

In place-based systems, predictions are usually presented on a map. Here’s an example from a system called PredPol:

The colored boxes show where the system assesses that future crimes are most likely to happen.

What’s driving the predictions behind the boxes? Most predictive policing systems rely on historical data held by the police department — typically, records of reported crimes and other crimes detected by the department.

Departments typically have data about 911 calls (“calls for service”), and about other crimes reported to the department. This includes both crimes reported by the community, and those identified by police on patrol. The information is usually stored in a police-operated database called a Records Management System.3

Predictive policing systems use this data, and sometimes other information, turning it into conjectures about what may happen next. Some programs and vendors also add variables like the weather, a location’s proximity to liquor stores, or even commercial data brokerage information in an attempt to enhance their models. In a typical system, the software automatically looks for historical patterns in the data, and uses those patterns to make its predictions — a process known as “machine learning.” It’s the same type of technology that Google uses to improve its language translation tools, or to automatically label uploaded photos (with generally strong, but sometimes inaccurate and offensive results).4

For example, PredPol, a startup predictive policing vendor, applies machine learning techniques to try to predict future crimes at locations other than where previous crimes took place. (This contrasts with traditional crime mapping systems that focus on mapping recent crimes, on the theory that future crime is more likely in the immediate vicinity of recent offenses.)

Person-based: Names on a list, or scores.

The other main type of predictive policing tool, rather than making predictions about where crime will happen, focuses on who will be involved — predicting the identities of people particularly likely to commit or to be victims of certain kinds of crimes.

The city of Chicago, for example, uses a “Strategic Subjects List” (or SSL, often referred to as a heat list) of people that a computer program has judged most likely to become involved in a shooting, either as a perpetrator or as a victim. The algorithm generating the heat list uses a social network analysis method, where each person’s risk “score” depends not only on his or her own past behavior, but also on the past offenses, apparent gang affiliations (as recorded by the police), and other criminal justice records of people the person has been “co-arrested” with.

Another such system, marketed under the name “Beware,” uses information collected by commercial data brokers to assign a “threat score” to each member of the community.5 Published sources do not make clear what these scores are intended to measure, much less whether they are accurate in doing so.6

Who sells predictive policing systems?

Table 1

A Survey of Vendors of Predictive Policing Systems

Download PDF
System What data does it use? Who made it? Output
PredPol Historical crime data Though PredPol is a private company, the two cofounders are both professors: one, an Assistant Professor Santa Clara University in Mathematics and Computer Science, the other a Professor of Anthropology at UCLA. 500’ x 500’ boxes that are possible criminal hotspots, specific to shifts and crime type. Can be delivered to officers via printout, smart phone, or tablet.
Risk Terrain Modeling Historical crime data; important geographic features Les Kennedy and Joel Caplan run RTM at Rutgers University. Tabular/map-based.
Strategic Subjects List (SSL) / aka Chicago’s Heat List Historical crime data; a commonly reported variable is rap sheets (arrest and conv; social connections/relationships; social media Miles Wernick is the Motorola professor and director of the Medical Imaging Research Center at Illinois Institute of Technology (IIT) in Chicago. He helped CPD develop SSL. Rank-order list of 400-1400 people judged most likely to be involved in a violent crime.
IBM SPPS Crime Prediction and Prevention Historical crime data; important geographic features; weather; social media No clear, public evidence of academic involvement. It appears to be a solely private venture. Multi-layer, color-coded hotspot maps.
Hitachi Visualization Predictive Crime Analytics Historical crime data; important geographic features; weather; social media; social connections/relationships No clear, public evidence of academic involvement. It appears to be a solely private venture. 200-meter squares that are assigned threat levels from 0-100%.
HunchLab Historical crime data; important geographic features; temporal data sets, like weather data, social events, or school schedules; U.S. Census data regarding vacancies HunchLab worked with Drs. Jerry Ratcliffe and Ralph Taylor at Temple University to model long-term crime trends based upon neighborhood demographic indicators. They also worked with Drs. Joel Caplan and Les Kennedy at Rutgers University, Azavea worked to automate their Risk Terrain Modeling approach to crime forecasting. Can be boxes/rectangles of 500’ x 500’ predicting likelihood of crimes in that area. List of crimes ranked by percentages, e.g. (robbery 60%, aggravated assault 28%, theft from vehicle 7%, motor vehicle theft 3%, homicide 1%, residential burglary 0%).
CommandCentral Predictive (Motorola) Historical crime data No clear, public evidence of academic involvement. It appears to be a solely private venture. Boxes on maps that can be as small as 500’ x 500’. Tier-ranked boxes (1, 2, 3) to help officers understand where to spend their time.
BAIR Analytics ATACRAIDS (LexisNexis) Historical crime data No clear, public evidence of academic involvement. It appears to be a solely private venture. Map-based predictive hotspots.
Esri GIS Crime Analysis Toolbox Historical crime data The GIS Toolbox uses Near Repeat Calculator — a tool developed by researchers at Temple University. Map-based predictive hotspots sorted by priority ranking.
Intrado’s Beware Historical crime data; social media; commercially available data from data brokers No clear, public evidence of academic involvement. It appears to be a solely private venture. A numerical threat score and a color coded threat-level (green, yellow, red) to any person, area or address that a police department searches.

How We Searched For Information

We reviewed vendor promotional materials, research literature, videos/webinars, relevant scholarly literature, trade press directed at police and the vendor community, available public contracts and scope of work agreements between vendors and cities, or vendors and police departments, local and national news reports, and relevant documents that have become public via FOIA requests.

Police statistics reflect enforcement, not just crime.

Recorded crime rates are an incomplete, distorted reflection of police performance — but predictive policing systems use these numbers as their yardstick of success.

Reductions in the observed crime rate — or increases in arrests, citations, or stops — do not necessarily correspond to real increases in community safety or improvements in police performance. Criminologists have long emphasized that crime reports, and other statistics gathered by the police, are not an accurate record of the crime that happens in a community. As one expert wrote:

It has been known for more than 30 years that, in general, police statistics are poor measures of true levels of crime. This is in part because citizens exercise an extraordinary degree of discretion in deciding what crimes to report to police, and police exercise an extraordinary degree of discretion in deciding what to report as crimes. Moreover, some unknown proportion of perpetrators are actively engaged in committing crimes in ways that make it unlikely that their crimes will ever be discovered. In addition, both crime and crime clearance rates can be manipulated dramatically by any police agency with a will to do so. It is also absolutely axiomatic that for certain types of crime (drug offenses, prostitution, corruption, illegal gambling, receiving stolen property, driving under the influence, etc.), police statistics are in no way reflective of the level of that type of crime or of the rise and fall of it, but they are reflective of the level of police agency resources dedicated to its detection.10

In short, the numbers are greatly influenced by what crimes citizens choose to report, the places police are sent on patrol, and how police decide to respond to the situations they encounter.

The National Crime Victimization Survey (conducted by the Department of Justice) found that from 2006-2010, 52 percent of violent crime victimizations11 went unreported to police and 60 percent of household property crime victimizations went unreported.12 Historically, the National Crime Victimization Survey “has shown that police are not notified of about half of all rapes, robberies and aggravated assaults.”13

Enforcement practices — which can vary widely from one neighborhood to another — have a much larger impact on statistics for some crimes than for others. For example, it’s likely that all (or very nearly all) bank robberies are reported to police.14 On the other hand, marijuana possession arrests are notoriously biased, with black Americans much more likely to be arrested than whites who use the drug at similar rates.15 Predictive systems that incorporate these sorts of statistics may not account for the inaccuracies reflected in historical data, leading to a cycle of self-fulfilling prophecies.

Predictive policing systems don’t measure other community needs.

Predictive policing systems could incorporate other measures of community need and police performance–beyond the crime rate–but currently do not do so. In most of the nation, police currently measure outcomes and assess performance based on only some of the activities, costs, and benefits that matter in policing.

Serious violent crimes will always be important. But violent crime doesn’t reflect the full scope of community concerns: “Many crimes are not reported, and therefore police would need to use a broader range of data sources — including public health information and victimization surveys — even to be able to see the full range of problems that matter.”16 We found no indication that predictive policing systems today incorporate this more holistic approach to community safety.

Moreover, experts on police performance measurement have long argued that police should track all uses of coercive authority so they can better promote public safety with minimum coercion.17 Likewise, the President’s Task Force on 21st Century policing found that “[l]aw enforcement agencies should track the level of trust in police by their communities just as they measure changes in crime.”18 And research on police performance measurement consistently calls for surveying victims to gather their feedback on the police officers with whom they interact.19

Beyond the basic goal of constitutional, lawful policing, measuring factors like these could allow the police to track and reward strategies that do a better job of balancing a community’s needs and interests. As a White House report recently found, “if feedback loops are not thoughtfully constructed, a predictive algorithmic system … could perpetuate policing practices that are not sufficiently attuned to community needs and potentially impede efforts to improve community trust and safety.”20 In other words, police and the systems they use might gather data, make predictions, and base decisions not only where future crimes may be found, but also on the need to economize their use of authority, maintain community trust, and provide positive experiences for crime victims.

Yet, so far none of the predictive policing systems identified in our research track and analyze all police uses of coercive authority. None, as far as we could tell, regularly incorporate feedback data from crime victims or community members. And none of these tools appear to be conducting racial impact assessment to correct for inherent or past bias or better serve community needs — even though some departments that use predictive policing tools already do track race of those stopped and arrested as part of their records management system.

Measuring police performance is today, and has always been, a hard problem. These measurement challenges are common to all policing, not specific to new predictive tools.21 But new technology, including predictive analytics, could be designed and used in ways that address these longstanding challenges. Right now, that opportunity is being missed.

A “ratchet effect” can reinforce discrimination.

Statistics experts warn that sampling bias in police data may lead to a “ratchet” of increasingly distorted enforcement. In the context of predictive policing, statistics generated by the policing process are often treated as though they are records of underlying criminal behavior. As discussed above, however, these numbers often reflect policing tactics as much as, or more than, they reflect actual offender behavior. Criminologists argue that “[a]rrest, conviction, and incarceration data are most appropriately viewed as measures of official response to criminal behavior.”22

The scientific term for this problem is “sampling bias.” When it comes to data, the word “bias” has a special meaning, different from its ordinary sense. “Biased data” does not mean data that was gathered with bad intentions, or data that will be unfair if used in a certain way. It simply means that the data does not perfectly reflect reality — that certain things are overrepresented or underrepresented in the sample relative to the actual population.

Of course, it makes sense for police to be responsive to community needs. Different communities served by the same police department often do have different needs, and different levels of need. But it is dangerous to treat the results of that process as though they were a neutral reflection of the world.

As data scientist Cathy O’Neil has explained, “people have too much trust in numbers to be intrinsically objective.”23

In his book Against Prediction, Bernard Harcourt points to the risk of a “ratchet effect.”24 If policing numbers are treated as though they do neutrally reflect the rate of criminal offending, he warns, the resulting distortion will “get incrementally worse each year if law enforcement departments rely on the evidence of last year’s correctional traces—arrest or conviction rates—in order to set next year’s [enforcement] targets.”25

Whenever departments focus their attention on a particular place or group of people, police will detect and respond to more of the of the crime that happens in those places, or among those people, than they will detect or respond to other crime. Even when police have a good reason to focus their attention, such as a particular neighborhood struggling with a violence problem, that focus will nonetheless distort the relationship between police statistics and true levels of crime. Systems that rely on these distorted statistics will similarly suffer from the blind spots they produce.26


Predictive policing may not improve community safety.

Predictive policing tools may incrementally improve on earlier methods of predicting crime reports, but independent research has yet to find any benefit for community safety.

Though system vendors often cite internally performed validation studies to demonstrate the value of their solutions, our research surfaced few rigorous analyses of predictive policing systems’ claims of efficacy, accuracy, or crime reduction.33

We are currently aware of two rigorous, scholarly studies of predictive policing in the United States whose authors have no interest in the success of the method being evaluated. Both of these were conducted by the RAND Corporation.34 Neither analysis found any safety benefit in the predictive policing tools studied:

Two other academic studies, whose authors did have an interest in the success of the tool or method being studied, found evidence that their tools outperform earlier methods of predicting the locations of future crime reports.

Professors George Mohler and Jeff Brantigham, two of the cofounders of PredPol, coauthored a study that analyzed PredPol’s technique through a randomized controlled trial in Los Angeles, and found that whereas the geographic boxes drawn by human analysts contained 1.6 and 3.2 percent of crimes observed during the study, boxes drawn with PredPol’s method contained between 3.4 and 6.4 percent of observed crimes.37 They argue that these numbers show that PredPol’s methods predicted reported crime more effectively than the crime analysts in Los Angeles to whom the methods were compared, with an “average 7.4% reduction in crime volume as a function of patrol time.”38

At the same time, the RAND researchers assessing predictive policing systems have argued that “small improvements can be made to appear as large percentage improvements, when they are rather insufficient to make a difference in the real world.”39 An independent study by a researcher based in Switzerland, reviewing the Mohler study, found that similar results could have been achieved by simply sending officers to existing crime hotspots, without using any predictive tools.40

The second such finding involved a study of Risk Terrain Modelling — a place-based approach that uses geographic features such as store locations to estimate crime risks — and concluded that it was better at predicting reported crime than a traditional “hotspot” analysis based purely on prior crime reports.41 The authors of this study, faculty at Rutgers University, invented the Risk Terrain Modelling approach.

These tools drive greater enforcement.

Rather than changing their tactics, police who use predictive tools have tended to focus on generating more citations and arrests. There is, generally, no standard for how police should use the predictions they are given — in a typical scenario, as the research cited below makes clear, boxes on a map, a list of names, or other predictions are given to command staff and filtered down to rank and file officers, with little if any guidance about what to do about the system’s predictions. By the same token, predictive policing systems typically do not measure what police do with the information, beyond the basic level of recording what geographic areas officers are told to patrol.

The evidence available today consistently indicates that officers are using predictive policing not to embrace new or different tactics like community engagement or other types of positive interventions, but instead to decide where and with whom to employ traditional, enforcement-oriented strategies of stops, citations and arrests.

For example, the RAND evaluation in Shreveport found that “preventive measures within predicted hot spots were never fully specified, much less standardized.”42 Without clearly-defined and standardized community policing tactics to complement insights provided by new, predictive technology, officers may arrive at an area of projected criminal activity and simply begin citing or arresting people. Indeed, in Shreveport, officers did just that: “Officers stopped individuals who were committing ordinance violations [e.g. walking in the middle of the street in a place where there is a sidewalk] or otherwise acting suspiciously.”43

A similar thing happened at the Greensboro Police Department in North Carolina. There, during the police department’s pilot study and evaluation of HunchLab, officers were not “given specific enough direction or training on how to respond … currently it’s go there [to the predicted areas of criminal activity] and be visible … do some stuff.”44

In the Mohler paper about PredPol’s approach (which involves the place-based method of predicting areas on maps), officers in Los Angeles were simply encouraged “to use available time to ‘get in the box’ and use their discretion to select appropriate field tactics.”45

Likewise, in Chicago, with its person-based Strategic Subjects List, an initial wave of PR for the project suggested that the approach would involve social service interventions. According to one newspaper report, this was meant to be a carrot-and-stick approach, where individuals on the list would be warned that “further criminal activity, even for the most petty offenses, will result in the full force of the law being brought down on them … At the same time, police extend them an olive branch of sorts, an offer of help obtaining a job or of social services.”46 Even Chicago Mayor Rahm Emmanuel said the program was a blend of social services with other tactics.47

But the recent RAND study of the program found the carrot got lost, and only the stick remained: “Interviews indicated that directing officers to increase contact with SSLs was likely the extent of the preventive intervention strategy for the majority of districts.”48 The RAND researchers found in Chicago: “It is not at all evident that contacting people at greater risk of being involved in violence — especially without further guidance on what to say to them or otherwise how to follow up — is the relevant strategy to reduce violence[.]” As they wrote, “the prevention strategy associated with the predictive strategy was not well developed and only led to increased contact with a group of people already in relatively frequent contact with police.”49 This is the same study that found the initial pilot “does not appear to have been successful in reducing gun violence.” At the same time, the number of people on the Strategic Subject List has more than tripled in the past three years.50

The Chicago Police Department, responding to the RAND study, emphasized that they now use a different algorithm to put people on their list than the one RAND evaluated, calling RAND’s findings “outdated.”51 But the department did not substantially dispute RAND’s observations of how police were using the list.52

Andrew Papachristos — a Yale sociologist and public health scholar whose research inspired Chicago’s efforts — recently published an op-ed in the Chicago Tribune expressing his consternation at this punitive approach. Many of the people involved in gang violence are at high risk both to shoot and to be shot, Papachristos noted, and the Chicago Police Department faces a crucial choice: will it treat “its list as offenders or as potential victims?” “[A] public health approach fundamentally means treating victims as victims,” wrote Papachristos:

[O]ne of the inherent dangers of police-led initiatives is that, at some level, any such efforts will become offender-focused … We’re now developing the data analytics to help us better understand who within our communities are most at risk for violence involvement. The question now is: How will we intervene? Will we repeat the mistakes of the past and attempt to arrest our way out of our problems? Or will we use the newly uncovered data to provide better solutions?53

We found just one predictive policing tool that attempts to incorporate an awareness of different tactics into its system. HunchLab, a predictive policing system made by azavea, has a separate product, Tactics, which allows analysts to “design evidence-based tactics to use in response to particular crime types. Tactics can also be limited to a particular geographic area (such as a division or district) or based upon proximity to a geographic feature such as a school or bar. Appropriate tactics are automatically displayed to officers when entering mission areas.”54 An approach like this opens the door so that, over time, analysts could potentially measure which tactics have the best impact on the community.

Departments are not saying how they use these tools.

Departments that adopt these tools aren’t providing clear, public policies on how the tools are used. In our canvas of the 50 largest U.S. police departments, as well as smaller police departments that received funding for predictive policing systems through DOJ grants, Chicago is the only department to publish a policy that describes how the department will use its predictive policing system. Chicago’s policy describes its Custom Notifications strategy — where police officers, social workers, and community leaders “deliver a joint message … informing [people on the list] of their risk for prosecution based on criminal history, and explaining their opportunities for community help and support”55 — though the policy leaves basic questions about the list and the department’s practices unanswered.56

Los Angeles is one example of a place where the absence of clear rules is particularly striking. The LAPD has a record of racial profiling, disproportionate policing of minority communities, and was under a federal consent decree for more than a decade.57 Though it is currently contracted with PredPol,58 the department has no public policy governing its use of PredPol. The same is true of the Philadelphia PD, where the police department uses HunchLab, as well as many other smaller departments throughout California that have adopted PredPol.

Without transparency, there’s little debate.

So far in our research, an open public debate regarding a police department’s potential adoption of a predictive policing system seems to be the exception to the rule. For example, although the Miami-Dade Police Department has received over $600,000 in BJA grant money to purchase predictive policing software, it appears that there was no meaningful public debate regarding whether or not the software should be purchased. The same goes for the NYPD’s pilot of HunchLab — the department will be testing the system for two years across three precincts, but there seems to have been little, if any, meaningful public discussion about the pilot. Police departments in Seattle, Atlanta, and Philadelphia likewise appear to have adopted systems with little scrutiny or public input.

The handful of exceptional cases in which these tools have occasioned public debate point strongly to the need for more transparency and a clearer view of how these tools work.

The case of Bellingham, Washington presents one illustration of how a lack of transparency can stifle a public debate. For two years, the police department considered purchasing a predictive policing system. In 2014, the police department attempted to use a Bureau of Justice Assistance (BJA) grant to acquire Intrado’s Beware software. After serious public pushback, the city council “took a vote asking the department to use the grant money to buy something else,”59 though the council did not have the authority to block the BJA grant or direct its funds toward a different use.60

Because of the community pushback, the department dropped their purchase of Beware and decided to look for a new system. In 2015, the police department used a BJA grant to acquire a different predictive policing system, this time from BAIR Analytics.61 Because the police department bought predictive software made by BAIR with a DOJ grant,62 the department didn’t need the city council’s approval — just as with the police department’s first attempt to purchase Beware. Nor did the police department hold community meetings to discuss, in advance, how the software worked or how the department would be using it.63 Instead, the community was given only seven business days’ notice of a city council meeting in which the department would solicit public comment — and, by then, the decision to purchase the equipment was already final.

As one Bellingham resident, Edward Alexander, put it: “This is a hearing for the people to give input, but we haven’t been given the time to investigate this … six or seven days, that’s not enough for a conversation.”64 Kim Harris, a spokeswoman for Bellingham’s Racial Justice Coalition, said that though “[w]e wanted a greater explanation for how this all worked … we were told it was all proprietary.”65

The experience of Fresno, California is also instructive. Like Bellingham, Fresno’s Police Department was interested in Intrado Beware. The Fresno Police Department, however, actually piloted Beware and was “one of the first departments in the nation to test the program.”66 But there was significant concern over the program’s color-coded threat level.

When the Fresno police briefed the city council about the Beware system, the council president asked a simple question about those color-coded threat levels: “How does a person get to red?” The officer didn’t know, because the vendor wasn’t saying. As the officer delivering the briefing explained: “We don’t know what their algorithm is exactly… We don’t have any kind of a list that will tell us, this is their criteria, this is what would make a person red, this is what would make a person yellow. It’s their own system.”67 Later in that meeting, one of the council members asked the officers to run threat assessment on the council member’s own home. It came back yellow, apparently indicating a medium threat in that home (though the council member himself came back green).

The Beware debate in Fresno garnered national press coverage, partly because it was so unusual to see these systems publicly debated. As Conor Friedersdorf wrote in The Atlantic:

“More and more secret code is being incorporated into the criminal-justice system, making it more opaque and vulnerable to mistakes. . . . so long as residents aren’t allowed to know what causes their local police force to stigmatize a person or an address as ‘dangerous,’ they’re effectively denied the ability to decide whether the software tool is just or prudent. My threat assessment: Beware of this product and proceed only with great caution.”68

In a subsequent city council meeting on March 31, 2016, Chief Dyer again requested permission to use grant funding for a scaled down deployment of Beware — one without the color-coded threat scores or social media monitoring.69

Community members who attended the meeting were not sanguine about the system, and urged the adoption of an ordinance that would govern how the software would and wouldn’t be used by the Fresno PD. Council members ultimately agreed that they would “host community meetings in their districts over the next 60 days to solicit public input about the use of the software and safeguards desired to provide oversight.”70

At the same meeting, hours after the discussion of Beware, the city council voted to approve purchase of a second system: PredPol.71 The department has been asked to report to the city council on its experience after using the system for six months.

Federal grants don’t drive transparency.

Federal and state grant monies used to purchase predictive policing systems do not currently encourage meaningful transparency or community input. A common thread through many departments adopting predictive policing systems is grant funding, coming into a local community from outside its borders. Federal and state grant money, whether it subsidizes or completely pays for a system, has helped a number of police departments acquire these tools. But, that grant money often comes with few requirements or conditions that would further meaningful public discussion.

For example, the Edward Byrne Memorial Justice Assistance Grant (JAG) — a common federal grant through which local police departments acquire predictive policing systems, nominally requires a 30 day period for the local governing body to review new grants, and also public comment. However, the grant program does not require actual approval of the governing body, “nor is any type of public hearing [mandated] unless state/local law requires one.”72 Similarly, though the JAG grant requires departments to provide an opportunity for the public to comment, it does not detail what this process must look like. It only suggests ways in which a grantee can provide notice to a community.73 Further, because the governing body review requirement does not actually require actual approval by the governing body of the grant’s use of funds, the policy impact of public comment is minimized.74

Within their own ranks, officers resist even low-risk predictions.

Police hesitate to use predictive technology to analyze their own performance, even though that use can be limited to low-risk, potentially high-benefit steps such as better counselling and training.

While applications of predictive analytics in police enforcement can be highly problematic for the reasons detailed above, turning these approaches inward can be a lower stakes way to apply data driven insights in the policing context.

Whereas the output of a predictive policing system (accurate or otherwise) may result in someone’s stop, search, or arrest, signals produced by predictive early intervention tools would more likely lead to increased counselling, training, and mentorship for officers. False positives flagged by an internally facing system are less likely to trigger punitive responses than flagging a member of the community as a likely criminal.

Evidence has shown that Early Intervention Systems — internal tools “designed to identify officers whose performance exhibits problems, and then to provide interventions, usually counseling or training, to correct those performance problems”75 — informed by data can better identify officers who would benefit from additional training, counseling, or other misconduct-prevention interventions, tangibly improving community safety without increasing enforcement activity. And departments have shown interest in these types of programs. For example, in the mid-to-late 1990s, the Chicago Police Department used BrainMaker, “an early warning system to identify officers likely to go bad.”76 BrainMaker was a commercially available program based on “neural network” technology that used “complaints of excessive force, work history and other factors … to allow the department to steer officers toward counseling before they committed a crime.”77 Police union pressure, among other factors, ultimately led to the project’s demise, even though it was described as an effective tool.78

More recent experiences at the Charlotte-Mecklenburg Police Department (CMPD) also suggest that Early Intervention Systems supported by data-driven analysis can outperform existing approaches. The CMPD and researchers at the University of Chicago’s Center for Data Science and Public Policy worked together to answer two prediction questions for the department:

1) Which officers are likely to have an adverse interaction in the next two years? 2) Which dispatches are likely to end up having an adverse interaction?79

The final model that was developed outperformed the department’s existing Early Intervention System.80 “Equally important,” the joint team notes, “it reduces the number of officers who were flagged but did not go on to have an adverse incident in the next year” (false positives).81 Encouragingly, the department and researchers plan to make “its code available for departments to use and invites other departments to join the project” — the Los Angeles, California Sheriff’s Department, the Knoxville, Tennessee Police Department, and the Metropolitan Nashville, Tennessee Department have already joined the project.82

Multiple studies by the Department of Justice have highlighted the value of Early Intervention Systems in advancing community-oriented policing goals and in helping reduce citizen complaints against officers.83 Partnerships, like the one between the Charlotte-Mecklenburg PD and the University of Chicago, and conscientious use of data can help law enforcement build better Early Intervention Systems. The more accurate a system’s predictions, the better targeted interventions for at-risk officers become, and the more likely that adverse interactions can be prevented — a boon for departments and their communities.

  1. See Walter L. Perry, Brian McInnis, Carter C. Price, Susan C. Smith, John S. Hollywood, Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations, RAND Corporation, 2013, available at https://www.rand.org/content/dam/rand/pubs/research_reports/RR200/RR233/RAND_RR233.pdf.

  2. RMS Technical Requirements for Crime Analysis - Standards, Methods, & Technology (SMT) Committee White Paper, International Association of Crime Analysts (IACA) (2013), http://www.iaca.net/publications/whitepapers/iacawp_2013_01_rms_requirements_for_crime_analysis.pdf.

  3. Alistair Barr, Google Mistakenly Tags Black People as “Gorillas,” Showing Limits of Algorithms, The Wall Street Journal, July 1, 2015, http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tags-black-people-as-gorillas-showing-limits-of-algorithms.

  4. Justin Jouvenal, The new way police are surveilling you: Calculating your threat “score,” The Washington Post, January 10, 2016, available at https://www.washingtonpost.com/local/public-safety/the-new-way-police-are-surveilling-you-calculating-your-threat-score/2016/01/10/e42bccac-8e15-11e5-baf4-bdf37355da0c_story.html?tid=pm_pop_b.

  5. David Robinson, Buyer Beware: A hard look at police “threat scores.” Equal Future (2016), available at https://medium.com/equal-future/buyer-beware-a-hard-look-at-police-threat-scores-961f73b88b10#.6ym7bx9x7.

  6. Carl B. Klockars, Some Really Cheap Ways of Measuring What Really Matters, in Measuring What Matters: Proc. from the Police Res. Inst. Meetings 201 (Nat’l Inst. of Justice (NIJ) 1999), available at https://www.ncjrs.gov/pdffiles1/nij/170610.pdf.

  7. “Crime victimization,” according to the Department of Justice, includes violent victimization (rape, sexual assault, robbery, aggravated assault, and simple assault) and property victimization (burglary, motor vehicle theft, and property theft) as well as domestic violence and intimate partner violence. U.S. Department of Justice, Office of Justice Programs, Bureau of Justice Statistics, Special Report, Criminal Victimization, 2014, August 2015, available at http://www.bjs.gov/index.cfm?ty=pbdetail&iid=5366.

  8. U.S. Department of Justice, Office of Justice Programs, Bureau of Justice Statistics, Special Report, Victimizations Not Reported to the Police, 2006-2010, August 2012, available at http://www.bjs.gov/content/pub/pdf/vnrp0610.pdf.

  9. Patrick Langan, Matthew Durose, Bureau of Justice Statistics, The Remarkable Drop in Crime in New York City, October 21, 2004, available at http://www3.istat.it/istat/eventi/2003/perunasocieta/relazioni/Langan_rel.pdf, 9.

  10. Carl B. Klockars, supra note 10, at 201 (“If I had to select a single type of crime for which its true level—the level at which it is reported—and the police statistics that record it were virtually identical, it would be bank robbery. Those figures are likely to be identical because banks are geared in all sorts of ways . . . to aid in the reporting and recording of robberies and the identification of robbers. And, because most everyone takes bank robbery seriously, both Federal and local police are highly motivated to record such events.”).

  11. ACLU, The War on Marijuana in Black and White: Billions of Dollars Wasted on Racially Biased Arrests, (2013), available at https://www.aclu.org/files/assets/aclu-thewaronmarijuana-rel2.pdf.

  12. Malcolm K. Sparrow, Measuring Performance in a Modern Police Organization, NEW PERSP. IN POLICING 21 (2009), available at https://www.ncjrs.gov/pdffiles1/nij/248476.pdf (“Particularly useful would be public health data from hospital emergency rooms that might reveal “the physical attacks that happen behind closed doors, or which are otherwise not reported to the police . . . [one review found] that, nationwide, 31 percent of victimizations from 2006 to 2010 involving a weapon and injury to the victim went unreported to police”) (internal citations omitted).

  13. See, e.g., Mark H. Moore & Anthony Braga, The “Bottom Line” of Policing: What Citizens Should Value (and Measure!) In Police Performance (Police Executive Research Forum (PERF) 2003) at 57 (“Measuring Fairness and Economy in the Use of Force and Authority”).

  14. President’s Task Force on 21st Century Policing. 2015. Final Report of the President’s Task Force on 21st Century Policing, Washington, DC: Office of Community Oriented Policing Services, available at http://www.cops.usdoj.gov/pdf/taskforce/taskforce_finalreport.pdf, 16.

  15. See, e.g., Mark H. Moore & Margaret Poethig, The Police as an Agency of Municipal Government: Implications for Measuring Police Effectiveness, in Measuring What Matters: Proc. from the Police Res. Inst. Meetings 170 (Nat’l Inst. of Justice (NIJ) 1999), available at https://www.ncjrs.gov/pdffiles1/nij/170610.pdf.

  16. Executive Office of the President, Big Data: A Report on Algorithmic Systems, Opportunity, and Civil Rights (May 2016), available at https://whitehouse.gov/sites/default/files/microsites/ostp/2016_0504_data_discrimination.pdf.

  17. See, e.g., Mark H. Moore & Anthony Braga, The “Bottom Line” of Policing: What Citizens Should Value (and Measure!) in Police Performance (Police Exec. Research Forum (PERF) 2003) 64, available at http://www.policeforum.org/assets/docs/Free_Online_Documents/Police_Evaluation/the%20bottom%20line%20of%20policing%202003.pdf.

  18. Delbert S. Elliott, Lies, Damn Lies, and Arrest Statistics (1995), available at http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.182.9427&rep=rep1&type=pdf, at 11.

  19. Cathy O’Neil, The Ethical Data Scientist, Slate, February 4, 2016, available at http://www.slate.com/articles/technology/future_tense/2016/02/how_to_bring_better_ethics_to_data_science.html.

  20. See Bernard E. Harcourt, Against Prediction: Profiling, Policing, and Punishing in an Actuarial Age (Univ. of Chi. Press 2008), at 272.

  21. Id.

  22. Logan Koepke, “The ‘streetlight effect’ of big data studies,” June 16, 2016, available at https://medium.com/equal-future/the-streetlight-effect-of-big-data-studies-e3b289a834a1.

  23. See, e.g. “Predictive Policing Research,” National Institute of Justice, available at http://www.nij.gov/topics/law-enforcement/strategies/predictive-policing/Pages/research.aspx.

  24. RAND was funded by the National Institute of Justice to serve as an independent evaluator of “the efficacy of the concept of predictive policing as seen in Chicago and Shreveport’s predictive policing models.” Id.

  25. Jessica Saunders, Priscilla Hunt & John S. Hollywood, Predictions put into practice: a quasi-experimental evaluation of Chicago’s predictive policing pilot, J Exp Criminol (2016), available at https://link.springer.com/article/10.1007/s11292-016-9272-0.

  26. Priscilla Hunt, Jessica M. Saunders & John S. Hollywood, Evaluation of the Shreveport predictive policing experiment (2014), available at http://www.rand.org/pubs/research_reports/RR531.html.

  27. George O. Mohler et al., Randomized controlled field trials of predictive policing (Preprint Version), 110 Journal of the American Statistical Association 1399–1411 (2015).

  28. Id. at 3.

  29. “[S]ince the baseline accuracy of predictions is still relatively low, small improvements can be made to appear as large percentage improvements, when they are rather insufficient to make a difference in the real world. For instance, a method or model may improve the prediction of homicide perpetrators in a city in a year from 1 out of 100,000 people to 6 out of 100,000—a 500 % improvement—but using the average homicide levels in cities, the new approach will still fail to identify nearly 99.5 % of homicide perpetrators.” Jessica Saunders, Priscilla Hunt & John S. Hollywood, Predictions put into practice: a quasi-experimental evaluation of Chicago’s predictive policing pilot, J Exp Criminol (2016), available at https://link.springer.com/article/10.1007/s11292-016-9272-0.

  30. See Matt Stroud, “Official Police Business: Does predictive policing actually work?,” The Verge, May 4, 2016, available at http://www.theverge.com/2016/5/4/11583204/official-police-business-predictive-policing-paper; Ismael Benslimane, Étude critique d’un système d’analyse prédictive appliqué à la criminalité : Predpol®, June 18, 2015, available at http://cortecs.org/wp-content/uploads/2014/10/rapport_stage_Ismael_Benslimane.pdf.

  31. Leslie W. Kennedy et al., Risk Clusters, Hotspots, and Spatial Intelligence: Risk Terrain Modeling as an Algorithm for Police Resource Allocation Strategies, 27 J. of Quantitative Criminology 339, 347 (2011), available at http://www.rutgerscps.org/uploads/2/7/3/7/27370595/newarkrtm_casestudy_brief.pdf.

  32. Priscilla Hunt, Jessica M. Saunders & John S. Hollywood, Evaluation of the Shreveport predictive policing experiment (2014), available at http://www.rand.org/pubs/research_reports/RR531.html, xv.

  33. Id. at 12.

  34. azavea, HunchLab Predictive Missions at Greensboro PD: “Tell me what I don’t know!” (2015), available at https://www.youtube.com/watch?v=E-QdYqZzQhY, (time mark 37:00 - 54:33).

  35. George O. Mohler et al., Randomized controlled field trials of predictive policing (Preprint Version), 110 Journal of the American Statistical Association 1399–1411 (2015), 6.

  36. Jeremy Gormer, “Chicago police use heat list as strategy to prevent violence,” Chicago Tribune, August 21, 2013, available at http://articles.chicagotribune.com/2013-08-21/news/ct-met-heat-list-20130821_1_chicago-police-commander-andrew-papachristos-heat-list.

  37. “Next, the District Commander and an influential community member visit the individual’s house and explain that the individual is subject to enhanced penalties based on his or her criminal history. Additionally, they explain that there are opportunities to connect with social services and job placement if he or she wants to turn away from a life of crime. “ Congresswoman Robin L. Kelley, 2014 Kelley Report: Gun Violence in America, available at http://robinkelly.house.gov/sites/robinkelly.house.gov/files/wysiwyg_uploaded/KellyReport_1.pdf.

  38. (Jessica Saunders et al :10)

  39. (Jessica Saunders et al :17) (emphasis added).

  40. David Robinson, In 3 years, Chicago police have tripled their use of a secret, computerized “heat list,” Equal Future, May 26, 2016, available at https://medium.com/equal-future/in-3-years-chicago-police-have-tripled-their-use-of-a-secret-computerized-heat-list-da7a0594ee78#.ngyd9scfq.

  41. Chicago Police Department, CPD Welcomes the Opportunity to Comment on Recently Published RAND Review, (2016), available at http://4abpn833c0nr1zvwp7447f2b.wpengine.netdna-cdn.com/wp-content/uploads/2016/08/RAND_Response-1.pdf.

  42. Id.

  43. Andrew V. Papachristos, “CPD’s crucial choice: Treat its list as offenders or as potential victims?,” Chicago Tribune, July 29, 2016, available at http://www.chicagotribune.com/news/opinion/commentary/ct-gun-violence-list-chicago-police-murder-perspec-0801-jm-20160729-story.html.

  44. HunchLab, “Features,” available at https://www.hunchlab.com/features.

  45. David M. Kennedy and Michael A. Friedrich, “Custom Notifications Individualized Communication in the Group Violence Intervention,” U.S. Department of Justice Community Oriented Policing Services, 15, available at http://ric-zai-inc.com/Publications/cops-p304-pub.pdf.

  46. While the Chicago Police Department does have a policy regarding its Custom Notifications tactic (Special Order S10-05), it does not have a specific policy regarding the Strategic Subjects List. For example, though one can receive a Custom Notification because of inclusion on the SSL, being on the SSL is not a necessary condition to receive a Custom Notification. See: http://directives.chicagopolice.org/directives/data/a7a57bf0-1456faf9-bfa14-570a-a2deebf33c56ae59.html.

  47. ACLU of Southern California, Racial Profiling & the LAPD, available at https://www.aclusocal.org/issues/police-practices/racial-profiling-the-lapd.

  48. “Proven Results of our Predictive Policing Software,” PredPol, available at http://www.predpol.com/results.

  49. Samantha Wohlfeil, “Bellingham police to get predictive policing despite concerns,” The Bellingham Herald, August 11, 2015, avaialble at http://www.bellinghamherald.com/news/local/article30797004.html.

  50. Tim Johnson, “Intrado Intrusion: City Council Backs Away from Social Spyware,” Cascadia Weekly, July 9, 2014, available at http://www.cascadiaweekly.com/cw/currents/17003.

  51. Memorandum from Police Chief Clifford Cook to the Bellingham City Council and Mayor, August 10, 2015, available at https://www.cob.org/sirepub/cache/2/hjybbze33q25ko24yselras3/269110829201610113470.PDF

  52. Bureau of Justice Assistance Predictive Policing Software Award Number 2015-DJ-BX-0547, available at http://grants.ojp.usdoj.gov:85/SelectorServer/awards/pdf/award/2015-DJ-BX-0547/2015-H3097-WA-DJ/2015.

  53. Matthew Harwood & Jay Stanley, Power Loves the Dark American Civil Liberties Union, available at https://www.aclu.org/blog/speak-freely/power-loves-dark; Wohlfeil, supra note 59.

  54. City of Bellingham City Council Regular Meeting Agenda, August 15, 2015), available at http://www.cob.org/sirepub/mtgviewer.aspx?meetid=342&doctype=AGENDA.

  55. Maurice Chammah, “Does Predictive Policing Lead to More Police in Black Communities? Readers React,” The Marshall Project (2016), available at https://www.themarshallproject.org/2016/02/09/does-predictive-policing-lead-to-more-police-in-black-communities-readers-react.

  56. Justin Jouvenal, “The new way police are surveilling you: Calculating your threat ‘score,’” The Washington Post, January 10, 2016, available at https://www.washingtonpost.com/local/public-safety/the-new-way-police-are-surveilling-you-calculating-your-threat-score/2016/01/10/e42bccac-8e15-11e5-baf4-bdf37355da0c_story.html.

  57. David Robinson, “Buyer Beware: A hard look at police ‘threat scores,’” January 14, 2016, available at https://medium.com/equal-future/buyer-beware-a-hard-look-at-police-threat-scores-961f73b88b10.

  58. Conor Friedersdorf, “A Police Department’s Secret Formula for Judging Danger,” The Atlantic, Jan. 13, 2016, available at http://www.theatlantic.com/politics/archive/2016/01/a-police-departments-secret-formula-for-judging-danger/423642.

  59. City Council Meeting Minutes on 2016-03-31, City of Fresno, available at http://fresno.granicus.com/MediaPlayer.php?view_id=1&clip_id=224; also see https://docs.google.com/viewerng/viewer?url=http://legistar2.granicus.com/fresno/meetings/2016/3/1079_M_City_Council_16-03-31_Meeting_Minutes.pdf.

  60. Id.

  61. City of Fresno, “Award a sole source contract for predictive policing software to PredPol, Inc. (“PredPol”), for a term of one year in the amount of $80,000, with options for two one-year extensions.,” (2016), available at https://fresno.legistar.com/LegislationDetail.aspx?ID=2601912&GUID=7FDC5705-B39F-4B87-8D64-3F3B9A88AD1A.

  62. Department of Justice, Burea of Justice Assistance, “Edward Byrne Memorial Justice Assistance Grant (JAG) Program — Frequently Asked Questions (FAQs) - Updated June 2016,” 7, available at https://www.bja.gov/Funding/JAGFAQ.pdf.

  63. See Id.: “Common forms of public notification include web-site and newspaper postings, and city council, tribal council, and county board hearings that are open to the public.”

  64. See above discussion regarding Bellingham, WA.

  65. Samuel Walker, “Early Intervention Systems for Law Enforcement Agencies: A Planning and Management Guide,” 2003, available at http://www.cops.usdoj.gov/html/cd_rom/inaction1/pubs/EarlyInterventionSystemsLawEnforcement.pdf.

  66. Steve Mills, “High-tech Tool To Weed Out Bad Cops Proved A Bust,” Chicago Tribune, October 15, 1997, available at http://articles.chicagotribune.com/1997-10-15/news/9710150457_1_police-brutality-police-department-matt-rodriguez.

  67. Id.

  68. Id. Also see: Taras Grescoe, “The Brain And The Badge,” Chicago Tribune, June 30, 1996, available at http://articles.chicagotribune.com/1996-06-30/features/9606300363_1_police-force-internal-affairs-division-chicago-police-department.

  69. Samuel Carton et al., Reducing Adverse Police Interactions Data Science for Social Good (2015), available at https://dssg.uchicago.edu/2015/10/27/reducing-adverse-police-interactions.

  70. “The current EIS captures many more low-risk officers than high-risk ones. As the figure shows, our models can flag more high-risk officers 75 more than the current system) while flagging fewer low-risk officers (180 fewer than the current system).” Samuel Carton, Kenneth Joseph, Ayesha Mahmud, Youngsoo Park, Joe Walsh & Lauren Haynes, “Reducing Adverse Police Interactions,” Data Science for Social Good, October 27, 2015, available at https://dssg.uchicago.edu/2015/10/27/reducing-adverse-police-interactions/

  71. Crystal Cody et al., “Building Better Early Intervention Systems,” The Police Chief 83 (August 2016): 20–25, available at http://www.policechiefmagazine.org/magazine/index.cfm?fuseaction=display&article_id=4228&issue_id=82016.

  72. Id.

  73. Samuel Walker, Early intervention systems for law enforcement agencies: A planning and management guide (2003), US Department of Justice, Office of Community Oriented Policing Services, avilable at https://www.ncjrs.gov/App/abstractdb/AbstractDBDetails.aspx?id=201245; Samuel Walker, Geoffrey P. Alpert & Dennis J. Kenney, Early Warning Systems: Responding to the Problem Police Officer (2001), U.S. Department of Justice Office of Justice Programs, National Institute of Justice, available at https://www.ncjrs.gov/pdffiles1/nij/188565.pdf.

A National Map - What's Happening Where

This interactive map highlights major cities that we found to have considered or adopted predictive policing tools, as well as several smaller cities whose experiences with predictive policing we found potentially instructive for policymakers and advocates. The map does not represent an exhaustive account of all use of predictive tools across the country. Rather, it’s meant to highlight the current state of affairs in certain key departments, based on information from public sources.

View all city data

We asked five basic questions about each city.

In researching these cities, we asked five common questions:

  1. Has the department piloted or tested any predictive policing system?
  2. Has the department adopted a predictive policing system?
  3. If yes to either of those questions, is or was there a public policy governing department’s use of the system?
  4. Did the department solicit community input regarding predictive policing?
  5. Was there any public debate or news coverage?

In order to determine if a police department had used a predictive policing system, we examined: national news reports, local news reports, vendor websites, webinars, whitepapers, and contracts, city council (and city council committee) agendas, resolutions, minutes, and meeting videos, and Bureau of Justice Assistance grants from 2009-2015.

We did not directly contact each department, and we did not file Freedom of Information requests. Some information — particularly which departments were actively exploring adoption of a system — we learned through conversations and meetings with vendors of predictive policing software.

Table 2

Predictive Policing Across the US

Download PDF
City Status Details
Atlanta Used predictive technology The Atlanta PD has been using PredPol since November 2013. Before fully rolling PredPol out across the department in 2013, the Atlanta PD conducted a 90-day pilot, using it in two of APD’s six zones. According to local reports, the Atlanta Police Foundation funded the APD’s purchase of PredPol.
Bellingham, WA Used predictive technology The Bellingham PD will begin using BAIR Analytics’ ATACRAIDS. The Bellingham PD attempted to use DOJ funds in 2014 to purchase Intrado’s Beware, but pushback from constituents and the city council forced the police department to drop their purchase of the software. In 2015, after researching BAIR’s ATACRAIDS and PredPol, the Bellingham PD decided to use DOJ grant money. As the Bellingham City Council Notice for Public Comment noted, the “Bureau of Justice grant requires that an Opportunity for Public Comment be held before the City’s legislative body, however, the City Council is not being asked to make a decision.” The public, however, was only given seven business days to give comment. At the city council meeting, almost every community member spoke against the police department’s purchase.’
Baltimore County Used predictive technology The Baltimore County PD has used predictive models. A 2013 IBM Center for The Business of Government report highlighted the Baltimore County PD as a “place on the frontier” of predictive policing, but it’s not clear what system the BCPD used, if it was only an internal system, or if it still employs any commercial system today.
Baltimore Considering predictive technology The Baltimore PD was recently reported to have “paid for a study by IBM on … predictive policing.” It appears that this was a 6-week study across 2011 and 2012. According to the evaluation of the pilot, “BPD and District command staff, Planning and Research, and District patrol officers were unsatisfied with the pilot, and found little utility in the predictive results provided by the IBM software.”
Boston Considering predictive technology The Boston PD worked with the University of Massachusetts Boston to explore a “methodology for reliably predicting the location, time, and/or likelihood of future criminal activity,” thanks to a $200,000 NIJ grant in 2009. We’ve not found evidence that Boston actually used this methodology in practice, however.
Cincinnati Considering predictive technology In August 2009 the city council approved an ordinance authorizing the city manager to “apply for, accept and appropriate a grant of up to $200,000 from the U.S. Department of Justice…for the purpose of providing funding for the Cincinnati Police Department to develop and demonstrate a predictive policing model.” They city did receive a Smart Policing grant in 2010, but it’s unclear if they were funded to establish a predictive policing model.
Charlotte-Mecklenburg Used predictive technology The Charlotte-Mecklenburg PD has been using Information-Builders’ LEA Predictive Analytics Solution/WebFOCUS RStat since October 2010. Based on our survey, it seems that the Charlotte-Mecklenburg PD is one of the earlier, if not the earliest, adopters of a predictive policing system. During an RFP process in 2009, the “choice came down to Information Builders and IBM.” CMPD went with Information Builders. The CMPD claims that their predictive system has led to “Increased arrests” and a “[s]ingle version of the TRUTH.”’
Chicago Used predictive technology The Chicago PD began using its Strategic Subject List in 2013. Chicago’s Strategic Subject List (SSL), more commonly known as its Heat List, is the most prominent example of a person-based predictive policing system we know of to date. The CPD received more than $2 million to test two phases of the SSL through NIJ funding. The RAND Corporation just released a study on the CPD’s Heat List, finding that the Heat List “does not appear to have been successful in reducing gun violence. The Chicago PD also worked with Rutgers and RTM in 2012 to predict gun violence.”
Fresno Used predictive technology The Fresno PD signed a contract with PredPol in late March 2016. The Fresno PD also piloted Beware. In February 2016, the Fresno PD attempted to enter into a five-year, $132,000 contract with Intrado for Beware. The City Council rejected the contract, though efforts may still be ongoing to revive Beware’s future in Fresno, CA. Separately, the Fresno PD also entered into a one-year, $80,000 contract with PredPol in late March 2016, with the option for two one-year extensions.
Houston Used predictive technology Though the Houston PD does work with Information Builders and uses their WebFOCUS/LEA solution, it’s unclear if they use those solutions in any predictive capacity. According to the Houston PD’s Chief of Police in September 2015, “[w]e even recently authorized the company PredPol to do a test using our data to see if their system is worthwhile. They have not produced any results yet, but there are many other solutions out there.”
Jacksonville Considering predictive technology As of October 2015, the Jacksonville Sheriff’s Office “says there are a number of [predictive policing systems] they’re looking into including the one used in Orange County [PredPol].”
Kansas City Used predictive technology The Kansas City PD has partnered with the Kansas City No Violence Alliance (KCNOVA). The Kansas City PD/KCNOVA joint effort is person-based predictive effort, based on social network analysis and is “done by using data such as friendships, social media activity, and drug use.” The effort appears to have started in earnest in 2012. The program’s aims are similar to Chicago, but “police and researchers use computers to figure out who is most likely to commit murders, robberies and rapes. Software also looks through law enforcement files to uncover who those high-risk individuals know so police can pressure the entire group to steer clear of violence.” The KCPD](http://www.govtech.com/public-safety/Predicting-Crime-Using-Analytics-and-Big-Data.html) was also a pilot site for Risk Terrain Modeling, which the department was still funded for in 2015.
Los Angeles Used predictive technology The LAPD was one of the first adopters of PredPol, in late 2011, early 2012. The LAPD’s use of PredPol is central to PredPol’s marketing efforts and the source of PredPol’s single study regarding the “efficacy” of their system.
Memphis Used predictive technology The Memphis PD has been using IBM’s SPSS predictive analytics since 2009. The Memphis PD is also one of the earlier adopters of a predictive policing system. The program is called “Blue CRUSH” (Blue Criminal Reduction Utilizing Statistical History). It appears the Memphis PD is still using Blue CRUSH, as funding for it was requested in the police department’s 2016 budget and works with the University of Memphis on the program.
Miami-Dade Used predictive technology The Miami-Dade PD uses HunchLab. In 2013, the City of Miami received over $600,000 in DOJ funding to acquire a predictive policing system and work with a researcher who would evaluate the system’s data.
Milpitas, CA Used predictive technology The Milpitas PD recently cancelled its contract with PredPol. The Milpitas PD had entered into a three-year, $37,500 contract with PredPol, but a little less than one-year into the contract cancelled it. According to Milpitas Police Chief Steve Pangelinan, “[a]fter approximately one year of usage, it was our experience that the minimal benefit did not justify continuing costs.”
Nassau County, NY Used predictive technology The Nassau County PD has been using predictive analytics, called Nass-Stat, since 2014.
Suffolk County, NY Used predictive technology The Suffolk County PD has been using predictive analytics, but it’s unclear what vendor or program the Suffolk County PD uses.
Newark Used predictive technology The Newark PD has used Risk Terrain Modeling (RTM) since 2012. The Newark PD worked with Rutgers researches and RTM in 2012 on gun violence.
Nashville Used predictive technology The Metropolitan Nashville PD has used predictive analytics. It’s unclear if the Metropolitan Nashville PD now uses a commercial vendor, but a RAND report notes that the department has used arcGIS mapping software and its own software.
New York City Used predictive technology In July 2015, the NYPD offered notice of their intention to enter into a two-and-a-half year pilot of HunchLab. The pilot, according to the notice, would be “implemented in up to three precincts” and would be “tailored to reflect the priorities of the Department and to measure the initial accuracy of the system.” The NYPD is currently in its second year of the pilot.
Oakland Considering predictive technology Oakland’s Mayor Libby Schaaf proposed spending $158,400 over two years on PredPol in 2015, but we found no evidence of this in Oakland’s FY 2015-2017 budget.
Oklahoma City Used predictive technology The Oklahoma City PD has used BAIR Analytics’ ATACRAIDS since 2014.
Philadelphia Used predictive technology The Philadelphia PD began testing HunchLab in 2013. Philadelphia was the first “big” city to pilot HunchLab, though Toledo, Ohio and Tacoma, Washington purchased HunchLab before Philadelphia.
Phoenix Used predictive technology The Phoenix PD purchased BAIR Analytics’ ATACRAIDS in March 2016. In 2015, the Arizona House passed a bill allocating more than $1 million to pilot “Predpol for a pilot program in Phoenix, Mesa, Maricopa and Sierra Vista. The money would come from the concealed-weapons permit fund.” The Governor ultimately vetoed the bill after strong lobbying from BAIR Analytics. In September 2015, the city of Phoenix amended its contract with BAIR, incorporating ATACRAIDS into the city’s contract. On March 2, 2016, the city of Phoenix authorized payment to LexisNexis Risk Data Management Inc. (who acquired BAIR Analytics) for $13,136.20 for a subscription to ATACRAIDS.
Portland, OR Considering predictive technology In the city’s proposed 2012-2013 budget, the Portland Police Bureau listed predictive policing as a “significant [budget] issue,” but noted that “[f]unds need to be available to ensure the bureau has the ability to implement this.”
Prince George’s County, MD Used predictive technology The Prince George’s County PD has used Information Builders’ WebFOCUS RStat since 2010.
Richmond, VA Used predictive technology The Richmond PD recently discontinued its use of PredPol. The Richmond PD had a three-year, $150,000 contract with PredPol, beginning May 2013. That contract expired in May 2016 and was not renewed. According to Richmond Police Chief Chris Magnus, “[i]n Richmond crime went down, yes, but now it’s going back up. We’re seeing double digit increases.”
San Diego Considering predictive technology The San Diego PD received $485,000 in BJA funding in 2010. The department’s application for the grant indicated that the “project seeks to blend what is known about intelligence-led policing (ILP), predictive policing, problem-oriented policing (POP) and community-oriented policing (COP) into a seamless and measurable effort,” and noted that “[p]redictive analytics will help the command quickly assess and stay focused on hot spots.” However, the grant award only noted that the program would conduct “correctional supervision through predictive analytics.”
San Francisco Considering predictive technology The San Francisco PD held extended talks with PredPol in 2012. One email between the SFPD’s CIO and the Chief of Police note “we we will be rolling out PredPol and the gun violence module,” as of October 30, 2013, the SFPD still had no contract with PredPol as “the department is concerned about launching the program prematurely.”
San Jose Used predictive technology The San Jose PD began using predictive analytics from The Omega Group 2015. In September 2014, San Jose released an RFP for a Crime and Mobile Predictive Analytics Software Suite. Six vendors responded: Corona Solutions, IBM, Information Builders, PredPol, PublicEngines, and The Omega Group. The Omega Group was awarded a five-year, $443,554 contract. PredPol wrote a letter to the city of San Jose to protest this award. The San Jose PD received $160,000 in funding from the California State Drug Forfeiture Fund for this technology. (Note: HunchLab is incorporated into The Omega Group’s CrimeView suite of products.)
Seattle Used predictive technology The Seattle PD has been using PredPol since March 2013 and was one of the first major police departments to adopt PredPol. It also quickly deployed the system department-wide.
St. Louis County Used predictive technology The St. Louis County PD has been using HunchLab since December 2015. Notably, Ferguson, MO is located within St. Louis County. The reported cost is $45,000 for the first year of use, and $35,000 for every subsequent year of use.
Tucson Considering predictive technology The Tucson PD “request[ed] information from vendors who have the ability to manufacture or distribute a predictive policing software system,” in a November 2015 Request for Information through the city’s Department of Procurement.
Washington, DC Considering predictive technology The MPD received over $100,000 in NIJ funding in 2009 to “develop a strategic plan to identify which variables, data, technology and methods are needed to enchance [sic] current predictive policing efforts.” We weren’t able to identify any “current predictive policing efforts.” We are also aware of conversations between the MPD and predictive policing vendors.

What we found: Lots of experimentation, little public discussion.

In a survey of the largest 50 police departments, based on public sources we found that:

Some departments have rejected predictive policing because they aren’t seeing a benefit.

Some police departments, after trying these tools, are moving away from them due to a lack of effectiveness.

In Richmond, California, Police Chief Chris Magnus told the East Bay Express that: “In Richmond crime went down, yes, but now it’s going back up. We’re seeing double digit increases.” Magnus said PredPol’s team worked hard with the city, but that he wasn’t convinced the software helped to reduce crime. “We’re not going to continue it. Our plan going forward is to rely less on predictive policing and more on what we learn through our crime analysis process and through the beat officers’ familiarity with the areas they’re assigned.”84 The Richmond PD’s contract with PredPol ended just months ago.

Similarly, the Milpitas, California Police Department pulled the plug on a three-year contract with PredPol just one year into the agreement. According to Milpitas Police Chief Steve Pangelinan, “[a]fter approximately one year of usage, it was our experience that the minimal benefit did not justify continuing costs.”85

The Baltimore PD also tested a predictive policing system from IBM in 2012. Despite excitement for the system, an evaluation of the six-week pilot found that “BPD and District command staff, Planning and Research, and District patrol officers were unsatisfied with the pilot, and found little utility in the predictive results provided by the IBM software.”86 Further, “[t]he results of the model had little value to the command staff and patrol officers, as it was not clearly operational or easy to interpret.”87

All stakeholders need to learn more.

This survey has several important limitations.

First, public information regarding what police departments are doing is limited. Unlike related investigations into body-worn cameras — another new technology that police departments have rushed to adopt — departments’ experiences with predictive policing systems are much harder to monitor. It’s easy to see when police departments are piloting body-worn cameras — the cameras are worn on the officer’s body, after all. But when a department pilots a predictive policing system, it happens behind-the-scenes — we just don’t have the same line of sight into what is or is not occurring in the central department computers.

Second, the largest 50 departments are not necessarily a representative sample of departments testing or using predictive technology in the U.S. While many police departments certainly take cues from decisions made by the largest police departments, predictive policing systems may play a different role in mid-sized or smaller police departments.

Predictive policing systems are often pitched by vendors as a way to help budget-strained departments. To the extent that mid-sized and smaller-sized police departments are more resource-constrained than larger departments, it’s possible that predictive policing systems seem to be a more attractive investment to small-to-midsize departments than to the largest departments.

PredPol, arguably the most popular predictive policing system on the market, shows how attractive this reasoning has been. Outside of the largest 50 police departments, our research identified at least 34 other police departments that have used PredPol — most of them are small-to-midsize departments.88 Other reports have found that “PredPol is being used in almost 60 departments,”89 and that “more than 150 police departments nationally are deploying predictive policing analytics.” 90

  1. Darwin BondGraham, “Oakland Mayor Schaaf and Police Seek Unproven ‘Predictive Policing’ Software,” East Bay Express, June 24, 2015, available at http://www.eastbayexpress.com/oakland/oakland-mayor-schaaf-and-police-seek-unproven-predictive-policing-software/Content?oid=4362343.

  2. Ian Baure, “Police: tech contract meant to predict, prevent crime in Milpitas nixed,” The Mercury News, July 11, 2016, available at http://www.mercurynews.com/milpitas/ci_30115970/police-tech-contract-meant-predict-prevent-crime-milpitas.

  3. Christine A. Eith, “Evaluation of the Baltimore Police Department Predictive Policing Pilot,” August 2, 2012, 2, available at https://assets.documentcloud.org/documents/3034237/Baltimore-Predictive-Policing-Report.pdf.

  4. Id.

  5. Those departments are: Santa Cruz, CA, Woodland, CA, Los Gatos, CA, Campbell, CA, Morgan Hill, CA, Burbank, CA, Brea, CA Salinas, CA, Piedmont, CA, Alhambra, CA, University of California Berkeley, PD, Livermore, CA, Paramount, CA, El Monte, CA, Richmond, CA, Milpitas, CA, Mountain View, CA, Rio Rancho, NM, Farmers Branch, TX, South Sound, WA, Lakewood, WA, Tacoma, WA, Elgin, IL, Reading, PA, Columbia, SC, Norcross, GA, Ocala, FL, Miami Gardens, FL, Cocoa, FL, Lauderhill, FL, Orange County Sheriff’s Department, FL, Hagerstown, MD, Oxford, AL, Little Rock, AR. The department serving the largest city population is Tacoma, WA, which has an estimated 203,446 residents. The smallest police department of the largest 50 is the Oakland PD, which serves a population of approximately 406,253 — almost double.

  6. Ellen Huet, Server And Protect: Predictive Policing Firm PredPol Promises To Map Crime Before It Happens, Forbes, February 11, 2015, available at http://www.forbes.com/sites/ellenhuet/2015/02/11/predpol-predictive-policing.

  7. Darwin Bond-Graham & Ali Winston, All Tomorrow’s Crimes: The Future of Policing Looks a Lot Like Good Branding, SF Weekly, October 30, 2013, available at http://www.sfweekly.com/sanfrancisco/all-tomorrows-crimes-the-future-of-policing-looks-a-lot-like-good-branding/Content?oid=2827968.

Conclusion

Predictive policing tools are being widely adopted before their impact can be measured, with little transparency and, often, no public engagement. Our research has highlighted significant concerns that must be addressed as departments consider deploying these tools, which directly impact and may harm the communities they serve. This report points to the need for:

Overall, we found little evidence to suggest that today’s systems live up to their billing, and significant reason to fear that these systems, as currently designed and implemented, may actually reinforce disproportionate and discriminatory policing practices.

Acknowledgements

This project is led by David Robinson and Logan Koepke at Upturn, who coauthored this report.

This work would not be possible without the tireless work of our colleagues, allies, and good natured critics. We are grateful to them, and we take responsibility for any errors that remain here. Special thanks to Patrick Ball, Miranda Bogen, Steven Boyd, Alex Busansky, Cheyenne Bridgewater, Jen Carnig, Sakira Cook, Andrew Ferguson, Barry Friedman, Abe Gong, Jennifer Helsby, Eleazer (Lee) D. Hunt, Michael Huggins, Kristian Lum, Morgan McClure, Cathy O’Neil, Aaron Rieke, Bailey Russell, Scott Simpson, Corrine Yu, and Harlan Yu.

We are also grateful to the many organizations that provided valuable feedback on this report, especially the signatories of Predictive Policing Today: A Shared Statement of Civil Rights Concerns.

This report was made possible through the support of the Ford Foundation and the Open Society Foundations.

Close

Return to top