Base Rate Fallacy
After wrapping up my series on self-driving cars, I wanted to take some time to focus more on some data science principals that can be useful to think about in daily life. One that is particularly useful to consider when reading the news is the base rate fallacy, which is a type of bias in which people tend to ignore the prevalance (or lack thereof) of an event occuring in favor of individual information. Wikipedia [1] has a great visualization of this, reproduced here:
This diagram shows a situation that was familiar to all of us a year or two ago - vaccines were here and a large majority of the population was vaccinated, but hospitals still had lots of COVID cases, and in many cases the number of vaccinated COVID patients was higher than the number of unvaccinated ones. This led many to question the efficacy of the vaccine - if it worked, wouldn’t there be more unvaccinated COVID patients? However, this logic ignores the base rate of our two classes (vaccinated and unvaccinated). If there are 10 unvaccinated people and 100 vaccinated people in a town, and the hospital has 5 unvaccinated cases and 6 vaccinated cases, then the rate of unvaccinated people being hospitalized for COVID is much larger. It may be helpful to think of this as fractions:
Comparing these rates, as opposed to raw numbers, clearly demonstrates the efficacy of the vaccine - if you are unvaccinated, there is a 50% chance you will be hospitalized for COVID, but that number drops to only 6% for vaccinated people.
Another example of the base rate fallacy can be found in reporting around New York City’s cash bail reform bill, which eliminated cash bail for most offenses. Newspapers eagerly report each time a crime is commited by someone who is out on bail [2, 3], making it seem as if all crime in New York is being committed by hardend felons let loose on the streets. However, the data tells a different story. According to data complied by the City University of New York [4], about 1,300 people are arrested per month for a violent felony offense. Of those 1,300, roughly 100 have an open violent felony case and are out on bail. For additional context, there are about 9,800 people out on bail for a violent felony at any given time. So, using our fractions from before, the percent of violent felonies commited by people who are out on bail is:
This means that 93% of violent felonies would have occurred even without bail reform, since only 7% were commited by those who would have otherwise been incarcerated. This indicates to me that the main focus of bail reform discussion should not be “how does it affect crime rates?”, but rather, “what do we value as a society?” I am not naive enough to think that every person sitting in jail right now is actually innocent and a wonderful human being, but it is abhorrent to me that we would jail 9,800 people to prevent 100 crimes. Instead, we should be focusing on ways to more accurately determine and help those who actually poses a danger to society or will try to dodge accountability, rather than holding large numbers of people in a prison system that is widely known to be dysfunctional [5].
(Side note : There is also data to support this trend for non-violent felonies and misdemenors, but the data is from 2020-2021, when such arrests dropped dramatically due to the pandemic, and it is harder to parse out what actually caused this decline.)
As for ways to spot the base rate fallacy in your own life - be on the lookout for comparisions of raw numbers, instead of rates or fractions. Make sure you are thinking about the entire problem scope of the problem, not just an individual case. And remember that our brains like to hold on to a narrative of a particular story, and it is natural to fall victim to this from time to time.
References
[1] By Marc RumillyVectorization: Mrmw - Own work using:
https://twitter.com/MarcRummy/status/1464178903224889345Own
work based on: Base rate fallacy with vaccines.jpg:, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=128814785
[2] https://nypost.com/2022/08/03/career-criminals-rack-up-nearly-500-arrests-since-ny-bail-reform-began/
[3] https://nypost.com/2023/08/22/driver-who-mowed-down-pedestrians-in-nyc-was-out-on-bail/
[4] http://criminaljustice.cityofnewyork.us/wp-content/uploads/2021/12/Pretrial-Docketed-Rearrest-Contextual-Overview-December-2021-Update.pdf
[5] https://www.nytimes.com/topic/organization/rikers-island-prison-complex
This will help me when reading the newspaper!