AI, Biopolitics &
Engineered
Oppression

AI systems are marketed as objective, scientific, and progressive, but they actually reproduce and amplify existing racial, gender, and class hierarchies. The "New Jim Code" describes how technology appears neutral while automating discrimination.

Beauty AI (Benjamin): All 44 winners across age groups except six were White; only one finalist had visibly dark skin. The algorithm was trained on biased data that privileged White features.

Obermeyer et al. (Health Algorithm): A widely used commercial health algorithm systematically underestimated the health needs of Black patients because it used healthcare costs as a proxy for need. Black patients had lower costs due to historical healthcare discrimination, so the algorithm read "lower cost" as "lower need" — even when they were actually sicker.

COMPAS Recidivism Algorithm (Benjamin): ProPublica found COMPAS was nearly twice as likely to falsely flag Black defendants as "high risk" compared to white defendants, and twice as likely to falsely flag white defendants as "low risk."

Amazon's Facial Recognition (Benjamin): The ACLU tested Amazon's Rekognition on members of Congress. The system misidentified 28 members as criminal suspects — 39% of false matches were people of color, though people of color make up only 20% of Congress.

HP Webcam (Browne): Black Desi Cryer's video showed an HP webcam that could track "white Wanda" but could not detect or follow his face. HP responded that the technology "is built on standard algorithms that measure the difference in intensity of contrast between the eyes and the upper cheek and nose" — meaning the algorithm was optimized for lighter skin.

Credit Scoring (Nopper): The FICO score was marketed as taking "prejudice out of the equation," yet it systematically disadvantages Black and brown communities who have been denied access to credit through redlining and other discriminatory practices.

Predictive systems are trained on data generated by biased policing, discriminatory courts, and unequal institutions. The algorithms inherit and amplify these biases, creating feedback loops that deepen inequality.

Chicago's Strategic Subjects List (RAND study): The SSL was built using arrest data from periods when the Chicago Police Department was under federal investigation for unconstitutional stops, racial bias, and data manipulation. The model identified less than 1% of actual homicide victims but increased arrests for shootings — not through prevention, but because police used the list as a suspect checklist after crimes occurred.

New Orleans Palantir System (Richardson et al.): The NOPD secretly contracted with Palantir for predictive analytics using data from 2005-2011, when officers targeted Black and transgender residents and fabricated evidence. The predictive outputs described perpetrators as "overwhelmingly young, African American, male" — directly mirroring the department's biased enforcement patterns.

Maricopa County (Richardson et al.): The Maricopa County Sheriff's Office was found guilty of racial profiling against Latinos. Multiple cities in the county use predictive policing and share data via regional platforms, meaning dirty MCSO data contaminated systems in neighboring jurisdictions.

PredPol's False Claims (Richardson et al.): PredPol claims to exclude drug arrests and traffic stops to avoid bias, but still uses 911 call data — which is influenced by racial bias in reporting (e.g., "shopping while Black" calls). Vendors do not independently validate police data.

The Confirmation Feedback Loop (Richardson et al.): Biased data leads police to revisit over-policed neighborhoods, generating more data from those areas, which then justifies further policing. In NYC, 311 calls for "quality of life" offenses spiked in gentrifying neighborhoods, leading to more police presence and arrests in Black and Latino areas — reinforcing the data loop.

Technologies marketed as "public safety," "decarbonization," or "efficiency" actually expand police power, deepen surveillance, and create new mechanisms of social control. "Reform" often means making the carceral system more efficient, not shrinking it.

Promise vs. Appolition (Benjamin): Promise is Jay-Z's "decarceration" startup that uses app-based tracking and GPS monitoring as an alternative to pretrial detention. Benjamin critiques it because it charges $17.80 per day (creating a "digital cell"), collaborates with law enforcement, extends surveillance into everyday life, and is part of the "technocorrections" industry. In contrast, Appolition is a bail crowdfunding app that actually frees people — raising $230,000 and freeing over 65 people.

NYPD Protest Surveillance (Cagle & Hirsch): In summer 2020, the SFPD illegally accessed hundreds of private surveillance cameras to place Black Lives Matter protests under live surveillance without democratic approval. The city's facial recognition ban stopped the SFPD from scanning faces and logging identities, but surveillance of protected First Amendment activity still occurred.

Israeli Border Technology (Dawson): Hermes drones manufactured by Israel's Elbit Systems conduct killings of civilians en masse in Gaza. The same drones were deployed at the U.S.-Mexico border (unarmed, but the technology is identical). Elbit's surveillance towers are marketed as preventing "infiltrators" in Palestine and "illegal immigrants" in the United States — both share the stated justification of preventing "terrorism."

Palantir in LAPD (Brayne): The LAPD uses Palantir, a data integration platform originally designed for national security, to merge data from police records, license plate readers, field interview cards, and non-police sources like utility bills, social media, and pizza delivery records. This blurs the line between traditional policing and intelligence-style surveillance.

Automatic License Plate Readers (Brayne): ALPRs capture data on every passing car, not just suspects. A person whose car is recorded near a crime scene can be added to a database without ever having police contact. ALPRs are deployed more in high-crime (often minority) areas, so residents there are more likely to be tracked.

Algorithms are often proprietary "trade secrets," making it impossible to audit them for bias or error. When systems fail, no one is held responsible — accountability is displaced onto "the algorithm" itself.

Houston Teacher Evaluation Algorithm (AI Litigation Report): The Houston school district refused to show teachers the algorithm evaluating them, claiming it was a trade secret owned by a third-party vendor. Teachers won a constitutional due process argument forcing disclosure.

Idaho Medicaid Algorithm (AI Litigation Report): A secret algorithm was used to cut Medicaid benefits for people with disabilities. Recipients were given no notice, explanation, or appeal. The algorithm's design specifications were considered trade secrets.

Arkansas Medicaid Algorithm (AI Litigation Report): An algorithmic system allocating home health care failed to recognize cerebral palsy and diabetes due to flawed design and coding errors. This was only discovered during litigation when experts examined the code.

D.C. Juvenile Risk Assessment (AI Litigation Report): A judge rejected a violence risk assessment tool for lacking scientific validity. The tool used factors like "parental criminality" and "community disorganization" — proxies for race — resulting in biased "high risk" scores for youth of color. The tool was based on only two outdated, non-peer-reviewed studies.

FBI Face Recognition (Garvie et al.): Maryland's face recognition system, which includes the license photos of over two million residents, was launched in 2011 and had never been audited. Only 9 of 52 agencies audit searches. The FBI hasn't audited its system either.

Drone Strike Secrecy (Crawford): The CIA has secretly conducted hundreds of drone strikes in Pakistan and Yemen. The criteria for choosing targets, procedures to protect civilians, and the actual number of civilian casualties are all concealed. The Obama administration counted all military-age males in a strike zone as combatants "unless there is explicit intelligence posthumously proving them innocent."

Automated systems allow institutions to distance themselves from violent outcomes. When an algorithm makes a decision, no individual feels responsible — "the computer did it."

"Drop Weapons" in Iraq and Afghanistan (Crawford): Soldiers were encouraged to carry "drop weapons" (AK-47s or shovels) so that if they accidentally killed an innocent person, they could plant the weapon to make the victim look like an insurgent. In Afghanistan, soldiers from the Fifth Stryker Brigade attacked civilians, planted weapons next to victims, and some took body parts as trophies. Staff Sergeant Hensley said: "Was it a good kill? It's a good kill because I say it's a good kill."

Automated Welfare Decisions (Benjamin): "Automated welfare decisions are not magically fairer than their human counterparts. Discrimination is displaced and accountability is outsourced."

Risk Assessment in Sentencing (Starr): EBS instruments produce higher risk estimates for poor, young, male, and minority defendants, exacerbating existing disparities. The "scientific" veneer makes discrimination appear objective.

China's Social Credit System (Benjamin): "The system not only investigates behaviour — it shapes it." Residents of Xinjiang are forced to download an app tracking "terrorist and illegal content" — targeting Muslim minorities.

Data is extracted from poor and minority communities as "raw material" for profit, while those communities receive no benefit and suffer increased surveillance and control. This mirrors historical colonialism.

FinTech in Africa (Birhane): Microfinancing apps like Tala and Branch use smartphone data (browsing history, likes, locations) to assess creditworthiness. Rather than "lifting people out of poverty," FinTech profits from poverty. Safaricom reported $620 million in annual profit directed to investors while leaving Kenya's poorest in perpetual debt. As Bateman says: "Like the colonial era mining operations that exploited Africa's mineral wealth, the microcredit industry in Africa essentially exists today for no other reason than to extract value from the poorest communities."

Kenya's Biometric ID System (Birhane): The national biometric ID system risks excluding racial, ethnic, and religious minorities who have historically faced challenges acquiring documentation. If implemented, these minority groups would be rendered stateless and face challenges registering a business, getting a job, or traveling.

Digital Colonialism (Kwet): U.S. tech monopolies (GAFAM) control the three core pillars of the digital ecosystem: software, hardware, and network connectivity. Facebook Free Basics offers limited, curated internet to the poor, violates net neutrality, and funnels users into Facebook's ecosystem. The NSA partners with tech corporations for mass surveillance in the Global South.

The "Scored Society" (Nopper): Marketplace lenders mine social media data (Facebook, Twitter, LinkedIn) and texting patterns to assess creditworthiness. This profiling often happens without explicit consent and can include data from before the loan application. The "scored society" makes algorithmic scoring a condition of economic survival.

Facial recognition systems misidentify people of color at dramatically higher rates than white people, and law enforcement uses them to target activists, protesters, and minority communities.

NYPD Misidentification (Amnesty International): Facial recognition has been used 22,000 times in New York City since 2017. Black and minority communities are at risk of being misidentified and falsely arrested — in some instances, facial recognition has been 95% inaccurate.

Detroit False Arrests (Amnesty International): Examples include false arrests of Black men in Detroit, where facial recognition led to wrongful arrests.

Black Lives Matter Activist (Amnesty International): NYPD officers attempted to arrest Derrick "Dwirek" Ingram by besieging his apartment for five hours after seemingly using facial recognition to identify and locate him. A reporter caught a facial recognition query report print-out in an officer's hand with Ingram's face and name on it.

Atlantic Plaza Towers (Amnesty International): Residents of a Black-majority complex in Brooklyn successfully resisted attempts to install facial recognition cameras. In response, the landlord sent tenants full-color surveillance print-outs of them standing in the lobby with their apartment numbers and time and date written on the photos — an act of intimidation through surveillance.

FBI Co-authored Study (Garvie et al.): FBI co-authored research suggests face recognition may be less accurate on Black people. Also, due to disproportionately high arrest rates, systems that rely on mug shot databases include a disproportionate number of African Americans.

San Francisco Ban (Cagle & Hirsch): In 2019, San Francisco passed the nation's first ban on municipal facial recognition. In 2022, Mayor Breed proposed a ballot measure to gut the law, but withdrew it after a citywide poll showed 60% of San Franciscans opposed giving SFPD access to private cameras for live surveillance.

Despite the scientific consensus that race is a social construct, genetic surveillance and biometric technologies treat race as a biological fact, re-racializing populations under the authority of science.

CODIS DNA Database (Roberts): The FBI's national DNA database (CODIS) vastly over-represents Black people because they are overrepresented in the criminal justice system. This means Black individuals are much more likely to be connected to a crime by familial DNA matching — even for crimes they did not commit — simply because their DNA or a relative's DNA is already in the database.

Grim Sleeper Case (Roberts): California's DNA database is 40% Black (while Black people make up 6% of the state population), meaning familial DNA searching in California is inherently targeted at Black families. Every Black person with a relative who has been arrested is effectively enrolled in a genetic surveillance database without consent.

Ferguson v. City of Charleston (Roberts): A public hospital in Charleston implemented a policy of drug testing pregnant women and reporting positive results to police. Almost all women arrested under this policy were Black. The hospital was functioning as a law enforcement arm specifically targeting Black women's pregnancies.

The Shirley Card (Roth): The Shirley card — a color reference card featuring a light-skinned Caucasian woman — was the industry standard for calibrating skin tones in photography and television. Film emulsions, digital sensors, and TV cameras were designed with Caucasian skin tones as the default "norm," resulting in poor reproduction of darker skin tones.

The Pain Treatment Disparity (Washington): A 2016 study in PNAS found that medical students and residents held false beliefs about biological differences between Black and white patients — including beliefs that Black people have thicker skin, less sensitive nerve endings, and higher pain tolerance. These beliefs, traceable to 19th-century race science developed through experimentation on enslaved people, predicted racial disparities in pain treatment.

Research claiming to predict criminality from facial features is based on unsound premises, uses biased data, and creates dangerous feedback loops that justify the repression of marginalized populations.

Harrisburg University Study (Coalition for Critical Technology): Researchers claimed to "predict if someone is a criminal based solely on a picture of their face" with "80 percent accuracy and with no racial bias." The Coalition for Critical Technology responded: "there is no way to develop a system that can predict or identify 'criminality' that is not racially biased — because the category of 'criminality' itself is racially biased."

No Such Pattern Exists (Coalition for Critical Technology): "No such pattern exists for facial features and criminality, because having a face that looks a certain way does not cause an individual to commit a crime — there simply is no 'physical features to criminality' function in nature."

Dangerous Feedback Loops (Coalition for Critical Technology): Because "criminality" operates as a proxy for race due to racially discriminatory policing, research of this nature creates dangerous feedback loops. Policing based on algorithmic recommendations generates more data fed back into the system, reproducing biased results.

The Drapetomania Precedent (Metzl): In 1851, Louisiana physician Samuel Cartwright proposed a psychiatric diagnosis called "drapetomania" — a supposed mental illness that caused enslaved people to flee captivity. The "treatment" was whipping. The criminality prediction research is the direct descendant of this tradition — the same logic, applied through new technical mechanisms.

Marginalized communities — the poor, racial minorities, Indigenous peoples, women, and disabled people — bear the brunt of harmful AI deployments, while the benefits accrue to the wealthy and powerful.

The Poor (Birhane): "The harms of digitization and 'technological solutions' affect individuals and communities that are already at the margins of society." Utilitarian AI ethics focus on "the greatest good for the greatest number," which by definition means solutions that centre minorities are never sought.

Indigenous Peoples (Dawson): Israeli surveillance technology deployed at the U.S.-Mexico border explicitly targets low-income communities, migrant groups, and indigenous peoples. Israel casts a "long shadow over migrant and refugee trails that are traveled by mostly low-income and indigenous peoples."

People with Disabilities (AI Litigation Report): In Arkansas, an algorithmic system allocating home health care to Medicaid patients failed to accurately understand the care needs of patients with cerebral palsy or diabetes. In Idaho, a secret algorithm cut Medicaid benefits for people with disabilities without explanation or appeal.

Transgender People (Benjamin): Appolition was created by Dr. Kortney Ziegler, a Black trans tech developer. The carceral system disproportionately targets and incarcerates trans people, especially Black trans women.

Prisoners (Washington): Under dermatologist Albert Kligman at Holmesburg Prison in Philadelphia, massive pharmaceutical and cosmetic testing was conducted on prisoners — overwhelmingly Black men. Companies including Johnson & Johnson, Dow Chemical, and the US Army tested products including dioxin (Agent Orange), radioactive isotopes, mind-altering drugs, and experimental skin treatments. Kligman described his first visit as seeing "acres of skin."

Predictive policing programs have not been shown to reduce gun violence or crime. Instead, they increase police contact with flagged individuals, generate more arrests, and expand surveillance without delivering public safety benefits.

Chicago SSL Pilot (RAND study): The SSL pilot did not reduce gun violence. Individuals on the SSL were 2.88 times more likely to be arrested for a shooting — not because of increased police contact, but because police used the list as a suspect checklist after shootings occurred. The model identified less than 1% of homicide victims (3 out of 405). Even "high-risk" individuals had only a 0.7% homicide rate over 12 months.

Poor Implementation (RAND study): In over two-thirds (68.8%) of presentations observed, there was no mention of the SSL. Prevention strategies were vague — mainly "increased contact" without clear guidance. One commander said: "We didn't know what to do with the list — so we just made more stops."

No Reduction in Gun Violence (RAND study): Out of 405 homicides in Chicago during the study period, only 3 victims were on the SSL — less than 1%. The SSL did not lower citywide murder rates.

Confirmation Bias (RAND study): The RAND study found that the SSL had a direct effect on increasing arrests for shootings that did not work through police contact. Police used the SSL as a suspect lead after shootings occurred, creating a self-fulfilling prophecy.

The belief that technology can solve social problems (tech solutionism) distracts from the structural changes needed to address inequality. "Fixing" algorithms leaves the underlying unjust systems intact.

Algorithmic Laundering (Obermeyer et al.): The health algorithm laundered existing structural inequalities — making them appear to be neutral technical outputs rather than the products of historical racism. The algorithm didn't say "this patient is Black, give them less care." It said "this patient has lower predicted costs, give them lower priority."

The Myth of Neutrality (Benjamin): "Technology is not neutral. It is always designed within social contexts that encode existing power relations, and in the American context, that means technology encodes anti-Blackness, racial capitalism, and carceral logic."

Reform vs. Abolition (Friedman): "Liberal reform seeks to revise within the carceral rather than imagine a world without it. Rather than die a swift death or even slowly, the carceral state evolves when faced with a direct challenge and further entrenches within our social institutions." Reforming the carceral state is "where the racial contract goes to receive diversity and inclusion training."

The Dialectics of Discipline (Friedman): Felber's "dialectics of discipline" describes how the carceral state morphed alongside the Black Freedom Movement, refining its technologies of surveillance and control when met with a coordinated attack. In times of distress, the carceral state responded with liberal reform — interpreting legal victories as opportunities to develop new carceral technologies.

The Double Bind of History Work (Guntarik, "Gathering"): The FIG project used gamified incentives (free coffee, treasure hunts) to encourage participation. Guntarik acknowledges: "In no way did I wish to dwarf these histories through the playing of a game. And yet this blurring of the lines between play and provocation and between people's histories and painful memories can have limitations. This is the double bind of history work."

Surveillance capitalism unilaterally claims human experience as free raw material for profit, predicting and shaping behavior at scale. This "instrumentarian power" threatens democracy, autonomy, and the right to a future tense.

Definition (Zuboff): "Surveillance capitalism unilaterally claims human experience as free raw material for translation into behavioral data. . . . With this reorientation from knowledge to power, it is no longer enough to automate information flows about us; the goal now is to automate us."

The Aware Home Lost Vision (Zuboff): In 2000, the Aware Home project envisioned a "human-home symbiosis" where data would empower individuals and remain under their control. By 2018, this vision was "gone with the wind." Today's smart-home devices extract and monetize personal data with oppressive terms of service.

Instrumentarian Power (Zuboff): "Instrumentarian power knows and shapes human behavior toward others' ends. Instead of armaments and armies, it works its will through the automated medium of an increasingly ubiquitous computational architecture of 'smart' networked devices, things, and spaces."

The Right to the Future Tense (Zuboff): Surveillance capitalism threatens "the elemental right to the future tense, which accounts for the individual's ability to imagine, intend, promise, and construct a future. It is an essential condition of free will and, more poignantly, of the inner resources from which we draw the will to will."

Google as Pioneer (Zuboff): "Google invented and perfected surveillance capitalism in much the same way that a century ago General Motors invented and perfected managerial capitalism."

U.S. tech monopolies exercise control over the Global South through software, hardware, and network connectivity — a twenty-first-century form of colonization that extracts data and wealth while leaving dependent economies.

Definition (Kwet): "In an insidious new phenomenon, digital colonialism, casts a shadow on the Global South. This structural form of domination is exercised through the centralised ownership and control of the three core pillars of the digital ecosystem: software, hardware, and network connectivity."

Nigeria's Software Imports (Birhane): "Nigeria, one of the more technically developed countries in Africa, imports 90% of all software used in the country. The local production of software is reduced to add-ons or extensions creation for mainstream packaged software."

Algorithmic Colonialism (Birhane): "Political, economic, and ideological domination in the age of AI takes the form of 'technological innovation,' 'state-of-the-art algorithms,' and 'AI solutions' to social problems. Algorithmic colonialism, driven by profit maximization at any cost, assumes that the human soul, behaviour, and action is raw material free for the taking."

Free Basics as Gatekeeping (Kwet): "Free Basics not only has Facebook playing Internet gatekeeper of the poor, it also violates net neutrality laws." Facebook controls what internet content poor people can access.

Data Extraction as Human Resource Mining (Birhane): "Knowledge, authority, and power to sort, categorize, and order human activity rests with the technologist, for which we are merely data producing 'human natural resources.'"

Algorithmic risk assessment tools used in sentencing and parole explicitly use demographic and socioeconomic variables (gender, age, employment, education) that correlate with race and class, violating equal protection principles.

Gender Discrimination (Starr): EBS tools give all men higher risk scores than women solely based on group averages — even though individual men may not pose higher risk. The Supreme Court in Craig v. Boren rejected using statistical differences between young men and women to justify gender-based drinking ages. The same principle applies to sentencing.

Poverty as Aggravating Factor (Starr): EBS tools assign higher risk to unemployed or less-educated defendants, treating poverty as an aggravating factor. In Bearden v. Georgia, the Supreme Court forbade revoking probation for failure to pay restitution based on poverty — even if data showed a correlation between poverty and recidivism.

Racial Proxy Variables (Starr): Although race is not directly included, variables like unemployment, low education, and neighborhood crime are highly correlated with race. A young Black man from a high-poverty neighborhood will score higher on risk tools than a white college graduate with the same crime — leading to longer sentences.

Imprecise Individual Predictions (Starr): A model might predict that men aged 20-24 with no high school diploma have a 40% recidivism rate on average. But the 95% prediction interval for an individual could range from 5% to 90% — so the instrument cannot reliably say whether a specific person will reoffend.

Wrong Question Asked (Starr): Risk prediction tools do not answer what judges need to know: "Will two years in prison reduce his future crime more than probation would?" They only predict who is risky, not whether incarceration reduces that risk.

Focusing on individual identity and "diversity" without changing the norms and structures of the tech industry offers little more than cosmetic change. The problem is not just who designs technology, but the logic of extraction and control embedded in the industry.

Cosmetic Diversity (Benjamin): "Cosmetic diversity too easily stands in for substantive change, with a focus on feel-good differences like food, language, and dress, not on systemic disadvantages."

The Default Imagined User (Costanza-Chock): "Designers most frequently assume that the unmarked user has access to a number of very powerful privileges, such as U.S. citizenship, English language proficiency, access to broadband internet, a smartphone, no disabilities, and so on."

Diversifying the Workforce Is Not Enough (Costanza-Chock): "Diversifying the software workforce, unfortunately, will not automatically produce a more diverse default imagined user. Unless the gender identity, sexual orientation, race/ethnicity, age, nationality, language, immigration status, and other aspects of end user identity are specified in advance, the imagined user for whom technology design teams develop products tends to default to the dominant social group."

From Equity to Accountability (Costanza-Chock): "Ultimately, we have moved from an argument for equity (we need diverse designers, and diverse users) to an argument for accountability (those most affected by the outcomes should lead and own digital design processes and products)."

Single-Axis Analysis (Costanza-Chock): "Universalist design principles and practices, and even evaluations of fairness or equity in design that are single-axis, erase certain groups of people, specifically those who are intersectionally disadvantaged or multiply-burdened under white supremacist heteropatriarchy, capitalism, and settler colonialism."