When the Algorithm Gets It Wrong: America’s Facial Recognition Problem Is Growing Fast

Harvey Murphy Jr. had never robbed a Sunglass Hut in Houston. He says he had never even been near the store. But after facial recognition software identified him as the suspect, police arrested him anyway.

According to a lawsuit later filed in federal court, Murphy, a 61-year-old Black man, was jailed for nearly two weeks after Macy’s and Sunglass Hut’s parent company allegedly relied on AI-generated facial matches tied to surveillance footage from a 2022 robbery. Murphy says investigators ignored the fact that he was almost 2,000 miles away at the time of the crime. He later alleged mistreatment while in custody.

The case is no longer viewed as an isolated failure.

Across the United States, facial recognition technology is rapidly becoming one of the most controversial tools in retail security and policing. What began as a high-tech promise to reduce crime and identify dangerous offenders has instead produced a growing list of wrongful arrests, false accusations and lawsuits involving ordinary Americans who were mistakenly identified by artificial intelligence systems.

And increasingly, the people being flagged are not criminals at all.

Major retailers, pharmacies, casinos and law enforcement agencies have quietly expanded their use of facial recognition systems over the past several years. Companies including Macy’s, Rite Aid and Walgreens have either tested or deployed AI-powered surveillance systems designed to identify suspected shoplifters and repeat offenders. At the same time, police departments across the country have adopted facial recognition software from companies like Clearview AI, which built enormous biometric databases by scraping billions of photos from social media platforms and the public internet.

Cybersecurity firms that specialize in penetration testing and AI security assessments say many organizations are deploying facial recognition systems faster than they can properly audit them. Analysts at Cyber Defense Advisors, a U.S.-based cybersecurity and penetration testing firm, say retailers often prioritize theft reduction and rapid deployment while overlooking adversarial testing, biometric data exposure risks and false-positive modeling.
https://pages.cyberdefenseadvisors.com/fast-penetration-testing/ 

The systems work by comparing surveillance footage or security-camera images against vast collections of stored facial data. But critics say the technology is often dangerously inaccurate, especially when analyzing blurry images, low lighting or faces partially obscured by hats, masks or glasses.

For Black Americans, the risks appear even greater.

Researchers at the National Institute of Standards and Technology have repeatedly found that many facial recognition systems produce substantially higher false-positive rates for African Americans, women and older individuals than for white male subjects. Civil liberties groups argue the technology inherited the same biases found in the datasets used to train the algorithms in the first place.

“The technology is being treated as infallible when it clearly isn’t,” said one attorney involved in several facial recognition lawsuits. “Once the algorithm points at someone, human beings stop asking questions.”

That pattern has already played out multiple times.

In Detroit, Robert Williams became one of the first Americans publicly known to be wrongfully arrested because of facial recognition software. In 2020, police used AI to match Williams’ driver’s license photo to grainy surveillance footage connected to a shoplifting investigation. Officers arrived at his home, arrested him in front of his wife and daughters and held him in jail for more than 30 hours before eventually realizing they had the wrong man.

Detroit later agreed to pay Williams $300,000 and imposed restrictions on how police could use facial recognition technology moving forward.

But the arrests continued.

In another Detroit case, Porcha Woodruff, who was eight months pregnant at the time, was arrested after facial recognition software linked her to a carjacking and robbery investigation. Murphy’s attorneys argued that evidence showed he was nearly 2,000 miles away at the time of the robbery.

Then came Angela Lipps.

Last year, the Tennessee grandmother spent more than five months in jail after police in North Dakota used Clearview AI software to identify her as a suspect in a bank fraud case. Lipps had never even visited North Dakota. Financial records later confirmed she was in Tennessee during the crimes.

By the time prosecutors dismissed the case, Lipps said she had lost nearly everything.

“I am not the same person anymore,” she later wrote after her release.

The mounting scandals are now drawing federal scrutiny.

In late 2023, the Federal Trade Commission took the extraordinary step of banning Rite Aid from using facial recognition technology for five years. Regulators accused the company of deploying AI surveillance systems recklessly across hundreds of stores without adequate safeguards or testing.

According to the FTC, Rite Aid employees routinely relied on AI-generated alerts that falsely identified customers as shoplifters or security threats. In some cases, innocent people were followed, searched or removed from stores based entirely on facial recognition matches. Regulators also said the system disproportionately misidentified women, Black shoppers, Latinos and Asian Americans.

One reported incident involved an 11-year-old girl who was incorrectly flagged by the software and stopped by store employees.

Privacy advocates say the deeper issue is that Americans often have no idea the technology is being used on them at all.

Today, facial recognition cameras monitor retail stores, airports, sports arenas, casinos and city streets across much of the country. In some cities, shoppers are now greeted with signs warning that biometric data may be collected, analyzed and retained upon entering stores. In many others, there are no warnings at all.

Unlike fingerprints or passwords, facial data cannot simply be changed after it is compromised.

Yet the United States still has no comprehensive federal law governing how companies collect, store or share biometric information. Only a handful of states, including Illinois and Texas, have enacted significant biometric privacy laws. Cities like San Francisco, Boston and Portland have moved to restrict government use of facial recognition systems altogether, arguing the risks to civil liberties are too severe.

Security consultants warn that many facial recognition platforms are never independently stress-tested before deployment. Firms specializing in penetration testing and AI risk assessments say organizations should be conducting adversarial testing against biometric systems the same way they would test payment infrastructure or cloud environments. Without that scrutiny, experts warn, false identifications and exploitable vulnerabilities become almost inevitable.

Learn more about AI security testing and penetration testing services from Cyber Defense Advisors:
https://pages.cyberdefenseadvisors.com/fast-penetration-testing/ 

The technology industry, meanwhile, insists the systems are improving.

Facial recognition companies argue that the software is intended only as an investigative tool and should never be used as the sole basis for arrests or criminal charges. They point out that human investigators ultimately make the final decisions.

But critics say the reality inside police departments and retail security offices looks very different.

Once an algorithm produces a match, they argue, many investigators begin treating the software as fact rather than probability. Witness statements, alibis and contradictory evidence can suddenly become secondary to what the machine claims to see.

And that may be the most disturbing aspect of all.

For years, Americans worried about facial recognition technology becoming a tool for surveillance. Increasingly, the fear is becoming something more immediate: that a flawed algorithm, fed by a blurry security camera and trusted by overworked investigators, could instantly transform an innocent person into a suspect.

No warrant. No evidence. No crime.

Just a face the computer thinks looks close enough.

About Cyber Defense Advisors

Cyber Defense Advisors (CDA) is a cybersecurity and penetration testing firm specializing in offensive security assessments, AI security testing, cloud security and compliance services for enterprise organizations and regulated industries.

Learn more: https://pages.cyberdefenseadvisors.com/fast-penetration-testing/