The Reversal Gap

This is the twelfth article in "The Shape of the Next Decade," a series on how AI reshapes work, institutions, and ordinary life. It follows Sovereignty Is Not a Model You Can Download.

Carolyn Crnich certified Humboldt County's results for the November 2008 election on the morning of December 1. That afternoon, volunteer Mitch Trachtenberg told her the count was wrong. His evidence included 197 mail ballots from a single precinct in Eureka that no longer appeared in the official result.

The ballots had been scanned three days before the election, counted on election night, and included in a report printed three weeks later. After that, the county's tabulation software deleted the batch. Its audit log preserved no record that the ballots had ever been there. Fortunately, the paper ballots were still in the building.

Trachtenberg's group had the traditional, now backup method to redo the ballot count. The Humboldt County Election Transparency Project spent 65 hours feeding all sixty thousand paper ballots through an ordinary commercial scanner and used purpose-built software to produce a second tally. It found 216 more ballots than the county system. Trachtenberg traced 197 of them to the Eureka precinct.

Premier Election Solutions, formerly Diebold, examined the database and confirmed that its GEMS software had deleted the batch. Deleting any later batch of ballots could erase the first deck loaded into the system. The company had known about the flaw since 2004 and had quietly supplied some counties with a workaround. California's secretary of state investigated and decertified the version Humboldt was using.

Finding the discrepancy did not by itself repair the result. The paper ballots preserved evidence outside the failed computerized system. The project's independent images and tally made the missing batch reconstructable. Volunteers had done the work of comparing the records, and Crnich had the authority to withdraw what she had certified that morning. Each part was necessary to turn a detected error into an official correction.

The routine control would not have caught it. California's mandatory post-election audit hand-counted one percent of precincts and covered only ballots cast at polling places. The missing 197 had arrived by mail.

Humboldt’s repair was unusually clean because the evidence was sitting in boxes and the correction was arithmetic. A person can have wages garnished for a fraud finding, face collection on a benefit debt, or lose a university place because of an exam result before the institution corrects the record.

Stopping a failing system and reversing its decisions are different institutional acts. The reversal gap is the time between them: the period after the machinery has been switched off but its decisions still matter in people's lives. Humboldt's gap was short because the county could do more than stop the software. It could reconstruct the count and change the result.

Four days

On August 13, 2020, A-level students in England opened results produced by Ofqual's standardization model. Nearly forty percent of the grades came out lower than the assessments submitted by their schools, with the largest effects falling on big state-school cohorts. For a student waiting on a university offer, a one-grade drop could decide whether an offer held.

By August 17, the government and Ofqual had withdrawn the calculated grades. Students would receive the grade submitted by their school, or the calculated grade where it happened to be higher. Gavin Williamson, then the education secretary, conceded that the process had "resulted in more significant inconsistencies than can be resolved through an appeals process."

That appeals process still existed. It simply could not reconsider several hundred thousand results before admissions decisions hardened around them. The government had a faster route because schools had already assessed every student and Ofqual still had those assessments on file. Over the weekend, ministers could use the school-submitted grades instead of rebuilding each result through an individual appeal.

Teacher assessments were an imperfect fallback. Ofqual’s later analysis found that schools had been generous, unevenly so between subjects, which was why the model had been built.

Even so, those assessments let England replace the affected grades in four days without waiting for individual appeals. Michigan had no comparable fallback. In August 2015, it restored human review for new unemployment fraud determinations, but existing fraud records, debts, garnishments, and expired appeal deadlines remained in force.

The morning after the switch

For the previous twenty-two months, MiDAS had determined fraud without human involvement. It cross-checked claimant records against employer, state, and federal data, then spread quarterly income evenly across the weeks. Someone who worked one week and earned nothing the next could therefore appear to have underreported. When MiDAS found a discrepancy, it sent a multiple-choice questionnaire to a state web account that a former claimant might not have checked in years. The questionnaire did not explain what income was in dispute. A wrong answer, or no answer within ten days, could become a finding of intentional fraud.

MiDAS required people to repay the benefits and imposed the maximum penalty, equal to four times the amount received. They had thirty days to appeal, but many did not learn about the fraud finding until that window had closed, when Michigan intercepted a tax refund or garnished their wages. The complaint alleges that the agency had neither answered nor returned any of the last 50,000 calls to its help line before the auditor’s review.

The Sixth Circuit record recounts the allegations of two claimants. MiDAS accused Patti Jo Cahoo of fraud in 2014, but she learned about it more than a year later when a new benefits application was denied. The complaint says she was later evicted after failing to pay rent. Kristen Mendyk learned in 2016 that the system had accused her three years earlier and alleged that the debt drove her into bankruptcy. The court was required to treat both accounts as true while deciding whether the case could proceed, but neither had been established at trial.

MiDAS treated an unanswered questionnaire as evidence of intent, leaving the recipient to disprove an accusation she might never have seen. The Michigan Auditor General reviewed roughly 22,000 cases the agency had identified as fraud and found that about 93 percent of them did not involve fraud. MiDAS could issue accusations by the thousand, while the agency still had to reopen them one at a time.

Fully automated fraud determinations ended in August 2015, but the old findings remained in force. A 2017 settlement required the agency to reverse certain determinations and refund affected claimants. Reviewing cases, correcting records, and returning money took years. Compensation for the harm suffered while those findings stood took longer still. In January 2024, the Michigan Court of Claims approved a separate $20 million settlement for roughly three thousand people, more than eight years after human review was restored.

The repair dragged through litigation, procedural failures, agency resistance, and the sheer number of cases. About a year before MiDAS launched, Michigan had cut roughly a third of its unemployment agency workforce, though public records do not show that MiDAS prompted those cuts or that the smaller staff extended the compensation timeline. MiDAS had removed case-by-case investigation from the act of accusing someone, so repairing thousands of findings meant performing that work only after the damage was done.

What Robodebt had to rebuild

Australia’s welfare department used a program known as Robodebt to identify supposed benefit overpayments from annual tax-office data and issue debts with little or no human investigation. It shifted the burden of checking those debts from the department to the recipient.

Before Robodebt, a discrepancy between reported income and tax records started an investigation. A compliance officer could pull the file, contact the recipient, and ask the employer what the person had actually earned in a given fortnight. Robodebt replaced that work by spreading annual income across the year, asserting a debt from the average, and leaving the recipient to find years-old payslips that showed what had really happened. The process became faster for the department because the person being accused now had to do the investigation.

After Robodebt ended, a royal commission produced 56 recommendations, all of which the government accepted in full or in principle. The recommendations called for a channel through which frontline staff could report what they were seeing, more social-work capacity, administrative-law training, better handling of legal advice, stronger recordkeeping, published review routes, and an oversight council that had been defunded.

Internal lawyers advised in December 2014 that income averaging was inconsistent with the legislation, but the government proceeded. Between 2016 and 2022, the Administrative Appeals Tribunal issued 431 first-tier decisions questioning the scheme's legal basis, all of them unpublished by default. In May 2018, the department was advised not to appeal an adverse ruling because losing could put the methodology into "the public arena." The Administrative Review Council that might have examined the pattern had lost its funding in the 2015–16 budget.

Robodebt kept running because none of those warnings could compel the government to stop. Lawyers could advise, tribunals could overturn individual debts, and staff could see the consequences, but the methodology stayed in place. The one route that could produce a binding public ruling was the route the department had been advised not to take.

In the Netherlands, a court supplied the power Australia’s review bodies lacked. In February 2020, a coalition of Dutch civil-society organizations won a judgment against SyRI, a system that combined government data to detect welfare and tax fraud. The Hague District Court found that the legislation failed the fair-balance test under Article 8 of the European Convention on Human Rights and declared its provisions non-binding. The ruling stopped the system before SyRI produced a repair queue on the scale of MiDAS or Robodebt.

Robodebt ended only after the damage had accumulated. Stopping the scheme prevented new debts, but it could not by itself return money already collected, repair damaged credit, give people back the years spent fighting an unlawful debt, or make every affected person whole.

The strongest case against redundancy

These examples also expose the limit of the argument. Redundancy only helps when the fallback can correct the production system rather than reproduce its failures. Centre-assessed grades carried bias, paper ballots can be lost or mishandled, and appeals can repeat the original decision’s prejudice while adding time and cost. Building a parallel review team around every automated process could make useful systems uneconomic and give threatened departments a respectable vocabulary for defending work that should disappear.

Whether that cost is justified depends on what the system can do to a person. A spelling suggestion needs no independent appeals office, while a system that can label someone a fraud, take wages, deny medical care, or alter a public record should preserve enough evidence for another party to reconstruct what happened. If that reconstruction changes the answer, someone must be reachable, and someone outside the operator must be able to compel a correction.

The fallback does not need to duplicate the production system. It can be much smaller as long as it can reconstruct the decision and reach an institution with the authority to force a correction. Humboldt’s volunteer tally was not an official second count, but it gave the county a reason to open the boxes.

Rehearse how to reverse

Hospitals already show the difference between possessing a backup and being able to operate through it. A study of two hospitals during electronic-record downtime found laboratory results delayed by an average of 62 percent, while paper records were often incomplete or inconsistent. One emergency department had forms ready for an outage but had never rehearsed a comprehensive response, so its staff had not practiced how to continue treating patients without the primary system.

A consequential automated system needs the administrative equivalent of a restore test: proof that it can reverse a harmful decision, not just point to a written appeals policy. The institution would take a sample of adverse decisions and follow them from discovery to remedy using records the production system does not control. Staff would need to recover the evidence, reopen the case, correct the official record, return what is owed, and reach someone with authority if the operator refused. The test would measure how many cases could be repaired in a week and how long each person waited, then ask whether the process still worked after staff turnover or the vendor’s exit.

England could replace calculated grades in four days because another judgment already existed for every student. Michigan spent years reopening accusations that had never received one.

Humboldt County could correct its count because the ballots were still in the building, another tally had exposed the discrepancy, and the registrar could withdraw a certified result. An administrative fallback may be digital and use a different process from the one that produced the decision. What matters is that the evidence survives the original process and reaches someone with the authority to change the official result.

Turning off a failed system prevents new decisions, but it does not undo the ones already embedded in official records, debts, and people’s lives. The reversal gap is the work between shutdown and repair, and it lasts until the institution can clear the accusation, correct the record, return what it took, and tell the affected person that the institution was wrong. Until then, the system is gone but its judgment remains.