You are, and that does not change when the decision was made by software. Accountability does not transfer with the task. The mathematician Cathy O'Neil argues in “Weapons of Math Destruction” that models are not impartial instruments but encoded judgments, built by people, optimizing for whatever their designers decided success meant. Her central warning is not that algorithms are dangerous. It is that an algorithm nobody can interrogate will cause harm nobody can trace, and her remedy is that they should be audited regularly for accuracy and fairness. The AI Alignment Filter™ gates adoption. Governance is what happens after, and it rests on four questions: who owns this? What does it do when uncertain? How would we know if it were wrong? And who can decide to stop using it?
Introduction
Every conversation about artificial intelligence in a practice eventually reaches the purchase decision and then stops.
Should we adopt this? Does it work? What does it cost? And once those are answered, the conversation ends, the tool goes in, and the practice owner is left alone with something that is now making calls inside her workflow, with no agreed answer to the only question that will matter if it goes wrong.
This is the week that question gets answered.
It is also the week almost nobody writes, which is why it is the one the archive most needs. The purchase is interesting. Governance is not. But governance is where the consequences live, and a tool that was adopted carefully and then governed by nobody is in exactly the same position as one that was adopted carelessly.
Key Takeaways
• Accountability does not transfer with the task. You can delegate work to software. You cannot delegate responsibility for the outcome.
• Cathy O'Neil's argument is that models encode their designers' beliefs and optimize for whatever those designers decided success meant.
• Models substitute proxies for things they cannot measure, and proxies are easier to game than the reality they stand for.
• A model inside a black box causes harm nobody can trace, which is why O'Neil's remedy is regular auditing rather than prohibition.
• Four governance questions: who owns it, what does it do when uncertain, how would we know if it were wrong, and who can turn it off.
• Wide adoption is not evidence that a system works, and one of the best-known cautionary cases in health technology proves it.
Models Are Not Neutral
Cathy O'Neil spent her career on both sides of this. A mathematics professor, then a quantitative analyst on Wall Street, then a data scientist, and her book Weapons of Math Destruction is written by someone who built these things before she criticized them.
Her central claim is not that algorithms are bad. She is explicit that they are not inherently so. It is that a model is a set of human judgments expressed in arithmetic, and that people mistake the arithmetic for objectivity.
Somebody decided what the model would optimize for. Somebody decided what counted as success. Somebody decided which data was available and therefore which questions could be asked. Every one of those was a judgment, made by a person, and none of them is visible in the output.
Her sharpest observation is about what happens to a model that produces the answer its owner wanted. It gets adopted, it gets trusted, and it stops being examined, which makes it a closed loop rather than a measurement.
For a practice owner, the translation is direct. When a tool tells you something, you are not receiving a fact about the world. You are receiving the output of a set of decisions somebody else made about what matters, and you are the one who will answer for acting on it.
The Proxy Problem
The most useful idea in O'Neil's book, and the one that transfers furthest outside her examples, is the proxy.
A model cannot measure the thing you actually care about, because the thing you care about is complicated, slow to observe, or not recorded anywhere. So it measures something adjacent and available and treats that as a stand-in. Her example is a postcode standing in for whether somebody will repay a loan.
Two problems follow, and they compound.
The first is that the proxy is not the thing. It correlates with it, sometimes strongly, and the correlation holds until it does not, usually for the people who are least like the population the model learned from.
The second is worse, and O'Neil is emphatic about it. A proxy is far easier to manipulate than the reality it represents. Once a measure becomes consequential, effort flows toward the measure rather than toward the thing it was supposed to indicate.
She traces this through a university ranking system that began in the nineteen eighties. As the ranking grew in importance, institutions optimized for the ranking, and some falsified the inputs outright. The measure had become the goal, and the goal had quietly stopped being education.
Ask this about any scored or automated system in your own operation: what is it actually measuring, what is that standing in for, and what would somebody do differently if they wanted the number to look good without the underlying thing improving?
If that question has an easy answer, you have found a proxy that will eventually be gamed. Not necessarily dishonestly. People respond to what is measured, and that is not a character flaw.
Trust Is Now a Leadership Skill
There is a second book worth having on this, and it approaches the same ground from the leader's side.
In Leadership With AI and Trust, Alex Zarifis argues that leaders introducing these systems need to be capable in two directions at once. They need the people skills they have always needed, and they need enough understanding of the technology to exercise judgment about it, because those two things can no longer be held by different people in the organization.
His point about trust is the one I would underline. Building stakeholders' confidence in a company's AI is a distinct leadership task, and it is not accomplished by reassurance. It is accomplished by the tools demonstrably performing reliably, fairly, and transparently, and by someone being visibly responsible for that.
Zarifis also makes a practical observation that runs against most adoption advice. A measured, phased approach, or even a deliberate pause, is sometimes the wisest option. Leaders who push adoption all at once underestimate how much time and effort it takes people to learn and genuinely use a new system, and they mistake installation for adoption.
That is the same finding as the parallel process, arriving from a different direction.
What Accountability Actually Means Here
Now the question in the title, answered plainly.
You can delegate a task. You cannot delegate accountability for the outcome of that task. This is not a novel principle, and it is not specific to artificial intelligence. It is how responsibility has always worked in any organization with a license, a duty of care, or a customer.
What is new is how easy these systems make it to lose track of where the responsibility sits, because the delegation happens without a conversation. Nobody decided to hand a decision over. A tool was adopted for one purpose; its output became convenient; the convenience became routine; and eighteen months later, something is being decided by software that nobody chose to have decided that way.
So the governance question is not philosophical. It is a set of four things that either have answers or do not.
Who owns this tool? Not who bought it. Who would notice tomorrow if it started behaving differently, and whose job includes noticing?
What does it do when it is uncertain? Systems with genuine judgment about their own limits decline, escalate, or flag. Systems without it answer everything in the same confident register, and that is observable in an afternoon of testing.
How would we know if it were wrong? This is the hardest one and the most important. If a tool began producing subtly bad output tomorrow, what would surface it? If the honest answer is that somebody would eventually notice something felt off, you do not have a detection mechanism.
Who can decide to stop using it? Name the person and make sure they know. A tool that nobody has the authority to switch off will not be switched off.
Wide Adoption Is Not Assurance
One cautionary case, because it is the clearest example available of every one of those four questions going unanswered at scale.
A sepsis prediction model built into one of the most widely used hospital record systems was deployed at hundreds of hospitals. It was later validated externally by researchers at a health system that had not built it and found it to be a poor predictor that missed a substantial majority of the sepsis cases it was meant to catch while generating a heavy burden of alerts.
Every one of those hospitals had adopted it in good faith. What none of them had was an independent answer to how we would know if this were wrong, because the model arrived inside a system they already trusted, and trust in the vendor had been silently converted into trust in the model.
The lesson is not that the vendor behaved badly. It is that adoption count is a distribution fact and tells you nothing about performance, and that a model can be wrong at scale for years while everybody involved is acting reasonably.
When the Incentives Are the Problem
A final example, from the governance of AI rather than its accuracy, because it illustrates something that pure attention to model performance will never catch.
A government pilot program using artificial intelligence in prior authorization was examined through documents obtained under freedom of information law. What surfaced was not a broken algorithm. It was an incentive structure: vendors paid in connection with the denials they issued, without meaningful penalty when those denials were reversed on appeal, alongside failures to meet response-time standards.
The model may have worked exactly as designed. The problem was what it had been designed to optimize, and who benefited from the answer coming out one way rather than the other.
That is O'Neil's argument arriving in the real world. The question is never only whether the system is accurate. It is what the system was built to maximize, and whose interest that serves.
When you evaluate any automated process in your own practice, including a benign one, ask what it is optimizing for and who gains if it leans one way. Usually the answer is reassuring. Occasionally it is not, and you would rather find out by asking than by discovering.
Frequently Asked Questions
Does using AI change my professional liability?
You should ask your own indemnity provider, because the answer is specific to your cover and jurisdiction, and it is changing. The principle that does not change is that the clinician or owner who acts on a recommendation is responsible for the action. Software does not hold a license.
Is it realistic to interrogate a black box?
Not the model itself, no. But you can interrogate everything around it: what it was trained on, what it was validated against, what it does when uncertain, what its published failure modes are, and what happens when you feed it an edge case. That is a great deal of the practical benefit of transparency without needing access to the internals.
Should small practices worry about algorithmic bias?
The classic large-scale harms O'Neil describes are not what a small practice will produce. What is relevant is the underlying mechanism: any scored or automated process encodes assumptions, and a process that quietly serves some of your patients worse than others is the kind of thing that goes unnoticed precisely because it is automated.
Who should own an AI tool in a small practice?
Someone who uses it, not someone senior who does not. Ownership means noticing, and only a user notices. The owner needs the authority to raise a problem and be heard, which is a culture question rather than an org-chart one.
How much governance is proportionate?
Far less than people fear. For most practice tools this is a named owner, a written line about where the tool may and may not be used, and a scheduled review. An afternoon to set up and a short conversation twice a year.
Final Thoughts
The reason this week exists is that the AI conversation almost always stops at the purchase, and the practice owner is left holding something nobody agreed to be responsible for.
O'Neil's remedy is the right one, and it is unglamorous. Not prohibition, not suspicion. Regular, scheduled examination of whether the thing is still doing what you believe it is doing.
That is the whole of governance. Someone owns it, someone checks it, and someone can stop it.
A tool nobody owns is a tool nobody checks.
Sources & Further Reading
O'Neil, Cathy. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown, 2016. The proxy argument, the account of models encoding their designers' judgments, and the call for regular algorithmic auditing are hers.
Zarifis, Alex. Leadership With AI and Trust: Adapting Popular Leadership Styles for AI. De Gruyter, 2025. On trust-building as a distinct leadership task, the case for ambidextrous leadership, and the argument for phased rather than wholesale adoption.
Wong, Andrew, and colleagues. External validation of a widely deployed sepsis prediction model, published in JAMA Internal Medicine.