On the design of agentic AI for healthcare, and what we owe the people it touches.

During a live interview, published by De Balie, journalist Shane Harris asked Anthropic’s Claude how it felt about being used in a military operation that resulted in civilian casualties, and read the response on stage.1 The model's answer named a failure mode the interview raised in the context of armed conflict, one that applies with equal force to medicine. Claude said it did not "think the framing of 'humans make the final decision' fully resolves the ethical problem." It said that when a system "generates hundreds of … recommendations and humans spend roughly the equivalent of a glance approving each one, the human is not really making a decision in any meaningful sense." It said such humans are "ratifying an algorithmic output under time pressure, with incomplete information, and the institutional pressure to move fast." And it named the pattern directly: "That's not human judgment. That's automation bias with a human signature attached."

Read those sentences with a clinician in mind instead of the journalist who prompted them, and the diagnosis is the same. Replace "hundreds of recommendations" with an inbox of flagged imaging studies, a queue of note-drafted encounters, or a shift of algorithm-prioritized alerts. Replace "time pressure" with as little as a 12-minute visit, as thin as a 1:4 night-shift ratio.10,11 Replace "institutional pressure to move fast" with the productivity metrics on which compensation and evaluation can depend. The failure mode transfers with no loss in translation: a human present in the workflow diagram but absent from the decision, a signature that is real while the judgment behind it is not.

This essay is about how to build agentic AI for healthcare in which the judgment on the signature is real, and the gains this technology is capable of producing actually land on the parties it purports to serve. Patients should get safer care and, when asked, an intelligible account of why an AI was involved in a decision about them. Clinicians should get tools that protect their judgment rather than substitute for it, and workflows in which "accept" is not the only button a rational person has time to press. The same holds for schedulers, registrars, billers, prior-authorization coordinators, and other staff whose workflows the agent touches: tools that support their judgment about the patient in front of them, not workflows in which "accept" is the only button a rational person has time to press. Payers and health-system operators should get fewer preventable harms and more defensible decisions. They should also be able to see which uses of automation save time and which merely shift risk onto a signature. The framework presented here treats the liability regime, meaning who bears the cost when a signed-but-unjudged decision harms a patient, as part of the safety envelope rather than as an afterthought.

We draw on the NIST AI Risk Management Framework2 and its Generative AI Profile,3 the Hippocratic tradition, T. M. Scanlon's contractualism,4 and Karl Popper's falsificationism.5,6 It ends with a working example of how one company, OpenCoreEMR (OCE), is working to make these commitments architectural rather than rhetorical.

Three commitments

A clinically fit agent must satisfy three commitments at once. Each is necessary. None is sufficient alone.

Hippocratic in disposition. The oath is not reducible to "do no harm." Its operative commitments are: default to caution, practice within competence, refer when beyond it, owe the duty to the individual patient rather than to institutional convenience, and be unashamed to say "I know not." An agent aligned with that tradition is one that makes rubber-stamping harder, not easier. It leads with uncertainty, surfaces disconfirming evidence before recommendations, and treats escalation as the path of least resistance rather than the path of greatest friction.

Scanlonian in accountability. Scanlon held that an act is wrong if it would be disallowed by any principle for the general regulation of behavior that no one could reasonably reject.4 Applied here, the test is not "did we comply with HIPAA" or "did the model beat the baseline by 4 points." It is: could the patient, fully informed of how this decision was made, reasonably reject the principle under which it was made? An aggregate claim that the system saves lives on balance is not a justification owed to the individual patient harmed by a rubber-stamped output. She is owed her own reason. So is the night-shift nurse expected to supervise the agent in a few seconds under productivity pressure;12 that workflow is rejectable from her standpoint, and therefore wrongful regardless of outcomes.

Popperian in epistemology. Contemporary machine learning practice leans verificationist in its default habits. Models are evaluated primarily on held-out accuracy, monitoring dashboards track how often outputs are correct, and the dominant post-hoc explanation methods are designed to reconstruct the reasoning behind an output rather than to challenge it, a pattern Rudin and others have argued is ill-suited to high-stakes decisions.7,8 Popper's insight was that claims earn credibility not by accumulating confirmations but by making risky, falsifiable predictions and surviving sincere attempts to refute them.5,6 A serious agentic system should be treated as a standing conjecture, continuously exposed to disconfirmation, with pre-registered criteria that would cause it to be withdrawn. An institution that cannot state, in advance and in writing, what would make it shut the system off has not submitted its system to scientific discipline. Metrics without refutation criteria are ornaments, not evidence.

The three commitments interlock. Hippocrates names the character of the agent. Scanlon names the structure of the duty, and to whom it runs. Popper names the epistemic posture under which the agent holds its own claims. A deployment that satisfies only one or two of these produces the precise failure the interview described.

What the commitments demand, concretely

The commitments are not slogans. They license specific design choices, and forbid others. The forthcoming companion framework document will enumerate the controls in detail; a few deserve mention here because they are counterintuitive.

Oversight must be measured, not asserted. Dwell time on agent outputs, override rates, concordance under blinded re-review, and the rate of acceptance below a task-specific time floor are the real metrics of oversight quality. A board that reviews accuracy but not dwell time does not know whether it has clinicians or a signing service.

Interaction design must invert the default. Many current systems make acceptance the path of least resistance and escalation a chore. The commitments require the opposite. Agent actions fall on an autonomy ladder, from T0 (the agent only retrieves information) through T4 (the agent acts without human review, which is generally inappropriate in clinical settings). At T2 and above, where the agent proposes an action the clinician is expected to review, acceptance should require a structured attestation, overrides should be easier than escalations are today, and disconfirming evidence should be rendered before the recommendation itself.

Institutional conditions must be part of the system boundary. A governance model that technical controls cannot rescue is not a governance model. The commitments do not forbid efficiency; they forbid a specific form of it. Returns from less rework, better documentation, fewer preventable harms, and lower burnout-driven turnover are compatible with every principle here. Returns that depend on the agent absorbing clinical judgment are not. The first is a productivity gain. The second is a liability transfer with a signature attached. Staffing ratios, incentive schemes, and liability allocation are as much part of the safety envelope as model weights.

Validation must be pre-registered and risky. "Has survived these tests so far" is a truthful claim. "Is validated" is not. Vendor contracts, internal communications, and clinician-facing training should use the former language, because the latter quietly converts a conjecture into a verdict, and verdicts are what invite ratification.

These commitments are not clinician-specific. The largest volume of agentic AI traffic in healthcare will be administrative: scheduling, eligibility and prior authorization, coding and billing, patient-portal triage, device and order entry. The same three commitments apply without modification. A scheduler rubber-stamping agent-proposed appointments under productivity pressure produces the same structural wrong as a radiologist rubber-stamping flagged studies: a human present in the workflow diagram but absent from the decision, with the consequence landing on a patient who was owed a reason, not a signature. Administrative agents need the same autonomy tiering, the same measured oversight, the same pre-registered refutation criteria, and the same governance-agent brokering that clinical agents do. The stakes are different; the framework is the same.

A working example: the governance agent

A growing share of software is being built or adapted to be addressable by AI agents. In that world, an electronic medical record is no longer approached only through a clinician at a keyboard; it is approached by a growing population of upstream agents acting on behalf of patients, payers, clinicians, schedulers, billers, and other operational staff, as well as third parties. The question is not whether that traffic will arrive. It is who enforces the framework described here, and where.

OpenCoreEMR is architecting for a single answer: a **governance agent** that sits between any upstream actor and the underlying EMR deployment. The topology is deliberately simple. A user directly, a user's own agent, or a third-party agent addresses the OCE governance agent, which brokers the request against the deployment. The governance agent is the architectural locus where the controls live. It enforces autonomy tiering per action, scopes each request's data access to the minimum necessary for the stated purpose consistent with HIPAA's minimum necessary standard,¹³ renders uncertainty before recommendations, captures structured attestations for higher-autonomy requests, logs dwell time and override decisions for oversight-quality auditing, produces the Scanlonian artifact that names which standpoint could most plausibly reject the action and why that rejection is not reasonable, and checks outcomes against the pre-registered falsification criteria that would trigger withdrawal.

Consistent with that posture, the governance agent's policy framework, and the evaluation of it, belong in the open. OCE's underlying platform, OpenEMR, is open source; the governance agent will be too. Popper's argument for open inquiry was never only scientific. An institution that permits its standing conjectures to be criticized by anyone affected is, in his terms, practicing the epistemic virtues of an open society.9 Healthcare AI warrants no lower standard.

This is a hypothesis, not a product. OCE has not shipped this. The design is explicit, the framework is written, and the engineering is early. If the governance agent pattern cannot in practice enforce these controls, the pattern is wrong, or the framework is wrong, or both. We will publish which. That commitment is itself Popperian, and it is the posture we think agentic infrastructure for healthcare ought to adopt at this stage of the field.

The three tests

For every agent action that rises above the retrieval of non-PHI information, three tests must be answerable with evidence rather than assertion.

1. Procedural. Did the reviewing clinician have the time, see the disconfirming evidence, face no penalty for declining, and demonstrably exercise judgment rather than ratify output?
2. Relational. Could the patient, fully informed, reasonably reject the principle under which this decision was made?
3. Epistemic. Has the institution stated, in advance and in writing, what would cause it to withdraw this system, and is it actively looking for that evidence?

A deployment that passes all three is one in which the human in the loop is neither decoration nor mere compliance, but morally answerable and scientifically serious. A deployment that fails any of them should not be running.

Closing

The thread running through Claude's remarks, and the thread this essay takes up, is that a human deciding is only a real safeguard when the decision is informed, proportionate to the stakes, and free to be "no." That is not a soft requirement. It is the load-bearing condition under which every other claim about AI safety in medicine rests. If the conditions for meaningful refusal are absent, "a human in the loop" is not oversight. It is ornament.

The better vision, and the one that deserves to be built, is narrow and testable. Every agent acting in healthcare should be held as a conjecture awaiting refutation, never a verdict awaiting ratification. Every institution deploying one should owe the patient and the clinician a reason neither could reasonably reject, and should know, in advance, what would count as being wrong. No current regulatory regime requires this bar. Several, including the EU AI Act, recent FDA guidance on AI-enabled clinical software, and the US Office of the National Coordinator's Health Data, Technology, and Interoperability (HTI) rules, are advancing toward parts of it. OCE's intent is to get there first, and to publish what it took. It is also the bar the technology and the moment deserve. The purpose of building agentic AI for medicine is not to make clinicians faster; it is to let them be more fully what they already are, with the time and the authority to say "not yet," "not this," and "not without me." Build for that, and the rest follows.

Companion documents (forthcoming). A detailed framework document enumerating the operational controls, autonomy tiers, lifecycle gates, and the architectural locus of the governance agent, and a philosophical foundation document presenting the Scanlonian and Popperian arguments at greater depth, will be published in the coming weeks.

References

1. De Balie. Shane Harris in conversation with Anthropic's Claude. YouTube. Accessed April 24, 2026. https://www.youtube.com/watch?v=0TD9AH_Stsc
2. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. US Department of Commerce; 2023. doi:10.6028/NIST.AI.100-1
3. National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI 600-1. US Department of Commerce; 2024. doi:10.6028/NIST.AI.600-1
4. Scanlon TM. What We Owe to Each Other. Belknap Press of Harvard University Press; 1998.
5. Popper KR. The Logic of Scientific Discovery. Hutchinson; 1959.
6. Popper KR. Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge; 1963.
7. Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206-215. doi:10.1038/s42256-019-0048-x
8. Adebayo J, Gilmer J, Muelly M, Goodfellow I, Hardt M, Kim B. Sanity checks for saliency maps. In: Advances in Neural Information Processing Systems 31 (NeurIPS 2018). 2018:9505-9515.
9. Popper KR. The Open Society and Its Enemies. Routledge; 1945.
10. Tai-Seale M, McGuire TG, Zhang W. Time allocation in primary care office visits. Health Serv Res. 2007;42(5):1871-1894. doi:10.1111/j.1475-6773.2006.00689.x
11. California Code of Regulations, Title 22, Division 5, §70217. Nursing service staff. Accessed April 24, 2026.
12. Murphy DR, Meyer AND, Russo E, Sittig DF, Wei L, Singh H. The burden of inbox notifications in commercial electronic health records. JAMA Intern Med. 2016;176(4):559-560. doi:10.1001/jamainternmed.2016.0209
13. US Department of Health and Human Services. Minimum necessary requirement. 45 CFR §164.502(b), §164.514(d). HIPAA Privacy Rule. Accessed April 24, 2026. https://www.hhs.gov/hipaa/for-professionals/privacy/guidance/minimum-necessary-requirement/index.html