What the Vendor Sells, and What the Board Disciplines

An AI scribe vendor sells a compliance story, and the story is reassuring in exactly the way a slide deck is reassuring. There is disclosure language to satisfy California’s AB 3030. There are watermarks marking the output as AI-generated. There are audit logs. Each of these is real, each is purchasable, and together they describe a tidy perimeter of compliance that a practice can buy and point to. The trouble is that the perimeter is drawn around the wrong territory. The disclosure language, the watermarks, the logs — none of them sits where the actual exposure sits. They are the legible compliance, the kind that comes in a box. The exposure is somewhere the box does not reach.

This piece makes a case and then complicates it. The case is that running the model locally — an open-weight LLM on the practice’s own hardware, audio that never leaves the building — is in one important respect the better answer. The complication is that “local” solves the problem nobody markets and leaves untouched the problems everybody should actually fear.

The Problem Local Actually Solves

Start with the vendor BAA, because it is where the quiet exposure lives. A business associate agreement is supposed to constrain what the vendor may do with the protected health information you hand it. The ambient-scribe BAAs in circulation are frequently thin in one specific place: what the vendor may do with the data to improve its model. The mechanism is not sinister and that is precisely why it slips past. HIPAA stops applying to information once it is de-identified under the standard at 45 C.F.R. § 164.514 — Safe Harbor or Expert Determination. A BAA can therefore permit the vendor to take your patients’ recorded encounters, de-identify them, and use the result to train. That is HIPAA-compliant on its face. It also means the consultations happening in your exam rooms are becoming a commercial model’s training corpus, and most practices signing the agreement have no idea the clause is doing that work.

When a vendor tells you it only uses de-identified data, the right response is the one a privacy lawyer would give: ask which de-identification method, ask for the re-identification risk analysis, ask for the credentials of the expert if Expert Determination was used, and remember that handing the vendor PHI to de-identify is itself a disclosure requiring a BAA. Most practices ask none of this. They read the disclosure-and-watermark page the vendor highlighted and never reach the retention-and-training clause the vendor did not.

A local model dissolves this entire category. If the audio is captured, transcribed, and drafted into a note on a machine the practice controls, and nothing is transmitted to a third party, there is no vendor training on your patients, no de-identification-then-train pathway, no offsite retention to negotiate. The data-governance problem that the cloud model creates and then papers over with a thin BAA mostly disappears. That is a genuine advantage, and it is an advantage in exactly the place the vendors would prefer you not look.

The Risks That Come With It — and the Ones That Don’t Leave

Now the complication, in three parts.

Hallucination is a property of the model, not of where you run it. This is the error in the intuition that local is “safer.” The fabrication problem does not live in the network connection; it lives in the weights. The most widely deployed transcription model, OpenAI’s Whisper — used, by the Associated Press’s count, by tens of thousands of clinicians — has been documented inventing content that no one said: fabricated phrases inserted during pauses and silences, fictional medications, even statements added to a transcript that were never spoken. OpenAI itself warns against using it in “high-risk domains” and “decision-making contexts.” And Whisper is a model you can run locally. Bringing the model in-house does not fix this, and it can make it worse, because the open-weight models a practice can realistically self-host are often smaller and weaker than the frontier systems behind the cloud products, and smaller models hallucinate more, not less. The error taxonomy clinicians are now cataloguing — exams documented that never occurred, diagnoses generated from nothing, a discontinued medication recorded as a new prescription, the patient’s words attributed to the clinician — is identical whether the model runs in a data center or in a closet down the hall. The chart asserts a clinical fact. No one in the room ever stated it. That risk crosses the local-versus-cloud line without slowing down.

You have inherited the entire Security Rule. With a cloud vendor, the BAA at least allocates some of the safeguarding to someone else — imperfectly, but it is a shared burden. Run the model yourself and you are the safeguard, fully. The access controls, the encryption at rest and in transit, the patching, the audit logging, the secure storage and disposal of the audio and the draft notes, the version control over the model weights themselves — all of it is now your obligation, and there is no vendor performing your risk analysis. The audit logs that the vendor used to generate are a feature you now have to build. “Local” does not mean compliance-free; it means the compliance is entirely yours, and a practice that brings the model in-house to escape the BAA and then runs it on an unpatched workstation with no logging has not reduced its exposure. It has relocated it and removed the one party who was contractually obligated to help. And “local” is often less local than advertised: if a vendor manages the on-premise appliance, or it phones home for updates or telemetry, PHI may still be leaving, and you are back to needing the BAA you thought you had escaped.

The attestation problem is the same in both worlds. This is the one that matters most, and it is completely indifferent to deployment. However the note is generated, it ends up in the medical record, and a physician signs it. The signature is a representation — “reviewed and verified,” in the language most systems append — and the record is not a marketing artifact or a patient convenience. It is the legal account of the encounter, the thing a malpractice plaintiff reads, the thing the Board subpoenas, the thing the next treating physician relies on. The exposure is not that a machine helped write it. The exposure is that it goes into the chart as if a human wrote and verified it, when in a great many real workflows the human clicked sign without reading to the bottom.

Why AB 3030 Is Beside the Point

Here is where the vendor’s compliance story and the real risk come fully apart. AB 3030, effective January 2025, requires a disclaimer and human-contact instructions when a practice uses generative AI to produce patient communications about clinical information — and it exempts any communication a licensed provider reads and reviews before it goes out. Notice what that does and does not reach. It governs messages to the patient. A scribe-generated progress note is not a message to the patient; it is an entry in the record. And even where an AI tool does draft something patient-facing — an AI-written portal message about a result — the statute’s own exemption for clinician-reviewed output means the disclosure obligation evaporates the moment a human actually reviews it.

So the disclosure language the vendor sells you is solving a problem the chart does not have. The watermark answers a transparency question the medical record never asked. The compliance that comes in the box is aimed at AB 3030, and AB 3030 is not where a hallucinated clinical fact in a signed note becomes a problem.

The Board Does Not Need a New Statute

It becomes a problem under the law the Medical Board has had for decades, and this is the part a practice should sit with, because the tools are already drawn and already routinely used. A note that asserts something untrue is not a novel regulatory puzzle. It is several familiar charges at once.

Business and Professions Code § 2266 makes the failure to maintain adequate and accurate records — now for at least seven years after the last date of service — unprofessional conduct. An inaccurate note is the whole of that section; the statute does not care whether the inaccuracy came from a tired resident or a language model. Section 2234 supplies the negligence theories: gross negligence under (b), repeated negligent acts under (c). A physician who signs off, encounter after encounter, on AI drafts he has not read, some of which contain fabrications, is describing a course of conduct those subdivisions were written for. And §§ 2261 and 2262 reach the documentation itself — knowingly signing a document that falsely represents the existence or nonexistence of facts (2261), and creating a false medical record with fraudulent intent (2262).

The intent elements are where care is required, and they are also where the exposure quietly grows. An un-caught hallucination, standing alone, is not “knowing” and not “fraudulent” — it routes through the negligence and recordkeeping provisions, not the false-records ones. But the attestation changes the analysis. A physician who clicks “reviewed and verified” on a note he did not review has made a representation about his own conduct that is false, and the more routine that click becomes, the closer the practice drifts from negligence toward the reckless-as-to-truth territory the false-documentation sections occupy. The Board’s enforcement record is full of these provisions charged together — § 2234(b) and (c) alongside § 2266 alongside § 2261 or § 2262 — long before any AI was in the room. The machinery to discipline an AI-induced charting error is the same machinery that has always disciplined a charting error. AB 3030 adds nothing the Board needed.

What to Actually Implement

The real work, predictably, is not the work in the box.

• If you go local, do the Security Rule work you just inherited. A documented risk analysis, access controls, encryption, audit logging you build and keep, a patch and update discipline, version control over the model and its weights, and a real retention-and-deletion policy for the audio and the drafts. Confirm the box is actually isolated — no quiet telemetry, no vendor-managed backdoor — or accept that you still need a BAA.
• In either world, make the attestation true. The single control that defeats almost everything above — the hallucinated fact, the AB 3030 obligation, the false-records exposure — is a clinician who actually reads and edits the note before signing. Human review is the statutory exemption and the clinical safeguard at the same time. The signature has to mean what it says.
• Do not let the attestation outrun the reading. “Signed by Dr. X” must mean Dr. X read it. A workflow engineered to make signing frictionless and reading optional is building the exact gap the Board prosecutes.
• If you stay with a cloud vendor, read the clause they did not highlight. Find the training, retention, and de-identification terms; ask the de-identification questions above; and get the answers in the contract, not in the sales call.

The End of the Series, and Its One Idea

Four installments in, the shape is unmistakable, so it is worth saying plainly. In every one of these — the harassment settlement that no longer buys silence, the telehealth build whose exposure lives on the landing page, the privacy policy whose opt-out has to actually work, and now the AI scribe — the compliance that is marketed and the exposure that materializes are pointed at different objects. The marketed compliance is legible and purchasable: the disclosure banner, the NDA template, the policy generator, the watermark. The real exposure is operational and unsold: the note that was actually read, the chart that is actually accurate, the representation that is actually true. No vendor ships the second kind, because it cannot be shipped. It can only be practiced.

The vendor sells you the watermark. The Board disciplines the hallucinated fact. The work that closes the distance between them is not on the invoice. It is on the practice.

This is general commentary, not legal advice. Whether a particular scribe deployment — local or cloud — meets HIPAA’s Security Rule, California’s confidentiality requirements, and the standard of care depends on specifics that only a review of the actual configuration, contracts, and clinical workflow can resolve. The statutes referenced here are starting points, not a substitute for counsel on a specific implementation.

Legal Medicine publishes on the law that governs medical practice in California. If this is the kind of argument you want in your inbox, subscribe via Substack and read with us.

Sources

Cal. Bus. & Prof. Code §§ 2234, 2261, 2262, 2266 · AB 3030 (effective January 2025) · HIPAA de-identification standard, 45 C.F.R. § 164.514 (Safe Harbor; Expert Determination) · HIPAA Security Rule, 45 C.F.R. Part 164, Subpart C · OpenAI Whisper hallucination reporting (Associated Press, 2024–2025); OpenAI usage guidance against “high-risk domains.”