What AI should not do in a clinical note

Every vendor will tell you what their AI can do. The more useful question when you are evaluating clinical software is what it is not allowed to do, and whether that limit is enforced in the code or in a promise.

Here are four things AI Dietitians deliberately does not do, and why.

1. It does not do the arithmetic

Every calculator runs in ordinary code, not in the model. Energy and protein requirements, fluid, refeeding rates, growth centiles, malnutrition scores, IDDSI levels — those are arithmetic with a citable derivation, and each one shows its working line by line.

This matters because a language model asked for a clinical figure will produce a confident, plausible, frequently wrong number, and it has no mechanism for knowing that it does not know. Food composition lookups go to the Australian Food Composition Database — the current release, 1,588 foods — rather than to the model’s recollection of what is in a slice of bread.

One honest exception. When the AI drafts a meal plan, the per-item macros in that draft are produced by the model. They are validated fail-closed before anything is stored — non-finite, negative or malformed values are rejected outright and nothing is written — but validation is a sanity check, not a source. Those figures are a starting point for you to check, not a database lookup.

We flag this because it is exactly the kind of thing that gets glossed as “built on the Australian Food Composition Database” across a whole product. The database is real and it backs the food reference. The AI meal-plan draft is a draft, including its numbers.

2. It does not send anything

There is no background process that emails a patient, notifies a GP, files an NDIS report or issues a handout. Every outbound communication requires a clinician to review the draft and act.

This is a deliberate constraint rather than a missing feature. The moment software can send clinical correspondence on its own, the question of who authored it becomes genuinely unclear — and you are the one whose name is on it.

3. It does not decide

Every output is a draft, generated from information you entered. The PES statement is a suggestion. The plan is a suggestion. The letter is a suggestion. You read it, change what is wrong, and sign it.

We are specific about this because the alternative framing — “AI-powered clinical decision support” — describes a different product with different regulatory obligations. AI Dietitians drafts documentation. It is not a diagnostic device and should not be used as one.

The practical test: if a clinician could not defend an output as their own reasoning after reading it, the software has overstepped. Everything here is designed to be readable, editable and attributable before it is signed.

4. It does not quietly edit what you wrote

Where the platform checks its own output — for example, scanning a patient-facing letter for numbers that would be unhelpful for someone recovering from an eating disorder — it flags what it found and shows you. It does not silently remove them.

That is the harder implementation and the right one. You sign the letter, so you have to be able to see everything in it. Silent edits mean signing text you have not read, which is worse than the problem being solved.

Two things it does that are worth knowing about

Patient free text is de-identified before it forms part of a prompt. That happens on the Australian server, before anything is sent, and every prompt builder in the platform routes through the same function — enforced by tests that fail the build if one stops. A linter cannot catch free text, so the tests are the control.

AI requests go to providers in the United States. We use OpenAI and Google. Your records are stored in Australia; the AI processing is not, and we would rather say so plainly than let you discover it on a governance form. If that is a blocker for your service, the Essentials tier has no AI features at all — and no code path to a provider, which is a different kind of assurance from a setting we promise not to turn on.

The question to ask any vendor

Not “what can your AI do”, but: which of these limits are enforced in code, and which are policy?

A limit in code survives a busy afternoon, a new feature and a staff change. A limit in a policy document survives until someone is in a hurry.

References

• TGA, Regulation of software as a medical device — the boundary between documentation support and a regulated device.
• OAIC, Australian Privacy Principles, particularly APP 8 (cross-border disclosure).
• FSANZ, Australian Food Composition Database
• Our Privacy Policy — the authoritative statement of which providers process what.

---

AI Dietitians produces drafts to support clinical documentation. It does not provide diagnosis or treatment decisions and is not a medical device. The treating clinician remains responsible for all clinical content and for what is communicated to patients and other providers. See our Privacy Policy for how data is handled.