Before You Buy an AI Note Tool: Accuracy, Context and Control Checklist
By The Kitt Team
Most demos show you a clean note. A clean note is easy to mistake for an accurate one.
The buying decision comes down to three questions: does it get the session right, does it know this client, and do you have genuine control over every record it touches? Nail those three before you sign up for anything.
What an AI note tool checklist is for
An AI note tool listens to or reads a session and drafts the clinical note for you to review. The draft is a starting point; the record is still yours.
An AI note tool checklist is a short set of questions you ask before you buy, grouped by what matters: accuracy, context, and control. It turns a demo impression into a decision you can defend.
What Ahpra expects before you buy
Ahpra’s guidance “Meeting your professional obligations when using Artificial Intelligence in healthcare” (August 2024) says you remain responsible for safe, quality care; must apply human judgement to any AI output; are responsible for checking the accuracy and relevance of records created using generative AI; should obtain informed consent when an AI tool needs the client’s personal data; and should know whether that data is used to train the AI model.
Every item on the checklist below maps to one of those obligations. Whichever tool you pick, the record-keeping obligations stay with you.
The AI note tool checklist
Three lenses, plus one question most buyers forget: what happens after the note.
Accuracy: does it get the session right?
A 2025 randomised trial at UCLA of 238 physicians across about 72,000 encounters found that time savings from ambient AI scribes depended on the tool: one cut time in the note by 9.5 per cent, while another showed no significant reduction (Lukac PJ et al, NEJM AI, 2025). The study involved US physicians, not allied health. Tools in the same category do not perform the same, so test the one you are buying.
- Does it get side, joint, and named tests right?
- Does it keep dosage (sets, reps, load, progression) rather than flattening it to “exercises given”?
- Does it invent findings you never said?
- Does it cope with a messy session (corrections, interruptions)?
- Can you see what it drafted from?
Before you commit, test AI clinical note accuracy with a structured method, not a single clean demo.
Context: does it know this client?
Research shows 40 to 80 per cent of medical information is forgotten immediately, and almost half of what is remembered is remembered incorrectly (Kessels RPC, Journal of the Royal Society of Medicine, 2003). The research was on general medical consultations, not allied health. Memory is not a record, so the tool has to carry the client forward.
- Does the note sit inside the client’s record or float as a standalone document?
- Does it bring up prior findings, progress, and goals before the next session?
- Does the second and third note read like a continuation or like a first appointment?
- Does it hold letters and plans for the same client in the same place?
Control: do you have the final say?
- Does every note come back as a draft you check, edit, and approve before it is final?
- Can you edit freely, and are changes to a finished note tracked?
- Do you know where client data is stored, who can see it, and whether it is used to train the model (ask the vendor directly and get it in writing)?
- How do you get informed consent from the client for the tool?
- Does the finished note land in your practice management system or do you retype it?
Beyond the note: what happens next?
A systematic review of 30 studies notes that non-adherence to home-based physical therapy can reach 70 per cent (Essery et al, Disability and Rehabilitation, 2017). Ask whether the plan reaches the client somewhere they will use it, and whether anything comes back to you before the next visit.
- Does the treatment plan reach the client in a format they can actually use, and do they confirm they’ve seen it?
- Does the tool give you any information back from the client, such as exercise completion or pain, before the next session?
A two-way record closes that loop.
The checklist at a glance
| Lens | Ask this | Red flag |
|---|---|---|
| Accuracy | Does it get side, joint, dosage, and named tests right in a real session? | Notes look clean in demos but collapse on your clinical language |
| Context | Does the second note start from the client’s record, not from scratch? | Every note reads like a first appointment |
| Control | Do you review, edit, and approve before anything is final? | Notes publish automatically or edits are not tracked |
| Beyond the note | Does the plan reach the client and does anything come back? | The tool stops at the note |
How kitt answers the checklist
kitt drafts, you decide. Every note, letter, and plan comes back as a draft for you to check, edit, and approve before it is published. kitt records who edited a finished note and when. kitt does not diagnose or make clinical decisions. kitt documentation supports clinicians’ AHPRA record-keeping obligations.
You dictate the session inside the client’s record, and the note comes back drafted from that session. Client Contextual Memory brings up the client’s prior findings, progress, and goals before each session, so the next note starts from the record, not from memory. If you use Cliniko or Nookal, completed notes can sync back to it; the connection is optional.
The treatment plan goes to the client through kitt Companion, which is included with every subscription, not a paid add-on. The client logs each set with a pain score on a 0 to 10 slider, and that lands back on your side against the plan. The plan goes out, adherence and pain come back.
FAQ
What should be on an AI note tool checklist?
It covers accuracy (does it capture your clinical language correctly), context (does it carry the client’s history into each note), and control (do you review and approve every draft before it is final). A fourth question most buyers skip: does the tool do anything useful after the note, or does it stop there?
Am I still responsible for an AI-drafted clinical note?
Yes. Ahpra’s guidance (August 2024) is clear that the practitioner remains responsible for safe, quality care and must apply human judgement to any AI output. Read the note, check it against what happened in the session, edit where it is wrong, and approve it only when it is accurate.
Do I need client consent to use an AI note tool?
Ahpra’s guidance says you should obtain informed consent when an AI tool needs the client’s personal data. What that looks like in practice depends on your circumstances and the tool you are using, so talk to your professional association if you are unsure.
How is kitt different from an AI scribe?
kitt drafts the note too, but the note is one input. The plan goes out through kitt Companion and adherence and pain come back between sessions.
The short version
- Test the tool on your own sessions before you decide.
- Accuracy, context, and control are the three things that matter. The fourth is what happens after the note.
- You stay accountable for the clinical record regardless of what generated the first draft.
- Ahpra’s guidance is worth reading before you buy, not after.
Try kitt free for a month, no credit card required.