AI Governance in Healthcare

AI governance in healthcare is the set of rules, reviews and controls that decide which AI tools may touch patient information and clinical work, under what conditions, and who is accountable for the result. It applies to a solo practice whose staff paste visit notes into a chatbot, to a multi-site group buying an ambient documentation tool, and to a health-tech vendor adding AI features to a product. The practical core: inventory every AI touchpoint, apply HIPAA to each one, tier the clinical risk, review the vendor, keep a human responsible for every consequential output, and monitor what the tools actually do. This is general information, not legal advice; verify the current official text of any rule mentioned here.

Where AI is showing up in healthcare operations

Most healthcare organizations already use AI, whether or not anyone approved it. A governance program starts by finding it. The usual places:

• Documentation: ambient AI scribes that listen to the visit and draft the note; chart summarization; drafting of referral letters and after-visit summaries.
• Patient communication: AI-drafted replies to portal messages, scheduling and intake chatbots, automated reminders and call-center assistants, translation.
• Revenue cycle: billing code suggestions from the note, claim scrubbing, denial appeal letter drafts, payer authorization paperwork.
• Clinical support: risk scores and early-warning models, image analysis and triage, differential suggestions, drug interaction and dosing checks, remote monitoring alerts.
• Back office: policy drafting, HR, marketing copy, spreadsheet analysis, meeting transcription, email writing.
• Features switched on inside existing vendors: the EHR, the phone system, the billing clearinghouse and the fax service may each have added AI features that process PHI by default.

The last category is the one most inventories miss. Responsible AI in healthcare includes the AI you did not buy on purpose.

HIPAA and AI: the practical rules

HIPAA was written before large language models, but its rules apply to them without modification. Six points cover most situations.

1. An AI vendor that receives PHI is a business associate. If a tool creates, receives, maintains or transmits protected health information on your behalf, you need a signed business associate agreement (BAA) before PHI goes in. A consumer AI account without a BAA is not an option for PHI, and pasting PHI into one is an impermissible disclosure that may also be a reportable breach.
2. Minimum necessary applies to prompts. Staff should send the least PHI a task requires. A request to rewrite a patient letter in plain language rarely needs the date of birth, the full record or the insurance ID.
3. De-identified data is outside HIPAA, but only if it is actually de-identified. The Privacy Rule recognizes two methods: removal of the listed identifiers under the safe harbor method, or a documented expert determination. Stripping names is not enough. Free-text notes are especially hard to de-identify.
4. Training on your PHI needs a legal basis. A business associate may only use PHI as the BAA permits. Check whether the vendor’s terms allow model training or “service improvement” on your data, and whether your BAA permits it. Many organizations require that training be switched off in writing.
5. The Security Rule risk analysis must include AI systems. Every AI tool that touches electronic PHI belongs in the risk analysis, with access controls, audit logging, encryption and workforce training applied to it like any other system. HHS publishes Security Rule guidance that applies regardless of the technology.
6. AI-generated content in the record is still the record. An ambient note signed by a clinician is part of the designated record set. Patients have the right to access it and to request amendments, and the organization is accountable for its accuracy.

Other rules sit alongside HIPAA. Several states require disclosure when generative AI is used in certain patient communications, and recording consent laws affect ambient scribes. Federal nondiscrimination rules address patient care decision support tools, certified health IT carries transparency obligations for predictive decision support, and some clinical AI is regulated as a medical device. Substance use disorder records under 42 CFR Part 2 carry stricter rules again, so verify each against the current official text with your counsel.

Clinical vs administrative risk tiers

Not every AI use carries the same risk, and a governance program that treats a marketing email generator like a sepsis model will be ignored. Tiering is how you spend attention where harm is possible. The model below is a starting point for organizations of every size; adapt the examples to your own operations.

Tier	Description	Examples	Minimum requirements
1. Administrative, no PHI	Internal work with no patient data and no clinical effect	Policy drafts, job descriptions, marketing copy, spreadsheet formulas	Approved tool list; acceptable use policy; no PHI rule enforced by training
2. Administrative with PHI	Operational tasks that involve patient data but no clinical judgment	Scheduling bots, portal message drafts, billing code suggestions, payer paperwork, transcription	BAA; training switched off; access control and logging; human review before anything reaches a patient or a payer
3. Clinical support, clinician in the loop	Output informs clinical work and a licensed professional reviews it before acting	Ambient notes, chart summaries, differential suggestions, interaction checks, risk scores shown to clinicians	Everything in tier 2 plus pre-deployment validation on your own patients and workflows, documented clinician sign-off, accuracy audits, bias review, patient disclosure where required
4. Autonomous or patient-facing clinical	Output reaches the patient or changes care without a clinician reviewing each instance	Symptom triage bots giving advice, automated result interpretation sent to patients, closed-loop alerts that change orders	Formal governance approval; regulatory status confirmed; prospective validation; monitoring with defined stop criteria; often prohibited by policy until proven

Two practical rules follow. First, the tier is set by the worst realistic use, not the vendor’s description: a “documentation assistant” that also suggests diagnoses is tier 3. Second, a tool can move tiers when its configuration changes, which is why the inventory records the configuration, not only the product name.

Vendor due diligence and BAAs

Healthcare AI solutions are mostly bought, not built, so vendor review is where governance has the most effect. The questions below fit on one page and apply to a large platform vendor and to a two-person startup alike. Scale the depth of evidence to the tier.

• Contract: Will you sign our BAA, or provide one that covers all services we use? Does it permit or prohibit training on our data? What are the retention and deletion terms for prompts, outputs and audio?
• Data flow: Which sub-processors see PHI, including the underlying model provider? Where is data hosted? Is PHI used for any purpose beyond our service?
• Security: What independent attestations exist (for example a SOC 2 report or a HITRUST assessment)? How are access, encryption, logging and incident notification handled? How do you integrate with our EHR: supported API, or screen scraping and shared logins?
• Performance: What validation evidence exists, on which populations and specialties? How is accuracy measured after deployment? How will we be told when the model changes?
• Clinical and regulatory: Is any part of the product regulated as a medical device? What are the intended use and the explicit limitations? How does the tool show its sources?
• Exit: How do we get our data back, and how is it destroyed, when we leave?

Keep the answers with the signed BAA in the inventory record. A vendor that cannot answer the data flow question should not receive PHI, whatever the demo looked like. Our HIPAA compliance checklist covers the broader vendor management process this fits into.

Controls and monitoring

The controls that make AI governance in healthcare real are mostly ordinary, and that is the point. They should feel like the rest of your compliance program, not a separate discipline.

Before deployment

• An AI inventory with owner, tier, data classes, configuration and BAA status for each tool, including features inside existing systems.
• A short AI acceptable use policy that names approved tools by data class and states the no-PHI rule for everything else.
• Single sign-on and access control for every approved tool; blocking or monitoring of unapproved tools on managed devices where feasible.
• A documented risk assessment per tier 3 and tier 4 tool, including a bias review for anything that scores or ranks patients.
• Patient-facing disclosure and consent wording, checked against state requirements, for recording and for AI-generated communications.

After deployment

• Human review rules that are specific: who signs the note, who reads the message draft, what can never be sent automatically.
• Sample audits of outputs on a schedule, with error rates tracked by tool and by workflow.
• A reporting path for staff and clinicians to flag wrong or harmful outputs, feeding the existing incident process.
• Vendor change monitoring: model updates, new sub-processors, changed terms.
• Annual reassessment of each tool’s tier and a refresh of the Security Rule risk analysis to include it.

A small practice can run this with one policy, one spreadsheet inventory and a quarterly review on the practice manager’s calendar. A health system needs a committee, a workflow tool and dedicated time from clinical informatics, compliance and security. The data side, including classification and retention of prompts and transcripts, is covered in our guide to data governance in healthcare, and the general framework is in what AI governance is.

Where PS3G fits

PS3G builds and runs its own HIPAA-compliant EHR and telehealth platform, DocVilla, so the vendor questions above are ones we answer for our own customers as well as ask on behalf of clients. Our AI governance services set up the inventory, policy, risk tiers, vendor reviews and monitoring for practices, groups and health-tech companies, aligned with NIST AI RMF and ISO 42001 and scoped to the size of the organization. Our healthcare software development team builds AI features, integrations and patient-facing tools with those controls designed in from the start, including HL7 and FHIR integration with the systems you already run.

Related reading

• What Is AI Governance? A Practical Guide for Any Size of Organization
• Data Governance in Healthcare: HIPAA, Quality, Interoperability
• HIPAA Compliance Checklist for Small Practices and Health-Tech Vendors
• AI governance services and healthcare software development from PS3G

Frequently asked questions

Is using a consumer AI chatbot with patient information a HIPAA violation?

If the tool receives protected health information and there is no business associate agreement in place, the disclosure is impermissible under the Privacy Rule, and it may require a breach risk assessment and notifications. Enterprise versions of the same tools offered under a BAA, with training switched off, can be used when they are included in your risk analysis and policies. Verify the current rules with counsel.

Do we need a BAA with an AI scribe vendor?

Yes. An ambient scribe records and processes the visit, which is PHI, on your behalf, so the vendor is a business associate. The BAA should cover the vendor and the model providers behind it, state whether audio and transcripts are retained and for how long, and address training. Recording consent requirements vary by state, so check the disclosure your clinicians give before each visit.

Can AI vendors train their models on our patient data?

Only if both the BAA and your organization permit it, and most organizations do not. Training is a use of PHI beyond providing the service unless the agreement says otherwise, and de-identification claims should be checked against the Privacy Rule’s safe harbor or expert determination methods. Ask for the setting to be disabled contractually and in the product, and keep the confirmation with the inventory record.

Does a small practice need AI governance?

Yes, scaled to its size. The obligations under HIPAA do not shrink for a two-provider office, and staff in small practices are often the heaviest users of consumer AI tools because nobody has provided alternatives. One policy page, one inventory spreadsheet, enterprise accounts under a BAA for the tools people actually use, and a quarterly check are enough to start.

Who is accountable when an AI tool contributes to a clinical error?

Accountability stays with the licensed clinician who acted on the output and with the organization that deployed the tool, regardless of what the vendor’s terms say about liability. That is why tier 3 and tier 4 uses require documented validation, explicit human review rules and a way to report errors. Contract terms about indemnity matter too, and they belong with counsel.

Which framework should a healthcare organization use for AI governance?

NIST AI RMF is the common reference in the US and works for organizations of every size; ISO/IEC 42001 adds a certifiable management system for vendors and larger organizations. Neither replaces HIPAA, state law or device regulation. Use the framework to structure the program, and use the laws to set the minimum requirements for each tier.