AI Governance Buyer Checklist for Healthcare

AI Governance Buyer Checklist for Healthcare

Healthcare teams do not buy “AI” as one function. They buy a particular system version for a named workflow, user, population, data flow, and decision. A useful buyer checklist must preserve those boundaries from intake through retirement. Otherwise, a polished demonstration can obscure who may rely on an output, where protected data travels, which evidence applies, and how the organization will stop unsafe use.

Use this guide before a request for proposal, during due diligence, before a pilot, and at renewal. It is a governance structure, not a legal opinion, clinical recommendation, product certification, or substitute for local review. Every conclusion still needs an accountable owner and evidence from the proposed configuration.

The result should be a decision packet that another reviewer can reconstruct. Each gate records pass, fail, conditional, or open; the evidence owner; the source and version; the observed date; the limitation; and the expiry or next action. If an essential item is absent, record open — local owner/evidence required. Do not convert silence into a passing score.

For adjacent work, use the AI governance vendor evaluation guide to organize market review, the healthcare AI vendor evaluation checklist for broader due diligence, and AI for healthcare compliance monitoring for that specific workflow. Keep governance review connected to the actual purchase rather than a generic policy exercise.

How to use the eight gates

Run the gates in order because later evidence depends on earlier scope. A security report cannot answer whether the wrong function is being evaluated. A favorable pilot average cannot cure an unapproved data use. A contract cannot prove a configured workflow works.

Treat privacy, security, clinical or operational integrity, lawful authority, auditability, and safe recovery as non-negotiable when they apply. A hard-stop failure cannot be offset by speed, adoption, price, projected savings, or a weighted score. Preferences may be compared only after the mandatory gates pass.

Separate three evidence classes throughout the review:

• A vendor assertion describes what the supplier says the product or control does.
• A buyer-observed result records what happened in the buyer's configured environment and test cohort.
• An authority record documents the accountable person's decision, conditions, scope, and expiry.

The NIST AI Risk Management Framework can help organize risk work, while AI RMF 1.0 describes Govern, Map, Measure, and Manage functions. These voluntary resources do not approve a product or replace sector-specific obligations. NIST also states that AI RMF 1.0 is being revised, so record which version informed the review.

Gate 1: inventory the system and approved scope

Start with the thing being evaluated, not the vendor category. Give every proposed use a unique inventory record. Name the legal entity, care setting, department, workflow, user role, patient or member population, language, geography, source systems, output destination, and intended decision. List exclusions with the same precision.

A single vendor may provide transcription, summarization, prioritization, prediction, search, drafting, coding support, patient messaging, or administrative automation. Record each function separately. A narrow approval for one function must not silently authorize another.

Capture the model provider, product release, model or service version when available, configuration, prompt or policy version, integration release, and deployment pattern. Record the source, version, observed date, reviewer, and known limitation for every artifact. If the vendor cannot identify the system version responsible for an output, historical investigation and change review may be impossible.

Map who can request access, approve use, change configuration, monitor operation, pause processing, investigate incidents, and retire the system. A committee name without a person or role accountable for the local workflow is not enough.

Use NIST's AI RMF Playbook as a source of suggested actions, not as a complete or ordered procurement checklist. Select actions that match the local context and document why they apply.

Evidence request:

• Current product and subservice inventory with version identifiers.
• Intended-use statement, included cohort, explicit exclusions, and prohibited uses.
• System context diagram showing users, inputs, outputs, integrations, and record of truth.
• Named business, technical, privacy, security, safety, and executive owners.
• Approval scope, decision date, expiry date, and conditions for reassessment.

Pass when the exact system and use are bounded, versioned, owned, and recoverable from the record. Mark conditional only when the unresolved item has an owner, due date, interim control, and limit on use.

Stop the gate when the buyer cannot identify the actual function, version, population, decision, data boundary, accountable owner, or person able to pause use. Do not proceed on a promise that inventory details will be supplied after contracting.

Gate 2: classify each function and decision authority

Describe what each function does to information and what a person may do because of its output. Distinguish drafting from finalization, ranking from diagnosis, routing from denial, recommendation from autonomous action, and monitoring from intervention. “Assistive” and “human in the loop” are labels, not operating controls.

For every output, name the decision owner, evidence visible to that person, time available for review, permitted override, recorded rationale, escalation path, and downstream consequence. Then test whether the review can be performed under ordinary workload rather than only in a staged demonstration.

FDA analysis turns on the exact software function, intended use, user, and output. The agency's digital health guidance collection and Clinical Decision Support Software guidance support a function-specific review; the presence of human review does not classify every healthcare AI function as non-device.

If the supplier represents an authorization or listing, capture the exact device, function, indication, version, and source record. FDA's AI-enabled medical device list is informative but is not a comprehensive approval list for every AI function a vendor may sell.

HTI-1 decision support intervention provisions require similarly careful scope. The HTI-1 final rule page and DSI fact sheet concern specified certified health IT and developer-supplied predictive decision support interventions. Source attributes and FAVES-related materials do not certify all healthcare AI or prove local fitness.

Evidence request:

• Function-by-function intended use and prohibited-use statement.
• Regulatory or certified-health-IT rationale tied to the exact version and configuration.
• Decision-authority map from output to human action and system of record.
• Override, escalation, disagreement, and pause records from configured tests.
• Vendor claims cross-referenced to the authoritative source they cite.

Pass when qualified local owners can explain each function, its status, the human decision boundary, and the consequence of error. Preserve the analysis used; a conclusion without its facts cannot be reassessed after change.

Stop the gate when material status or authority is unknown, when a user cannot independently review the output as designed, or when the marketing label is the only classification evidence.

Gate 3: trace data, privacy, and contract duties

Trace the minimum necessary data from collection to deletion. Include prompts, source records, audio or images, embeddings, retrieved context, outputs, feedback, telemetry, audit logs, support tickets, backups, and human review queues. For each store or transfer, record data class, legal entity, role, purpose, location, retention, deletion, access, and reuse.

Begin HIPAA analysis by determining whether the organization and counterparty are regulated entities for the activity and whether PHI or ePHI is involved. The HHS HIPAA Security Rule overview describes safeguards for ePHI, but this checklist cannot determine applicability for a specific arrangement.

When a business associate relationship exists, compare the proposed agreement with HHS sample business associate agreement provisions. A BAA is not a product certification. It allocates duties within a particular relationship and does not establish safe workflow design, accurate output, complete security, or lawful use in every context.

Cloud hosting does not remove the need to identify roles and risk. HHS cloud computing guidance explains that a cloud service provider can be a business associate even when it holds encrypted ePHI without the decryption key, subject to the facts described by HHS.

If the team plans to use data represented as de-identified, document the method, qualified determination where relevant, residual limits, linkage controls, and downstream use. HHS de-identification guidance describes Expert Determination and Safe Harbor; de-identification is not automatic merely because direct identifiers were removed.

The NIST Privacy Framework can help connect data processing with privacy risk management. Record that Privacy Framework 1.1 is presented as an initial public draft at the time of this review; do not treat draft status or framework use as legal compliance.

Evidence request:

• Field-level and system-level data-flow diagram, including logs, support, backups, and subprocessors.
• Role and purpose analysis for each organization and transfer.
• Contract, BAA when applicable, data-use limits, retention, deletion, return, breach, audit, and subprocessor terms.
• Model-training, service-improvement, secondary-use, and human-access settings with configured proof.
• De-identification method and re-identification risk record when that claim is used.

Pass when the data route, parties, purposes, configured controls, contract terms, retention, deletion, and escalation duties agree. Resolve differences between a sales answer, product setting, technical architecture, and contract before use.

Stop the gate when sensitive data movement is unknown, a required role or agreement is unresolved, prohibited reuse cannot be disabled, a deletion claim cannot be tested, or a subprocessor remains undisclosed.

For a deeper sequence of PHI questions, use what to check before using AI with PHI, how to vet healthcare AI vendor security and BAA terms, and the HIPAA-compliant AI tools guide. Product labels still do not replace local fact finding.

Gate 4: match claims to evidence and local validation

Create a register for every material claim: function, accuracy, safety, security, interoperability, accessibility, equity, workflow impact, savings, regulatory status, and service level. Quote the claim, record where it appeared, identify the intended audience, and specify the evidence needed to support it in the proposed use.

Keep vendor assertion, buyer-observed result, and authority record in separate fields. A published external result may justify further investigation. It is not proof of local validation for a different population, prevalence, language, specialty, version, workflow, threshold, or user behavior.

For studies or validation reports, capture the product and model version, study design, comparator, cohort selection, setting, sample size, missingness, exclusions, reference standard, endpoints, uncertainty, subgroup results, conflicts, and limitations. Do not copy a headline measure without its denominator and operating threshold.

NIST's Generative AI Profile identifies generative-AI risk considerations and actions. It can improve the risk register, but it does not verify a supplier's claims or prescribe one universal validation protocol.

Health product claims also require claim-matched support. The FTC's Health Products Compliance Guidance explains advertising substantiation principles. A citation about a technology class does not necessarily substantiate a claim about the purchased product, version, outcome, or audience.

Evidence request:

• Claim inventory linked to the exact product, version, configuration, cohort, and source.
• Full methods and limitation material behind material performance or outcome claims.
• Local validation protocol with prespecified acceptance and stop criteria.
• Test-data lineage, reference standard, reviewer qualifications, and disagreement process.
• Result record with uncertainty, subgroup analysis, failures, exclusions, and corrective action.

Pass when each material claim is supported at the level at which it will be used and the buyer can reproduce the local result. Mark a transferred result as contextual evidence, not a local outcome.

Stop the gate when a high-impact claim has no traceable evidence, the tested version differs materially without justification, limitations are withheld, or the buyer is asked to accept a demo as validation.

Gate 5: verify security and supplier controls

Review security against the actual architecture and threat model. Include identity, least privilege, privileged administration, tenant isolation, encryption and key responsibilities, secrets, interfaces, logging, monitoring, vulnerability management, secure development, incident response, backups, recovery, and supplier dependencies.

Use the NIST Cybersecurity Framework to organize outcomes and ownership rather than to award a binary badge. A framework mapping is not evidence that a control is configured, operating, or effective in the purchased service.

Ask for relevant independent reports and the scope, period, exceptions, subservice organizations, complementary user-entity controls, and remediation status. Verify that named products, regions, and services match the proposed deployment. A current report for a different service is contextual evidence only.

Map third parties beyond the contracting vendor: foundation model providers, hosting, observability, content filters, support tools, annotation services, identity providers, and integration partners. Record which party can access which data and who must notify the buyer of a material event or supplier change.

Evidence request:

• Current architecture, trust boundaries, threat model, and control-responsibility matrix.
• Independent assessment scope, exceptions, remediation evidence, and customer control duties.
• Access, configuration, audit, incident, recovery, and deletion evidence from the proposed tenant.
• Complete supplier and subprocessor chain with locations, purposes, and change notice terms.
• Incident communication, forensic preservation, service restoration, and customer support procedures.

Pass when the buyer can connect each material threat to an owned control and current evidence in scope. Record limitations and residual risk instead of reducing the review to a document count.

Stop the gate when critical access is uncontrolled, material logs are unavailable, tenant boundaries or subprocessors are unknown, unresolved high-impact findings lack treatment, or recovery responsibilities cannot be demonstrated.

Gate 6: test workflow, exceptions, and recovery

Convert the intended use into scenario tests. Include an ordinary case, incomplete input, conflicting information, unsupported request, out-of-scope population, user disagreement, integration failure, latency, unavailable service, duplicate transaction, incorrect identity, correction, rollback, and retrospective investigation.

Use the configured environment, realistic roles, and representative data permitted for the test. Record expected behavior before execution. For every run, preserve the input class, system and configuration version, observed behavior, reviewer, downstream effect, defect, correction, retest, and disposition.

Human review is a control only if the reviewer has time, competence, evidence, authority, and a usable path to disagree. Measure skipped reviews, overrides, late reviews, automation bias indicators, and unresolvable outputs. Training attendance alone does not show the control operates.

The ASTP/ONC SAFER Guides can support structured self-assessment of health IT safety practices. They remain self-assessment resources, not certification or proof that a local control passed.

Evidence request:

• Versioned scenario plan covering expected, boundary, exception, downtime, recovery, and misuse cases.
• Expected and observed results with reviewer, timestamp, defect, correction, and retest.
• User authority and escalation evidence under realistic workload and time pressure.
• Downtime, rollback, reconciliation, correction, notification, and record-preservation rehearsal.
• Support ownership, severity definitions, response paths, and unresolved issue register.

Pass when the team can detect failure, limit harm, maintain or safely suspend the workflow, recover, reconcile records, and explain the historical decision trail.

Stop the gate when a critical scenario fails, a failure is not detectable, affected records cannot be identified or corrected, the vendor alone controls pause or recovery without an adequate commitment, or frontline owners have not accepted the workflow.

Use how to run a healthcare AI pilot to turn these scenarios into a bounded evaluation. A pilot should reduce uncertainty; it should not operate as an unrecorded production rollout.

Gate 7: measure benefit, harm, equity, and incidents

Define the measurement dictionary before observing results. For each metric, specify the question, numerator, denominator, exclusions, source system, cohort, baseline window, comparison window, owner, review cadence, uncertainty, action threshold, and known limitation.

Pair benefits with balancing measures. Faster completion may coexist with correction burden, missed exceptions, alert fatigue, delayed escalation, lower accessibility, unequal error distribution, or increased support work. Track what people must redo, not only what the system completes.

Predefine subgroup analysis that is justified by the intended use and available data. Document small samples, missing attributes, proxy limits, and privacy constraints. An aggregate passing value must not conceal a harmful subgroup result.

WHO's Ethics and governance of artificial intelligence for health describes principles and governance considerations. Its guidance on large multi-modal models expands attention to LMM risks. These resources are not United States law, product authorization, or evidence that a local equity control operates.

Define an incident broadly enough to catch patient, workforce, privacy, security, integrity, access, financial, and operational harm. Connect detection, triage, containment, correction, notification, root cause, learning, and restart authority. Preserve near misses and repeated low-severity patterns.

If a product is outside HIPAA or another sector-specific regime, do not assume there is no breach duty. The FTC Health Breach Notification Rule page supports analysis for entities and events within that rule's scope; it does not mean every healthcare AI product or incident is covered.

Evidence request:

• Prespecified benefit, harm, rework, access, equity, and support measurement dictionary.
• Reproducible baseline and comparison extracts with data-quality checks.
• Cohort and subgroup results with uncertainty, missingness, exceptions, and limitations.
• Incident and near-miss taxonomy with severity, owner, response clock, and restart authority.
• Decision record connecting thresholds to continue, correct, pause, narrow, or stop.

Pass when selected measures are reproducible, interpreted with their limitations, and tied to action. Financial analysis should use observed local inputs and expose assumptions rather than importing a vendor calculator as fact.

Stop the gate when a material outcome cannot be reproduced, a harm or inequity threshold is crossed without an accepted treatment, incidents cannot be traced to affected use, or projected benefit is offered to offset an unresolved hard stop.

For value analysis after the safety and control gates, use the healthcare AI ROI and implementation guide. Keep projected savings separate from observed results and from the authority to continue.

Gate 8: control change, renewal, and exit

Define material change before contracting. Consider changes to intended use, model provider, model or product version, prompt, retrieval source, threshold, policy, integration, data field, population, language, geography, user role, subprocessor, hosting region, security control, and governing requirement.

Classify changes as non-material, review-required, or approval-resetting. Specify who receives notice, minimum notice period, evidence required, regression scope, approval authority, deployment hold, rollback, and emergency process. Vendor labels do not decide local materiality by themselves.

For a device within scope, FDA's PCCP guidance for AI-enabled device software functions may inform discussion of planned modifications. It does not create a generic permission for every vendor update or replace the buyer's local change control.

Review renewal as a new evidence date, not an automatic extension. Reconfirm inventory, ownership, legal and regulatory facts, data flows, suppliers, control evidence, open findings, incidents, performance, subgroup results, service levels, costs, changes, and exit readiness.

Plan exit before dependency grows. Specify export format, provenance, configuration and audit record retention, deletion confirmation, transition support, replacement workflow, unresolved incident access, model-output handling, and who verifies completion. Test at least the critical export and continuity path.

Evidence request:

• Material-change taxonomy and notice obligation covering product, model, data, integration, supplier, and policy changes.
• Regression and reassessment plan linked to the original approval scope and tests.
• Renewal packet with fresh evidence dates, open risks, incidents, performance, cost, and owner decisions.
• Exit, continuity, export, deletion, record-retention, and transition plan with test evidence.
• Final decision showing scope, conditions, expiry, dissent, and the next review trigger.

Pass when the organization can detect change, reassess before affected use, reconstruct historical outputs, make a fresh renewal decision, and leave without losing essential records or continuity.

Stop the gate when material updates can occur without notice, the affected version cannot be identified, approval-resetting changes bypass review, renewal evidence is stale, or safe exit and data disposition cannot be demonstrated.

Minimum decision-record rules

Record one function and one approved context per inventory entry.

Document included and excluded populations with equal precision.

Confirm the source, version, observed date, owner, and limitation for each material artifact.

Check that contract language, technical settings, diagrams, and test behavior agree.

Verify every claimed authorization against the exact function, version, indication, and authoritative record.

Compare external evidence with the local population, setting, workflow, threshold, and configuration.

Test ordinary use, boundary cases, misuse, downtime, correction, rollback, and recovery.

Measure benefit together with harm, rework, support demand, access, and subgroup outcomes.

Record failures, exclusions, dissent, and unresolved questions alongside favorable results.

Confirm who may pause use and who may authorize restart after an incident or material change.

Document the evidence date and trigger that will expire or reset the decision.

Verify export, continuity, record retention, and deletion before renewal reduces exit leverage.

Procurement questions by accountable owner

Send each question to the person who can own the answer. Vendor responses remain assertions until the relevant buyer owner verifies the evidence and records a decision.

Enterprise AI governance and executive sponsor

• Which exact uses, entities, populations, locations, and exclusions would this decision authorize?
• Which risks are non-delegable, and who has authority to pause, narrow, renew, or terminate use?
• What evidence expiry or change event forces the approval to be reconsidered?

Workflow and operational owner

• Where does the system enter the workflow, what record remains authoritative, and what action follows its output?
• Can users review, disagree, correct, escalate, and recover within normal workload and staffing?
• Which exception, downtime, support, and reconciliation tasks become the buyer's responsibility?

Clinical safety, quality, and HIM

• Could an output affect care, documentation integrity, identity, prioritization, access, or another safety-relevant decision?
• Which expected, boundary, misuse, failure, and recovery scenarios must pass before use?
• How will affected records, corrections, incidents, near misses, and restart decisions be preserved?

Privacy, legal, and compliance

• Which entities, roles, purposes, data classes, jurisdictions, agreements, and notices apply to each flow?
• Do contract terms match configured retention, deletion, secondary use, human access, and subprocessor behavior?
• Which unresolved applicability question requires qualified local advice before the workflow can proceed?

Security, IT, and architecture

• Which trust boundaries, identities, interfaces, logs, keys, tenants, suppliers, and customer controls are in scope?
• What current evidence shows critical controls operate in the proposed service, region, and configuration?
• Can the team detect, contain, investigate, recover, reconcile, and preserve evidence without an undocumented dependency?

Data science, model risk, and analytics

• Does each material claim identify the tested version, cohort, comparator, threshold, uncertainty, subgroup, and limitation?
• What local protocol separates a vendor assertion from a reproducible buyer-observed result?
• Which drift, data-quality, harm, equity, or performance threshold triggers correction, pause, or reassessment?

Procurement, vendor management, and finance

• Which promises belong in enforceable scope, service, audit, notice, remedy, transition, and termination terms?
• What buyer labor, integration, validation, monitoring, incident, support, and exit costs sit outside the quoted price?
• Which renewal evidence and exit deliverables are due before leverage or continuity declines?

Patient, accessibility, and equity representatives

• How could the workflow affect notice, choice, access, communication, disability accommodation, language, or burden?
• Which populations may be excluded, underrepresented, disproportionately wrong, or unable to contest an outcome?
• How will feedback, complaints, accessibility failures, disparate outcomes, and corrective decisions reach accountable owners?

Buyer-owned governance artifacts

The buyer should own the decision record even when a vendor supplies source material. Store linked evidence rather than a checkbox with no provenance. Across the packet, use pass, fail, conditional, or open and record no default numeric weight for hard-stop controls. The packet is not a vendor ranking.

AI system inventory and authority matrix

One row represents one function in one approved context. Record product, provider, version, configuration, purpose, population, location, users, inputs, outputs, integrations, decision owner, pause authority, approval scope, observed date, limitation, expiry, and next action.

Do not merge distinct functions under a single “AI platform” row. If the precise owner or version is missing, use open — local owner/evidence required.

Data, role, contract, and subprocessor map

Trace each data class and transfer through buyer, vendor, model provider, cloud, support, and other subprocessors. Record role, purpose, location, access, retention, deletion, reuse, agreement, configured control, source-version, observed date, limitation, and next action.

Flag disagreement between diagrams, settings, contracts, and testimony. A signed document does not close a technical contradiction.

Claim-evidence and validation register

Give each material claim a stable identifier. Preserve the claim language, source, product version, study context, limitation, evidence class, local protocol, buyer-observed result, authority decision, expiry, and next action.

Keep negative and inconclusive results. Removing them makes the final decision impossible to audit.

Control, workflow, and exception test log

Link control objectives to configured tests. Capture scenario, precondition, expected behavior, observed behavior, reviewer, timestamp, evidence location, defect, downstream effect, correction, retest, status, limitation, and next action.

Include ordinary use, edge cases, misuse, unavailable dependencies, rollback, reconciliation, and recovery. A policy statement without an operating test remains unverified.

Measurement, equity, and incident dictionary

Define every metric with numerator, denominator, exclusions, source, cohort, baseline, window, subgroup, missingness, uncertainty, owner, threshold, cadence, limitation, and response. Connect incidents and near misses to the versions and cohorts affected.

Do not use a dashboard label as the definition. Preserve the query or reproducible extraction method where permitted.

Change, renewal, and exit decision record

Record change class, notice date, affected scope, regression evidence, open risks, incidents, supplier changes, performance, cost, renewal decision, conditions, dissent, expiry, exit test, deletion evidence, continuity owner, and next action.

This record prevents an old approval from becoming permanent permission. It also gives a successor team enough context to renew, narrow, pause, or leave.

FAQs

Does NIST AI RMF certify a healthcare AI product?

No. NIST describes the AI RMF as voluntary guidance for managing AI risks. Using its functions or Playbook can improve governance structure, but it is not a certification, product approval, legal conclusion, or proof that a local control works. Record the framework version and the buyer evidence produced from it.

Does HTI-1 apply to every healthcare AI tool?

No. The relevant HTI-1 provisions have defined certified-health-IT and predictive DSI scope. Determine whether the exact developer, module, intervention, and activity are within that scope. Source attributes or FAVES-related information can be useful evidence where applicable, but they do not certify every AI tool or replace local validation.

Does a BAA make an AI vendor HIPAA compliant?

No. A BAA addresses duties for a particular business associate relationship when that relationship exists. It does not establish that every data flow is permitted, every Security Rule safeguard is satisfied, every output is accurate, or every workflow is safe. First map entity roles and PHI flows, then verify contract and configured behavior.

Does human review make an AI function non-device?

Not by itself. FDA analysis is function-specific and considers intended use, user, output, and whether the healthcare professional can independently review the basis for a recommendation under the relevant guidance. Record the exact facts and qualified analysis; do not rely on the phrase “human in the loop.”

Can vendor evidence be used as local validation?

It can inform the local protocol, but it is not proof of local validation when version, configuration, population, prevalence, language, workflow, comparator, threshold, or user behavior differs. Preserve it as a vendor assertion or external evidence, then record a buyer-observed result separately.

What should stop purchase, pilot, renewal, or expansion?

Stop when a material function, authority, data flow, legal or regulatory question, high-impact claim, critical control, workflow recovery path, harm threshold, change notice, or exit dependency remains unresolved. Record open — local owner/evidence required rather than averaging the gap into a score. Price, speed, adoption, projected savings, and favorable low-risk features do not offset a hard stop.

Evidence boundary

The official materials linked here frame questions within their stated scope. They do not establish that any named or unnamed vendor is approved, compliant, secure, clinically effective, equitable, financially beneficial, or suitable for a local workflow. WHO guidance is not United States law; NIST resources are not certification; ASTP/ONC materials do not cover every AI product; and FDA, HHS, and FTC conclusions depend on the facts and authorities within their remit.

No product testing, customer interview, local measurement, legal conclusion, ROI result, or vendor ranking is claimed in this article. A generic reviewer identity also does not establish specialist signoff. Before a real decision, assign qualified local reviewers, record their evidence and limits, and keep unresolved items open.