What to Check Before Using AI with PHI

First identify the data

Before putting AI into a healthcare workflow, identify whether the tool will receive PHI, de-identified data, operational data, or no patient data at all. The answer may change by feature. A vendor may support a safe public marketing workflow and a separate clinical workflow that requires stronger controls.

PHI can appear in prompts, uploaded files, screenshots, audio, transcripts, generated summaries, support tickets, logs, and analytics events. A procurement review should cover each of those data paths.

Contract and configuration questions

Ask whether the vendor offers a BAA for the exact product and feature set. Confirm subcontractors, data locations, retention, deletion, breach notification, audit logs, and whether customer data is used to train or improve models. If the answer depends on an enterprise configuration, document that configuration before a pilot.

For internal rollout, define who may use the tool, what data they may enter, where outputs may be copied, and how violations are monitored. A safe vendor can still be misused by an unclear workflow.

Practical pilot controls

Begin with a narrow workflow, limited users, documented allowed data types, and a review period. Monitor whether staff paste more information than expected, whether outputs include hidden PHI, and whether the tool stores content in places the organization did not intend.

Source notes

Related glossary entries include PHI, HIPAA, BAA, and HIPAA-compliant AI. HHS publishes official material on the HIPAA Privacy Rule, business associates, and the Security Rule.