Healthcare Data Platforms Vendor Evaluation Guide
A defensible vendor evaluation starts with one bounded data use and a buyer-owned evidence record. It does not start with a feature list, a generic compliance badge, or a composite score. The buyer should be able to trace what enters the platform, what changes, who can act on the result, what leaves, what remains after deletion or recovery, and which owner can stop the workflow.
This guide is for healthcare technology research and procurement planning. It is not medical, clinical, legal, privacy, security, billing, coding, reimbursement, regulatory, or compliance advice. It reports no product tests, customer interviews, local measurements, or vendor rankings. Qualified data, clinical or operational, architecture, privacy, security, legal, compliance, procurement, and finance owners must decide which requirements apply and whether the evidence is adequate.
Gate 1: define scope, authority, and downstream use
Write the decision scope before requesting a demo. Name the source systems, included data classes and fields, environments, organizations, populations, users, downstream recipients, permitted purposes, prohibited secondary uses, and the exact product, module, configuration, and version. For every handoff, record the authoritative source, the transformed or inferred value, the person allowed to correct it, and the person allowed to pause a downstream write or decision.
The United States Core Data for Interoperability is a standardized set of health data classes and elements for nationwide exchange. It can help a buyer build a field inventory, but it does not prove that a local source is complete, that two fields are semantically equivalent, or that a platform resolved identity correctly. A missing local definition remains open even when a data class appears in USCDI.
Check regulatory and network claims only when the proposed use is in scope. The ONC Health IT Certification Program applies to specific certified modules, criteria, standards, and versions; it is not an endorsement of an entire vendor or deployment. TEFCA provides a nationwide health information sharing framework and QHIN context, but participation or a connection claim does not establish local data quality, authorization, or workflow fit.
Evidence to request: an approved scope statement, data-flow diagram, system and field inventory, authoritative-source map, purpose and recipient register, version/configuration record, exclusions, named owners, and downstream pause authority.
Stop condition: a source, purpose, recipient, downstream action, authoritative record, excluded use, configuration, or accountable owner is undefined; or a narrow approval could be reused as enterprise-wide permission.
Gate 2: test schema, semantics, identity, and data quality
Treat syntactic exchange and semantic correctness as different questions. The FHIR Overview describes a standard for exchanging healthcare information and explains that CapabilityStatement and StructureDefinition artifacts define exposed interfaces and use-case constraints. The US Core Implementation Guide adds United States profiles and Must Support expectations. Neither source proves that the proposed platform sends the right profile version, preserves meaning, maps terminology correctly, or fits the buyer's downstream use.
Build a field-level contract for FHIR resources, HL7 messages, files, claims, notes, identifiers, attachments, derived values, linkage keys, terminology, units, status, negation, references, timestamps, provenance, and corrections. Save the exact standard and implementation-guide version, profile, resource, extension, code system, source/target field, transformation, validation rule, and owner. Test ordinary records as well as missing, duplicate, late, conflicting, invalid, merged, unmerged, and corrected records. Include patient-match conflicts and confirm how every downstream copy is reconciled.
Data quality measures need a reproducible definition. A dashboard label such as completeness, match rate, freshness, or transformation success is not evidence until the buyer records the numerator, denominator, exclusions, source query, time window, version, owner, and action threshold. Vendor benchmarks and industry averages cannot replace local baseline evidence.
Evidence to request: profile and field mappings, terminology/value-set mappings, identity rules, validation output, lineage examples, correction workflow, exception samples, reconciliation evidence, and definitions for every reported quality measure.
Stop condition: a required field or semantic transformation lacks an authoritative definition; identity conflicts can pass silently; corrections do not reach downstream systems; or a quality measure cannot be independently reproduced.
Gate 3: verify interfaces, access, export, and portability
Inventory every inbound and outbound interface, operation, actor, service identity, authentication method, authorization scope, endpoint, rate limit, error response, retry rule, and support dependency. A CapabilityStatement or architecture diagram belongs in the evidence record, but it does not replace an observed test against the proposed environment.
For FHIR-based access, the SMART App Launch Implementation Guide provides OAuth-based authorization, authentication, scopes, and capability discovery patterns. It does not prove that a deployment uses least privilege, rotates or revokes credentials correctly, separates tenants, or logs access completely. Test the exact scopes and identities, including excess permission, expired credentials, revoked service accounts, interrupted authorization, and emergency access where applicable.
For population export, the FHIR Bulk Data Access Implementation Guide can frame group/system export and authorization tests. Verify requested and returned resources, fields, history, metadata, attachments, provenance, pagination, updates, deletions, late records, duplicates, status polling, cancellation, retry, rate limits, latency, and reconciliation. A successful sample export does not prove portable or complete data.
The CMS Interoperability and Prior Authorization Final Rule CMS-0057-F sets specified API duties and dates for identified impacted payers. It is not a universal platform requirement. Record the entity, provision, API, and compliance date before using it in a requirement.
Evidence to request: interface inventory, capability/configuration artifacts, access matrix, scope tests, export specification, ordinary and exception results, retry/replay logs, returned-data reconciliation, portability limitations, and owner-approved acceptance criteria.
Stop condition: required data cannot be exported or reconciled; excess access cannot be removed; failures are silent; retry/replay creates unexplained duplicates; or the buyer cannot recover data without undocumented vendor action.
Gate 4: determine privacy, security, and data-rights scope
Map data by purpose and recipient before asking whether a platform is HIPAA compliant. The HHS OCR Business Associates guidance explains business associate and subcontractor relationships involving PHI. Applicability depends on the entity, service, activity, and data flow. A BAA is not proof that every use is permitted or that the system is secure.
For regulated ePHI, the HHS OCR Summary of the HIPAA Security Rule describes risk analysis and safeguards for confidentiality, integrity, and availability. It does not replace the buyer's applicable risk analysis or evidence from the proposed configuration. The NIST Privacy Framework is a voluntary tool for managing privacy risk; its landing page also distinguishes draft work from stable material. It is not certification or a legal determination.
Record collection purpose, minimum necessary or data-minimization decision, fields, identifiers, free text, derived data, logs, backups, support artifacts, subprocessors, support access, tenant and region, retention, secondary use, model training, sale or disclosure restrictions, incident duties, return/export, deletion, backup treatment, termination assistance, and evidence ownership. Contract language and observed controls should be linked separately.
The CISA Secure by Design guidance supports questions about secure defaults and manufacturer responsibility, including availability of controls such as MFA, logging, and SSO. It does not certify a product, guarantee an absence of vulnerabilities, or prove the deployed settings.
Evidence to request: entity/data-flow determination, contract and BAA decision where applicable, subprocessor register, purpose/use restrictions, access review, audit-event sample, risk analysis, configuration evidence, incident process, retention schedule, return/deletion test, backup handling, and exit obligations.
Stop condition: applicability is guessed; a subprocessor or secondary use is undisclosed; least privilege or audit coverage cannot be demonstrated; return/deletion duties are ambiguous; or an unresolved privacy/security failure is treated as a commercial exception.
Gate 5: test resilience, lineage, and change control
Ask how the platform behaves when dependencies fail and when the vendor changes something. Test source outages, schema and terminology changes, patient-match conflicts, delayed batches, duplicate delivery, revoked identities, partial transformations, downstream rejection, regional failover, backup restore, export interruption, support escalation, rollback, and reconciliation. Record detection time, alert owner, containment, corrected state, evidence link, and proof that stale or incorrect data did not silently continue.
The NIST Cybersecurity Framework 2.0 can organize governance and identify, protect, detect, respond, and recover evidence. It is not a certification, maturity score, HIPAA conclusion, or proof of the vendor's actual controls. A report mapped to a framework still needs deployment-specific scope, evidence, limitations, owner, date, and remediation status.
Maintain lineage from source record through identity, terminology, transformation, storage, export, analytics, model feature, and downstream activation. For each material change, preserve the previous and new version, affected paths and cohorts, notice date, classification, regression plan, decision owner, rollback, historical-output traceability, and renewed-review trigger.
If the platform includes AI or predictive functions, the NIST AI Risk Management Framework may organize voluntary risk questions. The ASTP/ONC HTI rules may matter for specific certified health IT and decision-support functions. Neither is universally applicable, and neither proves model quality, fairness, safety, or local fit.
Evidence to request: lineage sample, dependency map, alert and incident records, recovery test, backup/restore evidence, change register, version history, notification terms, regression results, rollback record, residual risks, and approval-resetting criteria.
Stop condition: a historical output cannot be tied to a version and configuration; recovery is untested; a material change can bypass review; rollback or reconciliation is undefined; or unresolved integrity and resilience failures remain hidden in a general risk score.
Gate 6: measure outcomes and record the decision
Define measures before the pilot. Potential measures include source and field coverage, rejected and duplicate records, identity exceptions, terminology and transformation failures, freshness, lineage and audit coverage, access-review exceptions, export completion and latency, recovery and reconciliation results, support effort, unresolved risks, and downstream correction burden. These are fields to populate with local evidence, not benchmarks supplied by this article.
Every measure needs an operational definition, unit, numerator, denominator, exclusions, source query or extract identifier, evidence link, cohort, environment, version/configuration, time window, owner, approver, threshold, missing-data treatment, and ordinary/exception split. Before/after association is not a causal result unless accountable owners approve an evaluation design and attribution method.
The decision record should separate verified facts, local test results, open evidence, failed gates, and not-applicable requirements. Do not assign default numeric weights, a composite score, a benchmark, or a green status. A buyer may introduce scoring only after owners document definitions, evidence requirements, thresholds, and how a failed gate overrides the total.
Record implementation, integration, migration, training, validation, security/privacy review, monitoring, support, exception, recovery, change, renewal, and exit costs as buyer inputs. This guide supplies no cost saving, ROI, adoption, latency, data-quality, incident, or performance number.
Evidence to request: metric dictionary, baseline and pilot extracts, query identifiers, decision thresholds, exception results, unresolved-evidence list, owner approvals, lifecycle cost record, support plan, and time-bounded expansion/renewal/exit decision.
Stop condition: a measure cannot be reproduced; missing evidence is replaced by zero, an industry average, a vendor projection, or a favorable assumption; or a failed privacy, security, integrity, semantic, interoperability, access, export, deletion, audit, or recovery gate is offset by convenience or a total score.
Procurement questions to ask
Use role-owned questions so an unanswered requirement remains visible rather than becoming group approval.
Data owners and clinical or operational owners
- Which source is authoritative for each included field, what downstream action may use it, and who can correct or pause that action?
- Which organizations, populations, environments, purposes, users, and secondary uses are included or excluded, and how will scope expansion be detected?
- Which local completeness, identity, semantic, freshness, correction, and downstream reconciliation results would pass, pause, or stop the pilot?
Architecture, interoperability, and data engineering
- Which exact interface, FHIR or other standard version, profile, resource, operation, terminology, identifier, extension, service identity, and configuration will be deployed?
- How will the team test missing, duplicate, late, conflicting, invalid, merged, corrected, and deleted records from ingestion through every downstream copy?
- Can the platform export and reconcile required data, history, metadata, attachments, provenance, updates, and deletions after failure, retry, replay, migration, or termination?
Privacy, security, legal, and compliance reviewers
- Which entity and data-flow determinations apply to each field, purpose, recipient, support path, subprocessor, region, and secondary use, and who owns unresolved applicability?
- What contract, BAA decision where applicable, access, audit, risk, incident, retention, return, deletion, backup, recovery, and termination evidence applies to the proposed configuration?
- How will the buyer verify least privilege, revoked access, tenant separation, support access, audit coverage, secure defaults, incident duties, and residual risk without relying on a badge or report title?
Procurement, finance, program, and support owners
- Which acquisition, integration, migration, validation, training, privacy/security review, monitoring, support, exception, recovery, renewal, and exit costs belong in the decision record?
- Which SLA, data-rights, change-notice, subprocessor, portability, deletion, transition-assistance, remediation, and termination terms remain open or conditional?
- What evidence triggers expansion, renewal, pause, remediation, exit, or a new review, and when does the current decision expire?
Buyer-owned artifacts
The final review should leave records another qualified reviewer can reconstruct. These are planning artifacts, not a certification, not a validated instrument, not a legal opinion or product test, and not a vendor ranking. Preserve negative findings and open evidence so the packet does not become a sales summary.
Scope, authority, and data inventory register
Create one row per source, data class, field group, purpose, recipient, and downstream use. Record the organization and environment, source system, authoritative status, data owner, included fields and cohorts, exclusions, permitted and prohibited use, downstream action, correction and pause authority, product/module/version/configuration, evidence link, status, expiration, and renewed-review trigger.
Include identifiers, free text, attachments, derived values, linkage keys, metadata, logs, temporary stores, backups, support artifacts, exports, model features, and downstream copies when they are in scope. A field that lacks an owner, purpose, or authoritative definition remains open — owner/evidence required.
Field mapping, lineage, and reconciliation matrix
Create one row per source-to-target mapping. Record the source and target field, FHIR profile/resource or other format, standard and implementation-guide version, terminology and value set, unit, identifier, extension, status/negation rule, transformation, provenance, validation, ordinary case, exception, correction path, downstream copies, reconciliation evidence, owner, and decision.
Test missing, duplicate, late, conflicting, invalid, merged, unmerged, changed, and deleted records. Link observed results rather than marking a row passed because a field name, USCDI class, FHIR resource, or vendor diagram exists.
Interface, access, export, deletion, and recovery test matrix
Create one row per endpoint, operation, actor, service identity, authorization scope, ordinary case, exception, export requirement, deletion path, and recovery path. Record the proposed environment and configuration, precondition, test data boundary, expected behavior, observed behavior, timestamp, reviewer, evidence link, error/alert, retry/replay, data coverage, detectability, correction, reconciliation, recovery, and pass, failed, open, or not-applicable decision.
Include credential expiry and revocation, excess permission, unavailable dependencies, partial responses, pagination, rate limits, late data, duplicates, updates, history, metadata, attachments, provenance, deletion markers, backup treatment, migration, termination export, and restore. A successful happy path does not close the exception rows.
Vendor evidence and change register
Create one row per standard, profile, certification criterion, framework mapping, security/control report, contract, BAA decision where applicable, subprocessor, SLA, incident commitment, support dependency, and material change. Record exact applicability, covered product, module, version, configuration, entity, and data flow, issuing party, evidence date, limitations, exceptions, owner, expiration, remediation, residual risk, change-notice terms, regression requirement, rollback, and review-reset trigger.
Separate the evidence label from the observed control. FHIR support, certification, TEFCA participation, a BAA, a NIST mapping, a CISA pledge, or a security report cannot inherit a passed result outside its documented scope.
Decision scorecard without invented scoring
Create one row per buyer requirement and use only these evidence states: verified fact / local test / open / failed / not applicable. Link the evidence, owner, scope, limitation, date, expiry, exception, remediation, and decision. Use open — owner/evidence required whenever a necessary input or approval is missing.
There is no default numeric weight, composite score, benchmark, or green status. If accountable owners later choose a scoring method, they must document the requirement definitions, evidence thresholds, weighting rationale, override rules, and approval. A failed privacy, security, integrity, semantic, interoperability, access, export, deletion, audit, or recovery gate overrides a favorable total until the failure is resolved or the proposed scope is changed.
FAQs
Does FHIR support prove semantic interoperability?
No. FHIR defines exchange structures and related implementation artifacts. A buyer still needs the exact version, profiles, resources, extensions, terminology, identifiers, operations, roles, configuration, and observed local tests. Transport success does not prove that fields are complete, semantically equivalent, correctly linked, or safe for the proposed downstream use.
Do ONC Certification, TEFCA, or CMS API rules apply to every platform?
No. ONC Certification applies to specific certified modules, criteria, standards, and versions. TEFCA participation and QHIN claims have defined network roles. CMS-0057-F identifies impacted payers, provisions, APIs, and compliance dates. Record the exact entity, module, role, use case, provision, version, and date before treating any of them as a requirement or evidence.
Does a BAA or security report prove compliance and security?
No. A qualified owner must first determine the entity, activity, data, purpose, and contract scope. A BAA, framework mapping, architecture diagram, penetration-test summary, certification, or control report may be an evidence item, but it does not establish complete legal compliance, least privilege, secure configuration, incident readiness, or recovery in the proposed deployment.
How should missing vendor evidence be scored?
Use open — owner/evidence required. Do not enter zero, an industry average, a vendor projection, a midpoint, a default weight, or a favorable color. Preserve the missing item, accountable owner, request date, decision impact, and condition required to close it.
What should stop expansion?
Stop when a required owner or authoritative definition is missing; semantics or identity cannot be reconciled; access, export, return, deletion, audit, backup, recovery, or rollback cannot be demonstrated; a material change can bypass review; or a privacy, security, integrity, interoperability, legal-applicability, or downstream safety issue remains unresolved. A strong demo or total score does not override a failed gate.
Evidence boundary
The linked public standards, rules, and frameworks define vocabulary, scope questions, and evidence requests. They do not establish product performance, local workflow fit, legal applicability, privacy or security compliance, semantic correctness, interoperability, data rights, or vendor approval. No product tests, customer interviews, local measurements, or rankings are claimed.
Continue your HealthAIdir research
- Ground vendor scoring in AI for healthcare data infrastructure and AI for EHR integration before comparing platform claims.
- Use the Healthcare AI Vendor Readiness Checker to keep verified facts, local tests, failed gates, and open evidence separate.
- Use the HIPAA / BAA Evidence Checklist before treating privacy, security, hosting, or support-access answers as complete.
- Review data provenance, data retention, and audit log, then move to reviews or compare pages only after unresolved risks are visible.