Healthcare AI Landscape#

Executive Summary#

Healthcare AI is not a single technology — it is a category spanning FDA-regulated diagnostic software, HIPAA-governed clinical documentation tools, administrative automation that bypasses clinical workflow entirely, and patient-facing conversational systems with their own safety and equity requirements. Understanding the landscape before designing a clinical AI system is not optional context — it determines which regulatory frameworks apply, which failure modes are clinically consequential, and which governance structures the system must operate within. This chapter maps the healthcare AI landscape: clinical AI categories, the FDA's Software as a Medical Device (SaMD) framework, the major market segments, and the architectural positioning decisions that follow from landscape understanding.

Learning Objectives#

After reading this chapter, you will be able to:

• Classify a clinical AI use case within the FDA SaMD framework and identify the applicable regulatory pathway
• Distinguish the four major healthcare AI categories and explain the different engineering requirements each imposes
• Identify the regulatory bodies, standards, and frameworks that govern clinical AI in the United States
• Apply the healthcare AI landscape taxonomy to use case prioritization decisions at an enterprise healthcare organization

Business Problem#

A hospital system that treats all clinical AI as the same type of problem will systematically make wrong decisions about governance, testing, deployment timelines, and vendor selection. An AI that assists medical coders with ICD-10 assignment has fundamentally different regulatory exposure, failure consequences, and governance requirements than an AI that recommends diagnosis codes to a physician reviewing a patient. The coding workflow is administrative; the physician recommendation is a potential SaMD subject to FDA oversight. Using the same governance process for both wastes resources on the administrative use case and provides inadequate protection for the clinical one.

The healthcare AI landscape exists as a structured framework to prevent this category error. Before designing, procuring, or deploying any clinical AI system, architects and executives must answer three landscape questions: What category of clinical function does this AI perform? Does it require FDA regulatory pathway engagement? What HIPAA obligations does it create?

Why This Technology Exists#

Healthcare AI as a defined category emerged from the intersection of two trends that converged around 2012–2017: the availability of large labeled clinical datasets (driven by EHR adoption mandated by the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009) and the maturation of deep learning methods that could exploit those datasets.

Early clinical AI deployments were primarily diagnostic imaging applications — chest X-ray interpretation, diabetic retinopathy screening — where large labeled datasets existed and the task was well-defined enough for supervised learning. The FDA, which had regulated traditional medical devices for decades under the 510(k) clearance pathway, began to develop a regulatory framework for this new class of software starting around 2017 with its Digital Health Innovation Action Plan.

Large Language Models entered healthcare AI in the 2022–2024 period, dramatically expanding the surface area of clinical AI to include clinical documentation, prior authorization, patient communication, and clinical decision support that previously required structured input/output formats incompatible with traditional ML models. This expansion brought both new capability (natural language reasoning over unstructured clinical text) and new risk (the hallucination failure mode in a domain where clinically plausible but factually wrong outputs cause direct patient harm).

Conceptual Explanation#

Healthcare AI can be organized along two axes:

Axis 1 — Clinical Function: What does the AI do in the clinical workflow? Diagnostic functions (helping identify disease states), therapeutic functions (informing or generating treatment plans), monitoring functions (tracking patient status over time), administrative functions (supporting operations and revenue cycle), and documentation functions (assisting with clinical record creation).

Axis 2 — Human-in-Loop Degree: Is a qualified human clinician reviewing and taking responsibility for the AI output before it affects patient care? Autonomous functions (AI directly generates output used in care without human review), advisory functions (AI output informs a clinician who makes a decision), and informational functions (AI output informs administrative processes with no direct patient care impact).

The intersection of these axes determines regulatory classification and governance requirements.

Core Architecture#

The Four Healthcare AI Categories#

Components#

FDA Software as a Medical Device (SaMD) Framework#

The FDA defines Software as a Medical Device as software "intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device." The critical determination is the software's intended use and its role in clinical decision-making.

The 2021 Clinical Decision Support (CDS) Software Guidance:

The FDA issued final guidance in 2022 clarifying which decision support software falls under device regulation and which does not. The key distinction is whether a clinician can independently review the basis for the software's recommendation:

• Non-device CDS: Software displays information the clinician can independently review and verify (e.g., reference information, guideline summaries). The clinician, not the software, makes the clinical decision.
• Device CDS (SaMD): Software makes a recommendation based on patient-specific data that the clinician cannot independently verify — the clinician is relying on the software's analysis, not their own. This is regulated as SaMD.

Regulatory Pathways for Clinical AI:

The PCCP is particularly important for LLM-based clinical AI: it allows a manufacturer to pre-define algorithm changes (including model version updates) that can be implemented without a new 510(k) submission, provided they fall within the scope of the approved PCCP.

ONC Information Blocking Rule#

The 21st Century Cures Act (2016) and its implementing rules from the Office of the National Coordinator for Health Information Technology (ONC) prohibit healthcare organizations, EHR developers, and health information networks from "information blocking" — practices that interfere with access, exchange, or use of electronic health information. Clinical AI systems that access or produce clinical data must not create information blocking conditions.

Joint Commission and Accreditation Standards#

The Joint Commission's accreditation standards increasingly address clinical decision support and clinical AI. Standards around medication management, patient identification, and clinical documentation apply to AI systems that participate in these workflows. Healthcare organizations must ensure that AI system governance aligns with their accreditation requirements.

Implementation Patterns#

Use Case Classification Workflow#

Before initiating any clinical AI project, apply this classification framework:

# Educational Example — Clinical AI Use Case Classifier
# Illustrative classification logic; not a regulatory determination tool
# Organizations must consult with regulatory counsel for actual FDA classification

from dataclasses import dataclass
from enum import Enum


class ClinicalAICategory(Enum):
    DIAGNOSTIC = "diagnostic"
    CLINICAL_DECISION_SUPPORT = "clinical_decision_support"
    CLINICAL_DOCUMENTATION = "clinical_documentation"
    ADMINISTRATIVE = "administrative"
    PATIENT_ENGAGEMENT = "patient_engagement"


class RegulatoryRisk(Enum):
    HIGH = "high"       # Likely SaMD — engage regulatory counsel
    MEDIUM = "medium"   # May qualify as non-device CDS — evaluate carefully
    LOW = "low"         # Organizational governance + HIPAA


@dataclass
class UseCaseClassification:
    category: ClinicalAICategory
    regulatory_risk: RegulatoryRisk
    clinician_in_loop: bool
    patient_data_used: bool
    output_directly_influences_treatment: bool
    regulatory_note: str


def classify_ai_use_case(
    performs_diagnosis_or_treatment_recommendation: bool,
    clinician_can_independently_verify_basis: bool,
    patient_data_used: bool,
    output_directly_influences_treatment: bool,
    is_documentation_only: bool,
    is_administrative_only: bool,
) -> UseCaseClassification:
    """
    Classify a clinical AI use case for regulatory risk and governance tier.
    This is educational scaffolding — final regulatory determinations require
    qualified regulatory counsel and submission to FDA if applicable.
    """
    if is_administrative_only and not output_directly_influences_treatment:
        return UseCaseClassification(
            category=ClinicalAICategory.ADMINISTRATIVE,
            regulatory_risk=RegulatoryRisk.LOW,
            clinician_in_loop=False,
            patient_data_used=patient_data_used,
            output_directly_influences_treatment=False,
            regulatory_note=(
                "Administrative AI. Subject to HIPAA and organizational governance. "
                "FDA SaMD framework likely does not apply."
            ),
        )

    if is_documentation_only and not output_directly_influences_treatment:
        return UseCaseClassification(
            category=ClinicalAICategory.CLINICAL_DOCUMENTATION,
            regulatory_risk=RegulatoryRisk.LOW,
            clinician_in_loop=True,
            patient_data_used=patient_data_used,
            output_directly_influences_treatment=False,
            regulatory_note=(
                "Clinical documentation assistance. Clinician authors final document. "
                "Organizational governance + HIPAA. FDA likely does not apply."
            ),
        )

    if (
        performs_diagnosis_or_treatment_recommendation
        and not clinician_can_independently_verify_basis
    ):
        return UseCaseClassification(
            category=ClinicalAICategory.DIAGNOSTIC,
            regulatory_risk=RegulatoryRisk.HIGH,
            clinician_in_loop=False,
            patient_data_used=True,
            output_directly_influences_treatment=output_directly_influences_treatment,
            regulatory_note=(
                "Likely SaMD under FDA 2022 CDS guidance. Clinician cannot independently "
                "verify basis for recommendation. Engage regulatory counsel before deployment."
            ),
        )

    # CDS where clinician can review the basis
    return UseCaseClassification(
        category=ClinicalAICategory.CLINICAL_DECISION_SUPPORT,
        regulatory_risk=RegulatoryRisk.MEDIUM,
        clinician_in_loop=True,
        patient_data_used=patient_data_used,
        output_directly_influences_treatment=output_directly_influences_treatment,
        regulatory_note=(
            "Potential non-device CDS if clinician can independently review basis. "
            "Evaluate against FDA 2022 CDS guidance. Document the independent review pathway."
        ),
    )

Enterprise Considerations#

AI Governance Committee Structure: Large healthcare organizations operating multiple clinical AI use cases require a formal AI governance structure. The typical structure is an AI Clinical Review Committee (or equivalent) chaired by the Chief Medical Information Officer (CMIO), with representation from clinical informatics, legal/compliance, risk management, and clinical champions from affected departments. The committee reviews new use cases for regulatory classification, clinical validation requirements, and deployment approval.

Vendor Landscape: The healthcare AI market includes three distinct vendor categories with different risk profiles:

• Horizontal AI platforms (Anthropic, Azure OpenAI, Google Vertex): General-purpose LLM infrastructure. No domain-specific clinical training; require clinical prompt engineering and evaluation by the deploying organization.
• Clinical AI vendors (Nuance, Ambient.ai, Pieces Technologies): Domain-specific clinical AI products built on foundation models. Pre-trained on clinical data, often with existing FDA clearances for specific indications. Higher upfront cost; less customization.
• EHR-embedded AI (Epic Cognitive Computing, Oracle Health AI): AI capabilities embedded directly in the EHR. Lowest integration effort; most constrained to the EHR vendor's capabilities and roadmap.

FDA Regulatory Timeline Implications: For any use case that may qualify as SaMD, the regulatory pathway timeline (90 days minimum for a standard 510(k); 12–24 months for De Novo) must be built into project planning. Organizations that discover their clinical AI requires a 510(k) after building and testing it face a forced pause that can be 12–18 months.

Security Considerations#

• Every clinical AI use case involving patient data requires a HIPAA Business Associate Agreement with every vendor in the data processing chain (see Chapter 2)
• Clinical AI that accesses EHR data via FHIR R4 APIs must be registered as an authorized application in the EHR platform with appropriate SMART on FHIR scopes
• AI output that becomes part of the legal medical record must be stored in the EHR, not in a separate AI vendor's data store, to maintain the completeness of the medical record under Joint Commission standards

Healthcare Example#

The Reference Healthcare Organization evaluates seven AI use cases for its AI program. Applying the classification framework:

Outcome: The radiology AI flagging use case requires engagement with regulatory counsel before procurement. The discharge summary and prior auth use cases can proceed under organizational governance and HIPAA. The three CDS use cases are evaluated against the 2022 FDA CDS guidance before classification is finalized.

Common Mistakes#

Assuming All Clinical AI Is the Same. Treating discharge summary AI with the same governance overhead as a diagnostic imaging AI wastes resources and slows non-regulated use cases. Classify first; govern proportionately.

Missing the PCCP for LLM-Based Clinical AI. If an LLM-based clinical AI application requires FDA clearance, failing to file a Predetermined Change Control Plan at submission time means every model version update requires a new 510(k) submission — potentially 90–180 days each. PCCP is the mechanism that allows FDA-cleared LLM applications to update their models within a pre-approved scope.

Procurement Before Regulatory Classification. Organizations that issue RFPs for clinical AI before determining whether the use case requires FDA clearance may procure a vendor product that lacks the required clearance, or discover late that a custom build requires an FDA submission timeline that was not in the project plan.

Best Practices#

• Classify every new clinical AI use case using the FDA SaMD framework and the 2022 CDS guidance before initiating technical design or vendor procurement
• Engage regulatory counsel for any use case where the classification is "Medium" or "High" regulatory risk
• For LLM-based clinical AI that does require FDA clearance, file a PCCP at submission time to enable model updates without repeated submissions
• Establish an AI Clinical Review Committee (or equivalent) before the second clinical AI use case is deployed — waiting until there are governance problems to create governance structure is too late
• Track FDA guidance updates in this space — the Digital Health Center of Excellence at FDA issues updated guidance and pre-submission resources as the regulatory framework for AI/ML-based SaMD continues to evolve

Alternatives#

No alternative exists to the FDA SaMD regulatory framework for clinical AI that meets the SaMD definition — it is federal law, not a choice. For organizations seeking to avoid SaMD classification:

• Non-device CDS design: Structure the AI output as reference information that a clinician can independently verify, rather than a recommendation the clinician relies on without independent verification. This is a legitimate design choice, but the design must genuinely meet the criteria, not merely use different language to describe the same function.
• Procurement of pre-cleared products: Using a vendor product that already has 510(k) clearance eliminates the regulatory burden for the deploying organization (though the organization must still ensure the cleared indication matches the intended use).

Trade-offs#

Interview Questions#

Q: A hospital CMIO asks you to classify the following AI use case under the FDA framework: "An AI system that analyzes a patient's lab results and vital signs over the past 24 hours and generates a risk score, which a nurse reviews and uses to decide whether to escalate to a physician." Is this likely SaMD?#

Category: Architecture / Regulatory Difficulty: Senior Role: AI Architect / FDE / Healthcare AI Consultant

Answer Framework:

This is likely SaMD under the FDA's 2022 CDS Software Guidance, and the key question is whether the clinician (nurse) can independently verify the basis for the risk score.

If the AI shows the nurse the specific lab values, vital sign trends, and thresholds that contributed to the risk score — and the nurse can look at those values herself and apply clinical judgment — this may qualify as non-device CDS under the 2022 guidance. The nurse is making an independent clinical assessment informed by the information the AI organized and displayed.

If the AI produces a risk score (e.g., "Risk: 8.3/10") without displaying the underlying factors, or uses a complex model that produces a score the nurse cannot independently derive from visible patient data, then the clinician is relying on the software's analysis rather than making an independent clinical assessment. This is likely device CDS regulated as SaMD.

The recommendation to the CMIO: document how the AI presents its basis to the nurse. If the design can be made transparent (displaying contributing factors), engage regulatory counsel to evaluate whether the transparent design qualifies as non-device CDS. If the model is inherently opaque (a deep learning score with no explainable output), assume SaMD classification and engage regulatory counsel about the applicable pathway.

Key Points to Hit:

• SaMD determination turns on the independent verification question, not just the clinical nature of the output
• Explainability/transparency is the engineering lever that can shift classification
• Document the design basis for regulatory review
• Engage regulatory counsel — this is a legal question with engineering implications, not an engineering question

Q: What is a Predetermined Change Control Plan (PCCP) and why is it critical for LLM-based clinical AI?#

Category: Architecture / Regulatory Difficulty: Principal Role: AI Architect

Answer Framework:

A PCCP is a plan filed with an FDA submission (510(k) or De Novo) that pre-specifies the types of algorithm changes the manufacturer intends to make post-market, and demonstrates that those changes can be made safely without a new submission. FDA approval of the PCCP allows the manufacturer to implement changes within the plan's scope without repeated 510(k) submissions.

For LLM-based clinical AI, this is critical because LLMs evolve rapidly. A clinical AI application that receives 510(k) clearance against a specific model version (e.g., claude-sonnet-4-6) will need to update its model as newer model versions become available — both to maintain performance and to access improved capabilities. Without a PCCP, each model version update requires a new 510(k) submission with a typical 90-day review timeline. For an application that updates its underlying model quarterly, this means the organization either falls years behind on model versions or files four 510(k) submissions per year.

A well-designed PCCP for LLM-based clinical AI would define: the performance bounds the application must remain within after a change, the evaluation protocol that must be run before implementing a change, the types of model changes in scope (version updates within the same model family) versus out of scope (changes to model architecture or training data class), and the post-market monitoring protocol.

Key Points to Hit:

• PCCP enables model updates within pre-approved scope without new 510(k) submissions
• Critical for LLM applications because foundation model versions change frequently
• PCCP must define performance bounds and evaluation protocol for in-scope changes
• Without PCCP, each model update either requires a new submission or falls outside the cleared product specification

Key Takeaways#

• Healthcare AI spans four distinct categories with different regulatory exposure, failure consequences, and governance requirements: Diagnostic AI, Clinical Decision Support, Clinical Documentation, and Administrative AI
• The FDA's 2022 CDS Software Guidance defines the line between regulated SaMD and non-device CDS: whether a clinician can independently verify the basis for the AI's recommendation
• FDA regulatory pathway timelines (90 days to 24+ months) must be incorporated into clinical AI project planning — discovering a clearance requirement after build is a costly failure
• The Predetermined Change Control Plan (PCCP) is the essential regulatory mechanism for LLM-based clinical AI requiring clearance — it allows model updates within pre-approved scope without repeated submissions
• Every clinical AI use case involving patient data requires HIPAA compliance regardless of FDA classification
• Classify before you build, procure, or commit budget — the classification determines the regulatory obligations that apply

Glossary#

Software as a Medical Device (SaMD): Software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device. Regulated by FDA under the Digital Health framework.

510(k) Pre-Market Notification: An FDA submission demonstrating that a new device is substantially equivalent to a legally marketed predicate device.

Predetermined Change Control Plan (PCCP): An FDA-approved plan that allows a manufacturer to make pre-specified algorithm changes post-market without a new submission.

Clinical Decision Support (CDS): Software designed to assist clinical decision-making by matching patient-specific characteristics to knowledge bases or algorithms.

De Novo Classification: An FDA pathway for novel, lower-risk devices that lack a predicate for 510(k) clearance.

CMIO: Chief Medical Information Officer — the senior clinical leader responsible for clinical informatics and, typically, AI governance in healthcare organizations.

Further Reading#

• Chapter 2: HIPAA and AI — PHI handling requirements for all healthcare AI categories
• Chapter 5: Clinical Decision Support — CDS architecture and the alert fatigue problem
• Chapter 6: HMS Reference Architecture — How the landscape maps to a complete HMS AI implementation
• FDA Digital Health Center of Excellence — Current FDA guidance on AI/ML-based SaMD
• FDA 2022 CDS Software Guidance — The definitive guidance for CDS classification