Healthcare AI Landscape#

Conceptual Explanation#

Healthcare AI can be organized along two axes:

Axis 1 — Clinical Function: What does the AI do in the clinical workflow? Diagnostic functions (helping identify disease states), therapeutic functions (informing or generating treatment plans), monitoring functions (tracking patient status over time), administrative functions (supporting operations and revenue cycle), and documentation functions (assisting with clinical record creation).

Axis 2 — Human-in-Loop Degree: Is a qualified human clinician reviewing and taking responsibility for the AI output before it affects patient care? Autonomous functions (AI directly generates output used in care without human review), advisory functions (AI output informs a clinician who makes a decision), and informational functions (AI output informs administrative processes with no direct patient care impact).

The intersection of these axes determines regulatory classification and governance requirements.

Core Architecture#

The Four Healthcare AI Categories#

Common Mistakes#

Assuming All Clinical AI Is the Same. Treating discharge summary AI with the same governance overhead as a diagnostic imaging AI wastes resources and slows non-regulated use cases. Classify first; govern proportionately.

Missing the PCCP for LLM-Based Clinical AI. If an LLM-based clinical AI application requires FDA clearance, failing to file a Predetermined Change Control Plan at submission time means every model version update requires a new 510(k) submission — potentially 90–180 days each. PCCP is the mechanism that allows FDA-cleared LLM applications to update their models within a pre-approved scope.

Procurement Before Regulatory Classification. Organizations that issue RFPs for clinical AI before determining whether the use case requires FDA clearance may procure a vendor product that lacks the required clearance, or discover late that a custom build requires an FDA submission timeline that was not in the project plan.

Best Practices#

• Classify every new clinical AI use case using the FDA SaMD framework and the 2022 CDS guidance before initiating technical design or vendor procurement
• Engage regulatory counsel for any use case where the classification is "Medium" or "High" regulatory risk
• For LLM-based clinical AI that does require FDA clearance, file a PCCP at submission time to enable model updates without repeated submissions
• Establish an AI Clinical Review Committee (or equivalent) before the second clinical AI use case is deployed — waiting until there are governance problems to create governance structure is too late
• Track FDA guidance updates in this space — the Digital Health Center of Excellence at FDA issues updated guidance and pre-submission resources as the regulatory framework for AI/ML-based SaMD continues to evolve

Alternatives#

No alternative exists to the FDA SaMD regulatory framework for clinical AI that meets the SaMD definition — it is federal law, not a choice. For organizations seeking to avoid SaMD classification:

• Non-device CDS design: Structure the AI output as reference information that a clinician can independently verify, rather than a recommendation the clinician relies on without independent verification. This is a legitimate design choice, but the design must genuinely meet the criteria, not merely use different language to describe the same function.
• Procurement of pre-cleared products: Using a vendor product that already has 510(k) clearance eliminates the regulatory burden for the deploying organization (though the organization must still ensure the cleared indication matches the intended use).

Trade-offs#

Interview Questions#

Q: A hospital CMIO asks you to classify the following AI use case under the FDA framework: "An AI system that analyzes a patient's lab results and vital signs over the past 24 hours and generates a risk score, which a nurse reviews and uses to decide whether to escalate to a physician." Is this likely SaMD?#

Category: Architecture / Regulatory Difficulty: Senior Role: AI Architect / FDE / Healthcare AI Consultant

Answer Framework:

This is likely SaMD under the FDA's 2022 CDS Software Guidance, and the key question is whether the clinician (nurse) can independently verify the basis for the risk score.

If the AI shows the nurse the specific lab values, vital sign trends, and thresholds that contributed to the risk score — and the nurse can look at those values herself and apply clinical judgment — this may qualify as non-device CDS under the 2022 guidance. The nurse is making an independent clinical assessment informed by the information the AI organized and displayed.

If the AI produces a risk score (e.g., "Risk: 8.3/10") without displaying the underlying factors, or uses a complex model that produces a score the nurse cannot independently derive from visible patient data, then the clinician is relying on the software's analysis rather than making an independent clinical assessment. This is likely device CDS regulated as SaMD.

The recommendation to the CMIO: document how the AI presents its basis to the nurse. If the design can be made transparent (displaying contributing factors), engage regulatory counsel to evaluate whether the transparent design qualifies as non-device CDS. If the model is inherently opaque (a deep learning score with no explainable output), assume SaMD classification and engage regulatory counsel about the applicable pathway.

Key Points to Hit:

• SaMD determination turns on the independent verification question, not just the clinical nature of the output
• Explainability/transparency is the engineering lever that can shift classification
• Document the design basis for regulatory review
• Engage regulatory counsel — this is a legal question with engineering implications, not an engineering question

Q: What is a Predetermined Change Control Plan (PCCP) and why is it critical for LLM-based clinical AI?#

Category: Architecture / Regulatory Difficulty: Principal Role: AI Architect

Answer Framework:

A PCCP is a plan filed with an FDA submission (510(k) or De Novo) that pre-specifies the types of algorithm changes the manufacturer intends to make post-market, and demonstrates that those changes can be made safely without a new submission. FDA approval of the PCCP allows the manufacturer to implement changes within the plan's scope without repeated 510(k) submissions.

For LLM-based clinical AI, this is critical because LLMs evolve rapidly. A clinical AI application that receives 510(k) clearance against a specific model version (e.g., claude-sonnet-4-6) will need to update its model as newer model versions become available — both to maintain performance and to access improved capabilities. Without a PCCP, each model version update requires a new 510(k) submission with a typical 90-day review timeline. For an application that updates its underlying model quarterly, this means the organization either falls years behind on model versions or files four 510(k) submissions per year.

A well-designed PCCP for LLM-based clinical AI would define: the performance bounds the application must remain within after a change, the evaluation protocol that must be run before implementing a change, the types of model changes in scope (version updates within the same model family) versus out of scope (changes to model architecture or training data class), and the post-market monitoring protocol.

Key Points to Hit:

• PCCP enables model updates within pre-approved scope without new 510(k) submissions
• Critical for LLM applications because foundation model versions change frequently
• PCCP must define performance bounds and evaluation protocol for in-scope changes
• Without PCCP, each model update either requires a new submission or falls outside the cleared product specification

Key Takeaways#

• Healthcare AI spans four distinct categories with different regulatory exposure, failure consequences, and governance requirements: Diagnostic AI, Clinical Decision Support, Clinical Documentation, and Administrative AI
• The FDA's 2022 CDS Software Guidance defines the line between regulated SaMD and non-device CDS: whether a clinician can independently verify the basis for the AI's recommendation
• FDA regulatory pathway timelines (90 days to 24+ months) must be incorporated into clinical AI project planning — discovering a clearance requirement after build is a costly failure
• The Predetermined Change Control Plan (PCCP) is the essential regulatory mechanism for LLM-based clinical AI requiring clearance — it allows model updates within pre-approved scope without repeated submissions
• Every clinical AI use case involving patient data requires HIPAA compliance regardless of FDA classification
• Classify before you build, procure, or commit budget — the classification determines the regulatory obligations that apply