AI Governance#

Conceptual Explanation#

AI governance operates at three layers that correspond to different organizational functions.

Policy Layer: The principles, standards, and requirements that define acceptable AI development and deployment. This includes responsible AI principles (fairness, transparency, accountability, safety), data governance policies that define acceptable data use for AI, and risk classification criteria that determine the level of review required for each AI system.

Process Layer: The workflows, approvals, and checkpoints that operationalize policy. This includes the model review board process (who approves clinical AI deployments?), the audit logging requirements (what events must be recorded and for how long?), the incident response procedure (what happens when an AI system produces a harmful output?), and the model change management process (how are model updates validated before reaching production?).

Technical Layer: The infrastructure that makes governance enforceable — audit logging pipelines, access control systems, model versioning registries, evaluation frameworks, and observability tools that surface governance-relevant signals (quality drift, demographic performance disparities, anomalous outputs).

Governance frameworks that operate only at the policy layer are aspirational. Governance frameworks that extend to the technical layer are operational. The distinction is critical: policy says "AI systems must be auditable"; technical layer says "every inference call writes a structured log entry to an append-only audit store with a 7-year retention policy."

Core Architecture#

An enterprise AI governance framework consists of five structural components:

AI Risk Classification: A tiering system that determines the level of governance scrutiny required for each AI use case. Clinical decision support systems in Tier 1 (highest risk) require full clinical validation, FDA classification review, and governance committee approval. Administrative automation tools in Tier 3 (lower risk) require standard IT security review and operational monitoring. Risk tier determines which governance processes apply.

Model Documentation (Model Cards): Standardized documentation produced for every AI model entering production. Model cards record: intended use, out-of-scope uses, training data sources, evaluation methodology, performance metrics by subgroup, known limitations, and update procedures.

Audit Logging Infrastructure: A technical system that records every AI inference — input, output, model version, timestamp, user identity, and confidence signal — in an immutable, queryable log. For clinical AI, audit logs must be retained for the duration required by applicable regulations and must be accessible for clinical investigation, legal discovery, and regulatory audit.

Bias and Fairness Evaluation: A structured process for evaluating AI performance across demographic subgroups before deployment and on an ongoing basis. Disparities in performance across subgroups — race, ethnicity, sex, age, payer status — must be documented and disclosed to clinical stakeholders before clinical deployment.

AI Incident Response: A defined procedure for responding to AI system failures, unexpected outputs, or harmful events. The procedure must specify: detection (how is the incident identified?), escalation (who is notified?), containment (can the system be immediately disabled?), investigation (how is root cause determined?), remediation (how is the model corrected?), and disclosure (who must be notified — patients, regulators, governing bodies?).

Architecture Diagram#

Common Mistakes#

Governance as Approval Theater. The most common failure mode is a governance committee that approves AI systems based on slide decks rather than technical documentation. If the committee has not reviewed a completed model card, a bias evaluation report, and a HIPAA data flow assessment, the approval is not meaningful governance — it is approval theater that creates the appearance of oversight without the substance.

Audit Logs That Contain PHI. Writing patient names, dates of birth, or other PHI to audit logs that are accessible to non-clinical systems is a HIPAA violation. Audit logs must contain hashed or de-identified identifiers, never raw PHI. The audit log design must be reviewed by legal before the first inference.

No Subgroup Analysis. An AI system that achieves 90% overall accuracy but 75% accuracy for Black patients and 78% accuracy for patients with limited English proficiency has a bias problem that the aggregate metric conceals. Subgroup analysis is non-negotiable for clinical AI.

Governance Without Teeth. A governance framework that lacks enforcement mechanisms — the authority to block a deployment, require a rollback, or terminate a vendor contract — is advisory at best. The governance committee must have explicit authority over production deployments.

Best Practices#

• Establish risk classification criteria before the first AI use case, so every subsequent system is evaluated consistently
• Treat model cards as living documents that are updated with every model version change
• Automate bias evaluation as part of the CI/CD pipeline so it runs every time a model is retrained
• Log AI inferences at the infrastructure layer (AI gateway), not the application layer, so logging cannot be bypassed by individual applications
• Require that incident response procedures be tested annually in a simulated governance exercise
• Make model cards accessible to clinical stakeholders in plain language, not just technical documentation
• Require a BAA and data residency confirmation before any PHI-containing data flows to an external AI service

Alternatives#

NIST AI Risk Management Framework (AI RMF). A voluntary framework published by NIST that provides structured guidance on AI risk identification, measurement, management, and governance. Suitable as a foundation for healthcare AI governance; complements rather than replaces healthcare-specific requirements.

EU AI Act Compliance Framework. For organizations with EU operations or patients, the EU AI Act classifies many clinical AI systems as "high-risk AI systems" subject to mandatory conformity assessment, technical documentation, and post-market monitoring. The EU framework is more prescriptive than US guidance.

ISO/IEC 42001 (AI Management Systems). An international standard for AI management systems, analogous to ISO 27001 for information security. Provides a certifiable framework that may satisfy regulatory and contractual requirements.

Trade-offs#

Interview Questions#

Q: Design an AI governance framework for a hospital deploying its first clinical AI system.#

Category: System Design Difficulty: Senior Role: AI Architect

Answer Framework:

Begin with a risk classification system. Clinical AI that influences direct patient care decisions — documentation, alerts, diagnosis support — is Tier 1 and requires the most rigorous governance. Administrative AI — scheduling, coding, prior authorization — is Tier 2 with reduced requirements. Internal tools — staff-facing search, knowledge retrieval — are Tier 3 with standard IT governance.

For Tier 1, the framework requires: a model card completed by the engineering team before governance review, bias evaluation across the patient demographics served, HIPAA data flow documentation including vendor BAA status, clinical validation by domain-expert physicians, and formal Model Review Board approval with documented minutes.

At the technical layer: audit logging for every inference (no raw PHI in logs), model version registry with immutable artifacts, anomaly detection on output quality, and a documented rollback procedure that can execute in under 15 minutes.

Key Points to Hit:

• Risk classification determines which processes apply
• Technical enforcement (audit logging, model registry) makes governance operational
• Bias evaluation is mandatory, not optional
• Clinical validation by qualified physicians is required for Tier 1
• Governance structure must have blocking authority, not just advisory authority

Q: An AI system in your hospital is producing systematically different outputs for patients of different racial groups. You discover this three months after production deployment. What do you do?#

Category: Behavioral / System Design Difficulty: Principal Role: AI Architect

Answer Framework:

This is an active safety issue, not a post-hoc optimization. The immediate response is: contain first, investigate second, remediate third, disclose fourth.

Contain: Determine within the first hour whether the disparity is in a domain that could affect patient safety. If yes, immediately route affected cases to human review and present outputs only as drafts requiring physician review. If the bias is severe enough to constitute a patient safety risk, disable the system until remediated.

Investigate: Pull audit logs for the affected period. Segment outputs by the demographic variable showing disparity. Determine the performance gap magnitude, the number of patients affected, and whether any adverse events are potentially attributable to biased outputs.

Remediate: Retrain with augmented data, or implement post-processing corrections, or restrict the system's use to the population where it performs equitably.

Disclose: Notify the governance committee, legal, and compliance. Evaluate whether any clinical events during the affected period require root cause analysis. Consult legal on whether affected patients or regulators must be notified.

Key Points to Hit:

• Safety first — contain before investigating
• Audit logs are only useful if they exist and are queryable; this is why audit infrastructure matters
• Disclosure to governance is not optional
• The correct response is never to quietly fix it without documentation

Key Takeaways#

• AI governance must be designed as an engineering constraint from the start, not assembled retrospectively for review committees
• Risk classification determines governance intensity; not all AI systems require the same scrutiny
• Model cards are the foundational governance artifact: one per AI system, updated with every version change
• Bias evaluation across demographic subgroups is non-negotiable for clinical AI
• Audit logging must be implemented at the infrastructure layer with no raw PHI in the log
• The Model Review Board must have blocking authority over production deployments to be meaningful
• In healthcare, AI governance intersects directly with HIPAA, FDA regulatory requirements, and clinical liability