AI Readiness Assessment#

Executive Summary#

An AI Readiness Assessment determines whether a client organization has the data infrastructure, technical architecture, and organizational capacity to support a production AI deployment — before committing to a POC design or resource allocation. Skipping this step is the single most common cause of POC failures and stalled production migrations: teams discover mid-execution that the data is too sparse, the security review will block cloud AI calls, or no one owns the workflow change management. This chapter provides a structured assessment framework covering three dimensions — data maturity, infrastructure readiness, and organizational readiness — with scoring models, assessment templates, and specific adaptations for healthcare clients where regulatory constraints materially affect every dimension.

Learning Objectives#

• Conduct a structured AI readiness assessment across the three dimensions: data, infrastructure, and organization
• Score client readiness on a defined scale and map scores to recommended engagement approaches
• Identify blocking constraints that must be resolved before a POC can succeed
• Design an assessment report that communicates findings to both technical and executive audiences
• Adapt the assessment framework for healthcare clients with HIPAA, FHIR, and clinical workflow constraints

Business Problem#

Enterprise AI POCs fail for predictable reasons. A study of failed enterprise AI pilots reveals consistent root causes: insufficient data volume or quality for the target use case; integration paths that do not exist or are blocked by security policy; no internal owner for the AI-generated output; and organizational change management that was not planned.

All of these root causes are identifiable before the POC begins — but only if a structured readiness assessment is conducted. An FDE who proceeds directly from discovery to POC design without an assessment is accepting unnecessary risk. The assessment exists to convert unknown risks into known constraints that can be planned around or that justify recommending against a POC in its proposed form.

Why This Framework Exists#

The AI readiness assessment addresses a structural asymmetry in enterprise AI engagements: the FDE has deep knowledge of what the product requires to succeed; the client organization has deep knowledge of its own environment. Neither party has complete information alone.

The assessment is the structured process for combining these knowledge sources. It is not a checklist the FDE fills out independently — it requires collaborative sessions with the client's technical and organizational stakeholders. The output is a shared understanding of readiness, not a vendor-produced evaluation.

Conceptual Explanation#

Readiness has three independent dimensions. An organization can be highly mature on one dimension and poorly prepared on another:

Dimension 1 — Data Maturity: Can the AI system access the data it needs, at the quality level it requires, at the volume that will produce meaningful results?

Dimension 2 — Infrastructure Readiness: Can the AI system run within the client's technical environment, given their cloud posture, network architecture, and security requirements?

Dimension 3 — Organizational Readiness: Does the organization have the team capacity, governance structures, and change management capability to deploy and adopt an AI system?

Each dimension is assessed independently and scored. The overall readiness score determines which engagement approach is appropriate.

Core Architecture: The Assessment Framework#

Dimension 1 — Data Maturity#

Data maturity is assessed across four sub-dimensions: availability, quality, accessibility, and governance.

Data Availability: Does the data the AI needs actually exist in the client's systems?

DATA_AVAILABILITY_ASSESSMENT = {
    "use_case": "Discharge Summary AI",
    "required_data_elements": [
        {
            "element": "Active diagnoses (FHIR Condition)",
            "required_completeness": 0.90,
            "observed_completeness": None,  # To be measured in assessment
            "source_system": "Epic",
            "fhir_resource": "Condition",
            "notes": "Must include ICD-10 codes, not just free text"
        },
        {
            "element": "Active medications (FHIR MedicationRequest)",
            "required_completeness": 0.85,
            "observed_completeness": None,
            "source_system": "Epic Pharmacy",
            "fhir_resource": "MedicationRequest",
            "notes": "Reconciliation status matters — discharged meds vs active"
        },
        {
            "element": "Vital signs (FHIR Observation)",
            "required_completeness": 0.80,
            "observed_completeness": None,
            "source_system": "Epic flowsheets",
            "fhir_resource": "Observation?category=vital-signs",
            "notes": "Admit and most recent — not full flowsheet history"
        },
        {
            "element": "Laboratory results (FHIR Observation)",
            "required_completeness": 0.75,
            "observed_completeness": None,
            "source_system": "Epic / LIS",
            "fhir_resource": "Observation?category=laboratory",
            "notes": "Pertinent labs — not necessarily all results"
        },
    ],
    "minimum_sample_size": 50,   # Encounters to assess for POC viability
    "recommended_sample_size": 200   # For meaningful quality evaluation
}

Data Quality: Is the data accurate, complete, and consistent enough to produce reliable AI output?

Common data quality issues in clinical AI:

• Conditions coded in free text rather than ICD-10 (not machine-parseable)
• Medications without dosing information
• Vital signs in non-standard units or with transcription errors
• Duplicate patient records (patient matching failures)
• Notes in scanned PDFs rather than structured text (requires OCR pipeline)

Data Accessibility: Can the AI system actually reach the data through approved integration paths?

DATA_ACCESSIBILITY_CHECKLIST = [
    "FHIR R4 endpoint URL and environment (sandbox/production) confirmed",
    "SMART on FHIR registration process and timeline understood",
    "App Orchard (or equivalent EHR app store) review process timeline understood",
    "API rate limits documented",
    "PHI-in-cloud policy confirmed (on-premises-only vs. cloud-approved vendors)",
    "Network path from AI system to FHIR endpoint confirmed (firewall rules)",
    "Authentication method confirmed (OAuth 2.0 token endpoint available)",
    "Bulk FHIR export available if batch processing is needed",
]

Data Governance: Who owns the data and can approve AI system access?

Data Maturity Score:

Dimension 2 — Infrastructure Readiness#

Infrastructure readiness covers cloud posture, network architecture, security, and integration capability.

Cloud and Network Architecture Assessment:

INFRASTRUCTURE_ASSESSMENT = {
    "cloud_posture": {
        "question": "What is your primary cloud provider and PHI policy?",
        "options": [
            "Cloud-first: PHI in cloud with signed BAAs (Ready)",
            "Cloud-hybrid: PHI in cloud for approved vendors only (Near-Ready)",
            "On-premises preference: PHI in cloud requires case-by-case approval (Developing)",
            "On-premises only: PHI must not leave on-premises (Blocking)"
        ]
    },
    "llm_connectivity": {
        "question": "Can the AI application reach external LLM API endpoints?",
        "checks": [
            "Outbound HTTPS to api.anthropic.com or api.openai.com is permitted",
            "No SSL inspection proxy that would break certificate chains",
            "DNS resolution for AI vendor domains is available",
            "Latency to external API endpoints is acceptable (< 2 seconds baseline)"
        ]
    },
    "ai_gateway": {
        "question": "Is an AI gateway deployed or planned?",
        "options": [
            "Deployed (LiteLLM Proxy, Azure AI Foundry, AWS Bedrock) (Ready)",
            "Planned within 4 weeks (Near-Ready)",
            "Not planned — direct API calls (Developing)",
            "Not permitted — on-premises LLM only (Blocking)"
        ]
    },
    "security_architecture": {
        "checks": [
            "HIPAA Security Rule technical safeguards in place (TLS 1.2+, AES-256)",
            "Audit logging infrastructure available",
            "Role-based access control (RBAC) for AI applications",
            "Security review process for new applications — timeline understood",
            "Penetration testing requirements for new applications understood"
        ]
    },
    "integration_engine": {
        "question": "What integration engine is in place?",
        "options": [
            "Azure Health Data Services / MuleSoft / InterSystems HealthShare (Ready)",
            "RHAPSODY or equivalent (Near-Ready with FHIR translation layer)",
            "Custom point-to-point integrations (Developing)",
            "None — manual data transfers only (Blocking)"
        ]
    }
}

Infrastructure Readiness Score:

Dimension 3 — Organizational Readiness#

Organizational readiness is the most frequently underestimated dimension. A technically ready organization with low organizational readiness will fail to adopt the AI system even if the POC is technically successful.

Team Capacity Assessment:

Governance Readiness:

GOVERNANCE_READINESS_CHECKS = [
    {
        "check": "AI governance policy exists",
        "why": "Defines approval process for new AI deployments",
        "risk_if_absent": "No clear path to production approval — ad hoc process will be slow and inconsistent"
    },
    {
        "check": "Model Review Board or equivalent exists",
        "why": "Provides formal approval mechanism for clinical AI",
        "risk_if_absent": "Clinical deployment approval has no defined path or timeline"
    },
    {
        "check": "AI vendor BAA process is established",
        "why": "BAA must be signed before PHI is transmitted to any external AI vendor",
        "risk_if_absent": "BAA negotiation can take 4–8 weeks — blocks POC start date"
    },
    {
        "check": "Clinical workflow change management process exists",
        "why": "Deploying AI into clinical workflows requires structured change management",
        "risk_if_absent": "Physician adoption will not happen without structured training and champion support"
    },
    {
        "check": "Executive sponsor is identified and committed",
        "why": "Cross-functional work stalls without executive authority to resolve conflicts",
        "risk_if_absent": "Engagement stalls at first organizational obstacle"
    },
]

Organizational Readiness Score:

Overall Readiness Score and Engagement Recommendation#

def calculate_readiness_recommendation(
    data_score: int,
    infrastructure_score: int,
    organizational_score: int
) -> dict:
    """
    Map three-dimensional readiness scores to engagement recommendation.
    
    Each dimension scored 1-4:
    4 = Ready, 3 = Near-Ready, 2 = Developing, 1 = Not Ready
    """
    overall = (data_score + infrastructure_score + organizational_score) / 3
    
    # Any dimension at 1 (Not Ready) is a blocker regardless of overall score
    blocking_dimensions = []
    if data_score == 1:
        blocking_dimensions.append("Data Maturity")
    if infrastructure_score == 1:
        blocking_dimensions.append("Infrastructure")
    if organizational_score == 1:
        blocking_dimensions.append("Organizational")
    
    if blocking_dimensions:
        return {
            "recommendation": "Remediation Required",
            "rationale": f"Blocking gaps in: {', '.join(blocking_dimensions)}",
            "next_step": "Remediation plan before POC Design",
            "estimated_remediation_weeks": "4–8 weeks depending on gap"
        }
    
    if overall >= 3.5:
        return {
            "recommendation": "Proceed to POC Design",
            "rationale": "High readiness across all dimensions",
            "next_step": "Begin POC scoping session",
            "estimated_poc_start": "2 weeks"
        }
    elif overall >= 2.5:
        return {
            "recommendation": "POC with Parallel Remediation",
            "rationale": "Ready to proceed with parallel track for identified gaps",
            "next_step": "POC Design with gap remediation plan",
            "estimated_poc_start": "3–4 weeks"
        }
    else:
        return {
            "recommendation": "Readiness Sprint",
            "rationale": "Multiple dimensions require strengthening before POC",
            "next_step": "4-week readiness sprint with defined exit criteria",
            "estimated_poc_start": "6–8 weeks"
        }

Architecture Diagram#

Implementation Patterns#

Readiness Assessment Report Structure#

# AI Readiness Assessment Report
**Client:** Reference Healthcare Organization
**Use Case:** Discharge Summary AI
**Assessment Date:** [Date]
**FDE:** [Name]

---

## Executive Summary

[3-sentence summary suitable for CIO/CMIO: overall readiness level,
 primary gaps identified, recommended next step]

## Readiness Scorecard

| Dimension | Score | Level | Key Finding |
|-----------|-------|-------|-------------|
| Data Maturity | 3/4 | Near-Ready | FHIR data complete but App Orchard access pending |
| Infrastructure | 3/4 | Near-Ready | PHI-in-cloud approved; AI gateway provisioning needed |
| Organization | 3/4 | Near-Ready | Executive sponsor confirmed; internal tech lead TBD |
| **Overall** | **3.0/4** | **Near-Ready** | **POC with Parallel Remediation** |

## Recommendation: POC with Parallel Remediation

**POC start estimate:** 3–4 weeks
**Estimated POC duration:** 4 weeks
**Production estimate:** 12–16 weeks post-POC

## Dimension Detail

### Data Maturity (Score: 3/4)

**Findings:**
- 47 of 50 sample encounters (94%) have active conditions in FHIR — above the 90% threshold
- 41 of 50 (82%) have active medications — above the 80% threshold
- Vital signs completeness 91%, labs 78%
- FHIR endpoint confirmed (R4); App Orchard registration in progress (est. 6 weeks)

**Gap:** App Orchard registration timeline creates POC start dependency

**Mitigation:** Use Epic FHIR R4 sandbox with de-identified data during App Orchard review;
transition to production credentials on approval

### Infrastructure Readiness (Score: 3/4)

**Findings:**
- Azure is the approved cloud provider; PHI-in-cloud approved via existing BAAs (Azure HIPAA)
- Outbound HTTPS to Anthropic API endpoints confirmed available
- LiteLLM Proxy not deployed; IT estimates 2 weeks to provision

**Gap:** AI gateway provisioning required before POC

**Mitigation:** Parallel track: IT provisions AI gateway during POC design phase

### Organizational Readiness (Score: 3/4)

**Findings:**
- CMIO confirmed as executive sponsor
- Physician champion (hospitalist) identified by CMIO; introductory meeting not yet scheduled
- No internal technical lead identified for post-FDE ownership
- IT project manager allocated 50%

**Gap:** Internal technical lead required for ownership transfer post-launch

**Mitigation:** Identify during POC phase; FDE to shadow-train throughout execution

## Blocking Issues — None

## Open Items Before POC Design

| Item | Owner | Target Date | Blocking? |
|------|-------|-------------|-----------|
| App Orchard registration submitted | IT Director | [Date] | Partial |
| AI gateway provisioning | IT | [Date + 2 weeks] | Yes |
| Physician champion meeting scheduled | CMIO | [Date + 1 week] | No |
| Internal tech lead identified | CIO | [Date + 2 weeks] | No |
| BAA with Anthropic signed | Compliance | [Date + 3 weeks] | Yes |

## Next Steps

1. Schedule POC Design session for [Date + 3 weeks]
2. FDE to support App Orchard submission documentation
3. IT begins AI gateway provisioning per specifications provided
4. Compliance officer initiates BAA process with Anthropic

Enterprise Considerations#

Portfolio-level readiness patterns: FDEs who have conducted assessments across many clients begin to recognize patterns in readiness gaps by organization type. Academic medical centers often have strong data maturity but slow compliance processes. Community hospitals often have infrastructure gaps but faster organizational decision-making. Building these patterns into assessment playbooks accelerates future assessments.

Readiness as a sales enablement input: The readiness assessment is not purely a technical tool. When shared with the client's executive sponsor, it communicates the FDE's depth of understanding of the client's environment and creates a credibility foundation that accelerates the rest of the engagement.

Remediation as an opportunity: When readiness gaps require remediation (data quality work, infrastructure provisioning, governance setup), the FDE can guide the remediation process — creating additional value and deepening the client relationship during what might otherwise be a waiting period.

Security Considerations#

The readiness assessment involves examining the client's security architecture, which is sensitive information. Assessment notes must be treated as confidential. The section on infrastructure readiness should not be shared in a form that reveals specific network topology, IP ranges, or security control gaps to unauthorized parties.

For healthcare clients, the assessment should specifically evaluate HIPAA Security Rule compliance for the proposed AI integration path — not to perform a HIPAA audit, but to identify integration design decisions that would create compliance risk.

Healthcare Example#

A Reference Healthcare Organization assessment reveals the following pattern, which is common in community hospital settings:

• Data Maturity: 4 — Epic FHIR R4 fully configured; SMART on FHIR available; data completeness exceeds thresholds for all required resources
• Infrastructure: 2 — PHI-in-cloud is not pre-approved; requires case-by-case CIO sign-off; no AI gateway in place; LLM connectivity not tested
• Organizational: 3 — CMIO sponsor confirmed; physician champion identified; no Model Review Board (reviews handled ad hoc)

Overall: 3.0 — POC with Parallel Remediation

The primary risk is infrastructure. The recommendation is a parallel track: IT team pursues PHI-in-cloud approval and AI gateway provisioning during the POC design phase. If this track is not completed in time, the POC falls back to a synthetic data environment that demonstrates the clinical quality of the AI output without requiring PHI connectivity — a meaningful alternative that can still inform the production decision.

Common Mistakes#

1. Skipping the data quality measurement. Self-reported data quality is almost always more optimistic than actual quality. The FDE must run sample queries and count completeness rates empirically.

2. Treating infrastructure readiness as binary. "We're on Azure so we're fine" is not an infrastructure assessment. The FDE must confirm that PHI-in-cloud is approved, that outbound connectivity to LLM APIs is permitted, and that the security review timeline is understood.

3. Not surfacing organizational blocking issues early. Discovering in week 6 of a POC that the compliance officer has not approved the integration is a failure of the assessment phase. Compliance review must be in the assessment scope.

4. Conflating score with recommendation. A score of 3/4 on organizational readiness does not mean "proceed without addressing the gap." The gap must be identified and a closure plan agreed before the POC begins.

5. Not documenting the remediation plan. When the recommendation is "POC with Parallel Remediation," the remediation plan must be specific: who does what, by when, and what is the POC start dependency.

Best Practices#

• Run data quality assessment against a minimum of 50 actual encounters before committing to a POC approach
• Test LLM API connectivity from within the client's network environment before the POC design session
• Include the compliance officer in the assessment — not at the end as a gate, but as a participant throughout
• Share the Assessment Report with the client for accuracy review before finalizing
• Define the remediation plan before the assessment report is delivered — not after
• Never start POC execution with a blocking gap unresolved

Trade-offs#

Assessment depth vs. time cost: A thorough assessment takes 1–2 weeks. Clients who want to start the POC immediately will push back. The FDE must hold the position that unknown blocking gaps discovered mid-POC are more expensive than a 2-week assessment.

Transparency about gaps: Sharing an honest assessment that reveals organizational or data gaps can create uncomfortable conversations. The alternative — proceeding without surfacing them — creates a failed POC that is harder to recover from.

Interview Questions#

Q: A client wants to skip the readiness assessment and go straight to POC execution. How do you respond?#

Category: Behavioral Difficulty: Senior Role: FDE

Answer Framework:

The response is to acknowledge the client's urgency, explain the concrete risk of proceeding without assessment, and propose a compressed assessment approach that addresses the most critical unknowns in the shortest possible time.

The concrete risk framing: the most common cause of POC failure and delay is a blocking constraint that was discoverable in week 1 but was only found in week 4 — a PHI-in-cloud policy that blocks LLM calls, a data quality issue that requires a different integration approach, or a missing BAA that prevents the POC from processing real patient data. These findings restart the POC from scratch and cost more time than the assessment would have taken.

The compressed alternative: if the client's constraint is real, propose a 1-week rapid assessment focused exclusively on the blocking items — data connectivity test, PHI-in-cloud policy confirmation, BAA status, and compliance officer meeting. This is not the full three-dimension assessment but it identifies the most dangerous unknowns before POC execution begins.

Key Points to Hit:

• Acknowledge the urgency rather than dismissing it
• Frame the risk in terms of POC restarts, not process compliance
• Propose a compressed alternative rather than a binary yes/no
• Hold on the non-negotiables (PHI policy, BAA, connectivity test)

Red Flags:

• Agreeing to skip assessment entirely
• Framing assessment as "our process" rather than the client's risk mitigation

Key Takeaways#

• AI readiness assessment covers three independent dimensions: data maturity, infrastructure readiness, and organizational readiness
• A single dimension at "Not Ready" (score 1) is a blocker regardless of overall score
• Data quality must be measured empirically — self-reported quality is unreliable
• Infrastructure readiness includes PHI-in-cloud policy, LLM API connectivity, AI gateway, and security review timeline
• Organizational readiness covers team capacity, governance, and change management — not just executive sponsorship
• The assessment produces a written report shared with the client — not an internal FDE document
• Remediation plans must be specific before the assessment is finalized

Glossary#

Data Maturity: A multi-dimensional measure of data readiness: availability, quality, accessibility, and governance.

PHI-in-Cloud Policy: A client organization's policy governing whether Protected Health Information may be processed in public cloud environments and under what conditions.

Model Review Board (MRB): A governance body that approves AI system deployments in clinical environments. Composition typically includes CMIO, clinical informatics, compliance, and physician champions.

App Orchard: Epic's third-party application review and certification program. Required for new SMART on FHIR applications accessing production Epic environments.

Further Reading#

• Client Discovery Framework — Discovery precedes assessment
• POC to Production — What happens after assessment
• HIPAA and AI — PHI handling requirements assessed in infrastructure dimension
• Enterprise AI Strategy — Organizational readiness in the broader enterprise AI strategy context