AI Readiness Assessment

Conceptual Explanation

Readiness has three independent dimensions. An organization can be highly mature on one dimension and poorly prepared on another:

Dimension 1 — Data Maturity: Can the AI system access the data it needs, at the quality level it requires, at the volume that will produce meaningful results?

Dimension 2 — Infrastructure Readiness: Can the AI system run within the client's technical environment, given their cloud posture, network architecture, and security requirements?

Dimension 3 — Organizational Readiness: Does the organization have the team capacity, governance structures, and change management capability to deploy and adopt an AI system?

Each dimension is assessed independently and scored. The overall readiness score determines which engagement approach is appropriate.

Core Architecture: The Assessment Framework

Dimension 1 — Data Maturity

Data maturity is assessed across four sub-dimensions: availability, quality, accessibility, and governance.

Data Availability: Does the data the AI needs actually exist in the client's systems?

python

DATA_AVAILABILITY_ASSESSMENT = {
    "use_case": "Discharge Summary AI",
    "required_data_elements": [
        {
            "element": "Active diagnoses (FHIR Condition)",
            "required_completeness": 0.90,
            "observed_completeness": None,  # To be measured in assessment
            "source_system": "Epic",
            "fhir_resource": "Condition",
            "notes": "Must include ICD-10 codes, not just free text"
        },
        {
            "element": "Active medications (FHIR MedicationRequest)",
            "required_completeness": 0.85,
            "observed_completeness": None,
            "source_system": "Epic Pharmacy",
            "fhir_resource": "MedicationRequest",
            "notes": "Reconciliation status matters — discharged meds vs active"
        },
        {
            "element": "Vital signs (FHIR Observation)",
            "required_completeness": 0.80,
            "observed_completeness": None,
            "source_system": "Epic flowsheets",
            "fhir_resource": "Observation?category=vital-signs",
            "notes": "Admit and most recent — not full flowsheet history"
        },
        {
            "element": "Laboratory results (FHIR Observation)",
            "required_completeness": 0.75,
            "observed_completeness": None,
            "source_system": "Epic / LIS",
            "fhir_resource": "Observation?category=laboratory",
            "notes": "Pertinent labs — not necessarily all results"
        },
    ],
    "minimum_sample_size": 50,   # Encounters to assess for POC viability
    "recommended_sample_size": 200   # For meaningful quality evaluation
}

Data Quality: Is the data accurate, complete, and consistent enough to produce reliable AI output?

Common data quality issues in clinical AI:

Conditions coded in free text rather than ICD-10 (not machine-parseable)
Medications without dosing information
Vital signs in non-standard units or with transcription errors
Duplicate patient records (patient matching failures)
Notes in scanned PDFs rather than structured text (requires OCR pipeline)

Data Accessibility: Can the AI system actually reach the data through approved integration paths?

python

DATA_ACCESSIBILITY_CHECKLIST = [
    "FHIR R4 endpoint URL and environment (sandbox/production) confirmed",
    "SMART on FHIR registration process and timeline understood",
    "App Orchard (or equivalent EHR app store) review process timeline understood",
    "API rate limits documented",
    "PHI-in-cloud policy confirmed (on-premises-only vs. cloud-approved vendors)",
    "Network path from AI system to FHIR endpoint confirmed (firewall rules)",
    "Authentication method confirmed (OAuth 2.0 token endpoint available)",
    "Bulk FHIR export available if batch processing is needed",
]

Data Governance: Who owns the data and can approve AI system access?

Question	Why It Matters
Who approves new application access to PHI?	Determines the approval timeline and process
What is the data residency requirement?	Some organizations prohibit PHI leaving on-premises
Is there a data classification policy that covers AI use?	PHI handling rules may already exist
Is there a BAA process established?	Determines time-to-contract for AI vendors
Who owns the output data (AI-generated summaries)?	Affects how results can be stored and used

Data Maturity Score:

Score	Level	Description	Recommended Approach
4	Ready	All required data available at required completeness; accessible via approved path	Proceed to POC Design
3	Near-Ready	Minor gaps in completeness or accessibility; resolvable within 2–4 weeks	POC Design with data remediation plan
2	Developing	Significant gaps in one area (quality, accessibility, or governance)	Assessment remediation before POC
1	Not Ready	Multiple blocking gaps	Data foundation work required before AI engagement

Dimension 2 — Infrastructure Readiness

Infrastructure readiness covers cloud posture, network architecture, security, and integration capability.

Cloud and Network Architecture Assessment:

python

INFRASTRUCTURE_ASSESSMENT = {
    "cloud_posture": {
        "question": "What is your primary cloud provider and PHI policy?",
        "options": [
            "Cloud-first: PHI in cloud with signed BAAs (Ready)",
            "Cloud-hybrid: PHI in cloud for approved vendors only (Near-Ready)",
            "On-premises preference: PHI in cloud requires case-by-case approval (Developing)",
            "On-premises only: PHI must not leave on-premises (Blocking)"
        ]
    },
    "llm_connectivity": {
        "question": "Can the AI application reach external LLM API endpoints?",
        "checks": [
            "Outbound HTTPS to api.anthropic.com or api.openai.com is permitted",
            "No SSL inspection proxy that would break certificate chains",
            "DNS resolution for AI vendor domains is available",
            "Latency to external API endpoints is acceptable (< 2 seconds baseline)"
        ]
    },
    "ai_gateway": {
        "question": "Is an AI gateway deployed or planned?",
        "options": [
            "Deployed (LiteLLM Proxy, Azure AI Foundry, AWS Bedrock) (Ready)",
            "Planned within 4 weeks (Near-Ready)",
            "Not planned — direct API calls (Developing)",
            "Not permitted — on-premises LLM only (Blocking)"
        ]
    },
    "security_architecture": {
        "checks": [
            "HIPAA Security Rule technical safeguards in place (TLS 1.2+, AES-256)",
            "Audit logging infrastructure available",
            "Role-based access control (RBAC) for AI applications",
            "Security review process for new applications — timeline understood",
            "Penetration testing requirements for new applications understood"
        ]
    },
    "integration_engine": {
        "question": "What integration engine is in place?",
        "options": [
            "Azure Health Data Services / MuleSoft / InterSystems HealthShare (Ready)",
            "RHAPSODY or equivalent (Near-Ready with FHIR translation layer)",
            "Custom point-to-point integrations (Developing)",
            "None — manual data transfers only (Blocking)"
        ]
    }
}

Infrastructure Readiness Score:

Score	Level	Description	Recommended Approach
4	Ready	Cloud PHI approved; LLM connectivity confirmed; security review process defined	Proceed to POC Design
3	Near-Ready	PHI cloud approval pending; AI gateway not deployed but planned	POC Design with infrastructure provisioning track
2	Developing	PHI cloud policy unclear; connectivity blocked pending security review	Infrastructure readiness sprint before POC
1	Not Ready	On-premises only; no connectivity; security review timeline > 8 weeks	Recommend private deployment option or defer

Dimension 3 — Organizational Readiness

Organizational readiness is the most frequently underestimated dimension. A technically ready organization with low organizational readiness will fail to adopt the AI system even if the POC is technically successful.

Team Capacity Assessment:

Role	Question	Why It Matters
Internal AI technical lead	Is there an engineer who will own the integration post-FDE?	Without ownership transfer, the system atrophies
Clinical champion	Is there a physician who will advocate internally for adoption?	Clinical adoption does not happen without peer advocates
IT project manager	Is there a project manager who can coordinate the cross-functional work?	FDE time is too expensive to spend on coordination overhead
Change management lead	Is there someone responsible for training and adoption?	POC success does not produce adoption without active change management
Compliance resource	Is the privacy officer allocated to this project?	HIPAA review delays are the most common production blocker

Governance Readiness:

python

GOVERNANCE_READINESS_CHECKS = [
    {
        "check": "AI governance policy exists",
        "why": "Defines approval process for new AI deployments",
        "risk_if_absent": "No clear path to production approval — ad hoc process will be slow and inconsistent"
    },
    {
        "check": "Model Review Board or equivalent exists",
        "why": "Provides formal approval mechanism for clinical AI",
        "risk_if_absent": "Clinical deployment approval has no defined path or timeline"
    },
    {
        "check": "AI vendor BAA process is established",
        "why": "BAA must be signed before PHI is transmitted to any external AI vendor",
        "risk_if_absent": "BAA negotiation can take 4–8 weeks — blocks POC start date"
    },
    {
        "check": "Clinical workflow change management process exists",
        "why": "Deploying AI into clinical workflows requires structured change management",
        "risk_if_absent": "Physician adoption will not happen without structured training and champion support"
    },
    {
        "check": "Executive sponsor is identified and committed",
        "why": "Cross-functional work stalls without executive authority to resolve conflicts",
        "risk_if_absent": "Engagement stalls at first organizational obstacle"
    },
]

Organizational Readiness Score:

Score	Level	Description	Recommended Approach
4	Ready	Executive sponsor confirmed; clinical champion identified; AI governance in place; internal technical owner designated	Proceed to POC Design
3	Near-Ready	Most elements in place; 1–2 gaps resolvable within 2 weeks	POC Design with organizational gap closure plan
2	Developing	Clinical champion or internal technical owner absent; governance unclear	Recommend organizational readiness sprint
1	Not Ready	No executive sponsor; no internal technical capacity; no AI governance	Defer until organizational foundation is in place

Overall Readiness Score and Engagement Recommendation

python

def calculate_readiness_recommendation(
    data_score: int,
    infrastructure_score: int,
    organizational_score: int
) -> dict:
    """
    Map three-dimensional readiness scores to engagement recommendation.
    
    Each dimension scored 1-4:
    4 = Ready, 3 = Near-Ready, 2 = Developing, 1 = Not Ready
    """
    overall = (data_score + infrastructure_score + organizational_score) / 3
    
    # Any dimension at 1 (Not Ready) is a blocker regardless of overall score
    blocking_dimensions = []
    if data_score == 1:
        blocking_dimensions.append("Data Maturity")
    if infrastructure_score == 1:
        blocking_dimensions.append("Infrastructure")
    if organizational_score == 1:
        blocking_dimensions.append("Organizational")
    
    if blocking_dimensions:
        return {
            "recommendation": "Remediation Required",
            "rationale": f"Blocking gaps in: {', '.join(blocking_dimensions)}",
            "next_step": "Remediation plan before POC Design",
            "estimated_remediation_weeks": "4–8 weeks depending on gap"
        }
    
    if overall >= 3.5:
        return {
            "recommendation": "Proceed to POC Design",
            "rationale": "High readiness across all dimensions",
            "next_step": "Begin POC scoping session",
            "estimated_poc_start": "2 weeks"
        }
    elif overall >= 2.5:
        return {
            "recommendation": "POC with Parallel Remediation",
            "rationale": "Ready to proceed with parallel track for identified gaps",
            "next_step": "POC Design with gap remediation plan",
            "estimated_poc_start": "3–4 weeks"
        }
    else:
        return {
            "recommendation": "Readiness Sprint",
            "rationale": "Multiple dimensions require strengthening before POC",
            "next_step": "4-week readiness sprint with defined exit criteria",
            "estimated_poc_start": "6–8 weeks"
        }

Architecture Diagram

graph TD START["Discovery Complete\nDiscovery Summary delivered"] subgraph "Dimension 1 — Data Maturity" D1["Data Availability\nRequired elements present?"] D2["Data Quality\nCompleteness, accuracy, consistency"] D3["Data Accessibility\nFHIR access, network path, auth"] D4["Data Governance\nApproval process, BAA, residency"] DS["Data Score\n1–4"] end subgraph "Dimension 2 — Infrastructure" I1["Cloud Posture\nPHI in cloud approved?"] I2["LLM Connectivity\nOutbound API access?"] I3["AI Gateway\nDeployed or planned?"] I4["Security Architecture\nTLS, audit log, RBAC"] IS["Infrastructure Score\n1–4"] end subgraph "Dimension 3 — Organization" O1["Team Capacity\nTech lead, champion, PM"] O2["Governance\nAI policy, MRB, exec sponsor"] O3["Change Management\nTraining plan, adoption owner"] OS["Organizational Score\n1–4"] end subgraph "Recommendation" R1["Proceed to POC Design\n≥ 3.5 overall, no blocking"] R2["POC + Parallel Remediation\n2.5–3.5 overall"] R3["Readiness Sprint\n< 2.5 or one dimension blocking"] R4["Defer\nMultiple blocking gaps"] end START --> D1 & I1 & O1 D1 & D2 & D3 & D4 --> DS I1 & I2 & I3 & I4 --> IS O1 & O2 & O3 --> OS DS & IS & OS --> R1 DS & IS & OS --> R2 DS & IS & OS --> R3 DS & IS & OS --> R4

Common Mistakes

1. Skipping the data quality measurement. Self-reported data quality is almost always more optimistic than actual quality. The FDE must run sample queries and count completeness rates empirically.

2. Treating infrastructure readiness as binary. "We're on Azure so we're fine" is not an infrastructure assessment. The FDE must confirm that PHI-in-cloud is approved, that outbound connectivity to LLM APIs is permitted, and that the security review timeline is understood.

3. Not surfacing organizational blocking issues early. Discovering in week 6 of a POC that the compliance officer has not approved the integration is a failure of the assessment phase. Compliance review must be in the assessment scope.

4. Conflating score with recommendation. A score of 3/4 on organizational readiness does not mean "proceed without addressing the gap." The gap must be identified and a closure plan agreed before the POC begins.

5. Not documenting the remediation plan. When the recommendation is "POC with Parallel Remediation," the remediation plan must be specific: who does what, by when, and what is the POC start dependency.

Best Practices

Run data quality assessment against a minimum of 50 actual encounters before committing to a POC approach
Test LLM API connectivity from within the client's network environment before the POC design session
Include the compliance officer in the assessment — not at the end as a gate, but as a participant throughout
Share the Assessment Report with the client for accuracy review before finalizing
Define the remediation plan before the assessment report is delivered — not after
Never start POC execution with a blocking gap unresolved

Trade-offs

Assessment depth vs. time cost: A thorough assessment takes 1–2 weeks. Clients who want to start the POC immediately will push back. The FDE must hold the position that unknown blocking gaps discovered mid-POC are more expensive than a 2-week assessment.

Transparency about gaps: Sharing an honest assessment that reveals organizational or data gaps can create uncomfortable conversations. The alternative — proceeding without surfacing them — creates a failed POC that is harder to recover from.

Interview Questions

Q: A client wants to skip the readiness assessment and go straight to POC execution. How do you respond?

Category: Behavioral Difficulty: Senior Role: FDE

Answer Framework:

The response is to acknowledge the client's urgency, explain the concrete risk of proceeding without assessment, and propose a compressed assessment approach that addresses the most critical unknowns in the shortest possible time.

The concrete risk framing: the most common cause of POC failure and delay is a blocking constraint that was discoverable in week 1 but was only found in week 4 — a PHI-in-cloud policy that blocks LLM calls, a data quality issue that requires a different integration approach, or a missing BAA that prevents the POC from processing real patient data. These findings restart the POC from scratch and cost more time than the assessment would have taken.

The compressed alternative: if the client's constraint is real, propose a 1-week rapid assessment focused exclusively on the blocking items — data connectivity test, PHI-in-cloud policy confirmation, BAA status, and compliance officer meeting. This is not the full three-dimension assessment but it identifies the most dangerous unknowns before POC execution begins.

Key Points to Hit:

Acknowledge the urgency rather than dismissing it
Frame the risk in terms of POC restarts, not process compliance
Propose a compressed alternative rather than a binary yes/no
Hold on the non-negotiables (PHI policy, BAA, connectivity test)

Red Flags:

Agreeing to skip assessment entirely
Framing assessment as "our process" rather than the client's risk mitigation

Key Takeaways

AI readiness assessment covers three independent dimensions: data maturity, infrastructure readiness, and organizational readiness
A single dimension at "Not Ready" (score 1) is a blocker regardless of overall score
Data quality must be measured empirically — self-reported quality is unreliable
Infrastructure readiness includes PHI-in-cloud policy, LLM API connectivity, AI gateway, and security review timeline
Organizational readiness covers team capacity, governance, and change management — not just executive sponsorship
The assessment produces a written report shared with the client — not an internal FDE document
Remediation plans must be specific before the assessment is finalized

AI Readiness Assessment#

Conceptual Explanation#

Core Architecture: The Assessment Framework#

Dimension 1 — Data Maturity#

Dimension 2 — Infrastructure Readiness#

Dimension 3 — Organizational Readiness#

Overall Readiness Score and Engagement Recommendation#

Architecture Diagram#

Common Mistakes#

Best Practices#

Trade-offs#

Interview Questions#

Q: A client wants to skip the readiness assessment and go straight to POC execution. How do you respond?#

Key Takeaways#

AI Readiness Assessment

Conceptual Explanation

Core Architecture: The Assessment Framework

Dimension 1 — Data Maturity

Dimension 2 — Infrastructure Readiness

Dimension 3 — Organizational Readiness

Overall Readiness Score and Engagement Recommendation

Architecture Diagram

Common Mistakes

Best Practices

Trade-offs

Interview Questions

Q: A client wants to skip the readiness assessment and go straight to POC execution. How do you respond?

Key Takeaways