AI Security Fundamentals#

Common Mistakes#

1. Treating LLM security as identical to traditional injection defense. SQL injection defenses (parameterized queries) do not translate to prompt injection. LLMs process natural language, not structured queries; input sanitization alone does not prevent prompt injection.

2. No threat model before deployment. AI capabilities are deployed without a systematic threat assessment, leaving significant risks unaddressed. Conduct a structured threat modeling session (STRIDE or an AI-adapted equivalent) before any clinical AI capability goes to production.

3. Assuming the AI provider handles all security. LLM providers handle inference-layer security. The enterprise is responsible for authentication, authorization, input validation, output validation, PHI handling, and audit logging. These are not provided by the LLM API.

Key Takeaways#

• AI systems introduce four threat categories absent from traditional models: prompt injection, data exfiltration, agent privilege escalation, and model-specific DoS
• Indirect prompt injection (via RAG-retrieved documents) is harder to detect and prevent than direct injection
• Defense-in-depth for AI requires controls at six layers: perimeter, AI gateway, orchestration, inference, and data
• Threat models for AI systems must be maintained as living documents and updated when capabilities change
• The enterprise is responsible for input validation, output validation, PHI handling, and audit logging — the LLM provider handles inference-layer security