Audit and Logging for AI Systems
Common Mistakes
1. Logging request and response content for PHI-handling AI features. Even "debug" logs that include prompt content contain PHI for clinical AI features. Every log store that receives AI request/response content becomes a HIPAA data store with full compliance implications.
2. Not setting log retention to 6 years. Default CloudWatch log retention is configurable; the default is often 30–90 days. HIPAA requires 6-year retention for audit logs. Set explicit retention policies on creation and audit them quarterly.
3. Mixing PHI audit logs with operational logs. Operational logs have different retention requirements, different access control requirements, and different compliance implications. Keep them separate.
4. No integrity checking on audit logs. An attacker who can modify the audit log can cover their tracks. Store integrity hashes (SHA-256 of each log entry) separately from the log entries, or use cloud-native tamper-evident logging (AWS CloudTrail with log file integrity validation).
Key Takeaways
- HIPAA audit logs must include patientid and userid but must never include PHI content, prompt text, or AI response text
- PHI audit logs must be immutable (write-once), encrypted, and retained for 6 years
- Operational logs and HIPAA audit logs must be in separate stores with separate access controls
- AI quality logs capture model-specific metadata (model version, retrieval scores, citation counts, clinician feedback) that traditional observability systems do not
- Integrity hashing of audit log entries enables detection of tampered audit trails