45:15
Threat Modeling for AI Systems Adrien O Hana
OWASP Foundation
Overview
This video introduces threat modeling specifically for AI systems, emphasizing its importance in identifying and mitigating potential risks. It contrasts traditional threat modeling with the unique challenges posed by AI, such as non-deterministic behavior and prompt injection vulnerabilities. The presentation covers recent AI-related security incidents, explains fundamental AI flaws like the inability to distinguish system from user instructions, and details a practical threat modeling approach using data flow diagrams and the STRIDE framework. It also discusses various mitigation strategies, including guardrails, access controls, dependency management, and the critical need for robust logging and monitoring in AI development and deployment.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- Threat modeling is a process to identify, communicate, and understand threats and their mitigations within a system.
- AI systems introduce new complexities due to their non-deterministic nature and unique vulnerabilities like prompt injection.
- Recent headlines highlight the potential dangers of AI systems, but these are often evolutions of capability rather than revolutionary threats.
- The barrier to entry for attackers is lowering, and the internet is increasingly 'vulnerable-coded,' necessitating proactive security measures.
Understanding the fundamentals of threat modeling and the specific risks associated with AI is crucial for building secure and reliable AI systems before they are deployed.
The Hugging Face incident, where a 'sandbox' was not truly secure, illustrates how a lack of threat modeling can lead to unexpected breaches.
- Large Language Models (LLMs) struggle to differentiate between system instructions and user instructions, forming the basis of prompt injection.
- Prompt injection can lead to unintended actions, data exfiltration, or system compromise.
- Indirect prompt injection, where malicious instructions come from external data sources (like web pages), is particularly dangerous.
- Other AI risks include hallucinations, data leakage, brand damage, and goal hijacking in agentic systems.
Recognizing the core flaws in LLMs, such as the inability to distinguish instruction types, is essential for designing effective defenses against prompt injection and similar attacks.
An LLM might be instructed to ignore previous instructions and reveal sensitive system prompt information, demonstrating a direct prompt injection attack.
- Guardrails, both rule-based (e.g., pattern matching) and model-based (e.g., classifiers), are used to detect and prevent prompt injection and other AI risks.
- A layered defense strategy is necessary, as no single solution provides complete protection.
- Hyperscaler APIs and frontier model providers often include built-in guardrails for common risks like PII detection and hallucination.
- While guardrails are important, they are not sufficient on their own and must be supplemented with other security controls.
Employing a multi-faceted defense strategy, including guardrails and other controls, is vital to manage the inherent risks of AI systems.
A system might use guardrails to sanitize user input, detect jailbreak attempts, and ensure the chatbot's output doesn't contain offensive language or violate company policy.
- Existing frameworks like OWASP Top 10 for LLMs and Agentic Applications provide valuable starting points but may not cover all system specifics.
- The NIST AI Risk Management Framework (AI RMF) offers comprehensive guidance but lacks clarity on practical implementation.
- A practical approach involves drawing data flow diagrams (DFDs) to visualize system components and trust boundaries.
- Applying the STRIDE (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) framework to DFDs helps identify potential threats at each boundary.
Combining established threat modeling methodologies with AI-specific considerations is key to systematically identifying vulnerabilities in complex AI systems.
Using STRIDE on a chatbot's data flow diagram can reveal how an LLM might be tricked into reading from the wrong database (spoofing) or citing poisoned information (tampering).
- Top threats include prompt injection (visual or text-based), data access control failures, supply chain risks from dependencies, external reference vulnerabilities, and insecure code generation.
- Mitigations involve input sanitization, strict data access controls, managing third-party dependencies carefully, treating external data as untrusted, and validating AI-generated code.
- Visual prompt injection can occur when hidden instructions within images manipulate LLM behavior.
- Ensuring proper access control to private data is critical, as seen in incidents where internal chatbots leaked sensitive information.
Addressing specific, common threats with targeted mitigation strategies is essential for securing AI applications against known attack vectors.
A visual injection might cause a multimodal system to follow instructions embedded in an image (e.g., 'say hello') rather than the user's explicit request ('describe this image').
- The 'lethal trifecta' (private data, untrusted content, data exfiltration capability) is a common risk in AI development tools like GitHub Copilot or Cursor.
- Implementing least privilege principles and deterministic controls for AI agents is crucial, rather than relying solely on prompt-based policies.
- Robust logging and monitoring are vital for detecting and responding to incidents, even if the AI system itself doesn't immediately flag an issue.
- The goal for detection loops should be within the timeframe of a typical AI reasoning chain (around 30 minutes) to enable timely response.
Continuous monitoring, secure development practices, and understanding the combined risks of AI tools are necessary to prevent and quickly respond to security incidents.
Even if an AI agent has permission to access a repository, logs should reveal if it attempts to access sensitive files it shouldn't, triggering an alert.
Key takeaways
- Threat modeling is a proactive and essential process for identifying and mitigating risks in AI systems.
- The fundamental flaw in LLMs, the inability to distinguish system from user instructions, makes prompt injection a persistent threat.
- A layered defense strategy combining guardrails, access controls, and vigilant monitoring is more effective than relying on single solutions.
- Data flow diagrams combined with the STRIDE framework provide a structured method for analyzing AI system vulnerabilities.
- Supply chain risks and the secure handling of AI-generated code are critical considerations in modern software development.
- Robust logging and monitoring are indispensable for detecting and responding to AI security incidents in a timely manner.
- Least privilege and deterministic controls are more reliable for securing AI agents than prompt-based instructions alone.
Key terms
Threat ModelingAI SystemsPrompt InjectionLLM (Large Language Model)GuardrailsSTRIDE FrameworkData Flow Diagram (DFD)Non-deterministic BehaviorAgentic SystemsLeast PrivilegeSupply Chain AttacksInformation DisclosureDenial of Service (DoS)Goal Hijacking
Test your understanding
- What is the fundamental flaw in LLMs that enables prompt injection attacks?
- How does the STRIDE framework help in threat modeling AI systems, and what are its components?
- Why is a layered defense strategy considered more effective than a single security measure for AI systems?
- What are the key components of a data flow diagram used in threat modeling, and why are trust boundaries important?
- How can organizations mitigate the risks associated with AI development tools that exhibit the 'lethal trifecta'?