Chapter 7: Case Studies: From Best-in-Class to Cautionary Tales

We will analyze real-world examples to illustrate the concepts discussed. Using the documentation you provided, we will perform a mock assessment of a security-conscious provider (like 'Factory.ai') to highlight strong security postures. Conversely, we will analyze provided examples of "bad" terms of service (like the 'Postman AI' example) to identify red flags and potential risks.

7.1 The Power of Real-World Examples

This chapter will make the theoretical concepts from the course concrete by applying them to real-life (or realistic) scenarios. We will take the Vendor Assessment Checklist from Chapter 6 and apply it to our case studies.

7.2 Case Study 1: The Gold Standard (Based on "Factory.ai")

Scenario: Introduce a fictional company, "SecureAI Corp," whose security posture is based on the best practices described in the Factory.ai documentation.

Applying the Checklist

• Data Governance: "SecureAI Corp explicitly states they do not use customer data for model training (Green). Their data is hosted in a single-tenant VPC (Green)."
• Infrastructure Security: "They are SOC 2, GDPR, and ISO 42001 certified (all Green). All data is encrypted with AES-256 at rest and TLS 1.2+ in transit (Green)."
• AI Safety: "Their platform includes features like DroidShield for real-time code analysis and requires manual approval for agent functionality (Green)."

7.3 Case Study 2: The Red Flags (Based on "Postman AI")

Scenario: Introduce a second fictional company, "VagueAI Inc.," whose terms of service are based on the "bad example" provided.

Applying the Checklist

• Data Governance: "VagueAI's terms state they may use user prompts to 'improve the service.' This is ambiguous and a potential risk (Red)."
• Data Retention: "Their policy does not specify a data retention period (Red)."
• Compliance: "They do not mention any specific compliance certifications (Red)."

7.4 Case Study 3: The Data Breach Scenario

Scenario: Create a fictional scenario of a company, "ChatCo," that used a seemingly secure LLM-powered chatbot. However, due to a combination of insecure output handling (LLM02) and excessive agency (LLM08), an attacker was able to craft a prompt that tricked the chatbot into exfiltrating customer data from an integrated CRM.

Root Cause Analysis

• Trace the attack chain, linking it back to the OWASP Top 10 for LLMs.
• Show how a proper vendor assessment and internal security controls could have prevented this breach.

Key Lesson: Even a secure vendor is not enough if your own application is not built securely. Security is a shared responsibility.

7.5 Key Takeaways and Lessons Learned