Introduction: The Double-Edged Sword of Generative AI

Generative AI is no longer the stuff of science fiction. From chatbots that can draft a marketing copy to image generators that design custom logos, its transformative potential is undeniable. But with great power comes great responsibility or, more accurately, monumental risk. As organisations rush to harness this technology, they must also grapple with the unique security challenges it presents. This is not just about safeguarding data but it’s about protecting trust, reputation, and compliance in an era where misuse can have far-reaching consequences.

This piece dives into the critical steps for securing generative AI applications, using a structured framework—the Generative AI Scoping Matrix—to help organisations navigate risks and implement tailored controls. Whether you’re a security architect, developer, or business leader, this guide offers actionable insights to keep your AI journey secure.

Scope 1: Consumer Applications—Guarding the Front Door

In Scope 1, employees use consumer-grade generative AI tools like public chatbots, image generators, etc under standard terms of service. Here’s the catch: these tools operate outside your organisation’s control. Data shared with them could be exposed to risks ranging from inadvertent leaks to malicious exploitation.

Key Controls:

Network-Based Protections: Use web proxies, egress firewalls, and cloud access security brokers (CASBs) to monitor and block traffic to unauthorised applications. However, beware of limitations—users can bypass these controls via external networks.

Host-Based Protections: Deploy endpoint detection and response (EDR) tools to block access to risky URLs and inspect outgoing data.

Policy Enforcement: Train employees on data handling policies and the risks of exposing sensitive information to public AI tools. Reinforce their role as the first line of defence.

The golden rule? If you can’t control the application’s backend, control what goes into it.

Scope 2: Enterprise Applications—Trust but Verify

Scope 2 involves enterprise-level agreements with generative AI providers. These applications often come with enhanced features and terms tailored for organisational use. While this setup offers more control than consumer apps, it also demands a deeper understanding of contractual obligations.

Key Questions to Ask Providers:

Is your data used for model training or improvement? Can you opt out?

Is your data shared with third parties?

Who owns the intellectual property of input and output data?

Will the provider indemnify you against IP infringement claims?

Additional Controls:

Leverage built-in access controls like role-based access control (RBAC) to restrict who can use specific features or view sensitive data.

Implement Data Loss Prevention (DLP) solutions to prevent unauthorised uploads of sensitive information.

Remember, trust is good, but verification is better. Review third-party audit reports and design documents to assess the provider’s security measures.

Scope 3: Pre-Trained Models—Balancing Flexibility and Risk

Here, your organisation builds applications using pre-trained foundation models (FMs) provided by third parties. While you control the application layer and user data, the FM itself remains in the provider’s domain.

Best Practices for Scope 3:

Identity Management: Enforce strong user authentication (e.g., OpenID Connect, OAuth 2) and multifactor authentication (MFA).

Access Control: Limit model inference endpoints to authorised users via IAM policies.

Data Minimisation: Only provide the FM with data it absolutely needs for processing—less is more when it comes to security.

Prompt Engineering: Prevent sensitive information from being included in prompts by redacting unnecessary fields at the retrieval stage.

Design patterns like Retrieval Augmented Generation (RAG) can enhance functionality while maintaining security by ensuring that only authorised users access specific data subsets.

Scope 4: Fine-Tuned Models—Custom Power Comes with Custom Risks

When you fine-tune a pre-trained model with proprietary data, you gain performance tailored to your needs but also inherit additional responsibilities. The fine-tuned model now contains weights derived from your data, making it a potential liability if mishandled.

Critical Considerations:

Avoid fine-tuning on volatile or sensitive data that may need deletion later, retraining models can be costly.

Encrypt fine-tuned model artifacts using customer-managed keys.

Implement strict access controls for both inference endpoints and fine-tuned models.

Fine-tuning allows unparalleled customisation but demands vigilance in protecting both the process and its outputs.

Scope 5: Self-Trained Models—The Road Less Travelled

This scope represents full ownership—you train a foundation model from scratch using vast datasets. While this approach offers maximum control, it also requires significant investment in resources and expertise.

Responsibilities Include:

Curating high-quality training data while filtering out toxic or biased content.

Implementing responsible AI practices across design, deployment, and monitoring phases.

Monitoring inputs and outputs during inference for abuse or policy violations using auxiliary ML models for content filtering or toxicity scoring.

For most organisations, Scope 5 is an outlier—a path reserved for those with highly specialised needs that justify its complexity and cost.

Cross-Scope Insights: Building Secure-by-Design Systems

Across all scopes, some universal principles apply:

Mitigate Prompt Injection Risks: Assume users can manipulate prompts to bypass safeguards. Design systems that limit the impact of such attacks by enforcing strict identity propagation and API-level access controls.

Monitor Activity: Use web application firewalls (WAFs) to prevent flooding attacks that could drive up costs or disrupt services.

Leverage Established Frameworks: Map controls to frameworks like MITRE ATLAS or NIST’s AI Risk Management Framework for comprehensive coverage.

Generative AI security isn’t about reinventing the wheel, it’s about adapting proven IT security practices to meet new challenges.

Conclusion

Generative AI offers immense promise but demands equal parts caution and creativity in its implementation. The stakes are high, and missteps can lead not only to financial losses but also to reputational damage that lingers far longer than any breach notification email.

As you integrate generative AI into your operations, remember this: security isn’t a onetime checklist, it’s an ongoing commitment. Stay informed, stay vigilant, and above all, stay adaptable. The technology will evolve and so must your defences.