Understanding AI Risk Management – Securing Cloud Services with OWASP LLM Top 10

7. Insecure Plugin Design

LLM plugins can have insecure inputs and insufficient access control. This lack of application control makes them easier to exploit and can result in consequences like remote code execution.

Plugins are essentially a tool belt of processes that can be invoked by a model using agents, often transparently to the user, that help respond to complex queries more accurately. 

Potential Impact:

Plugins can be exploited to extend the reach of a malicious attack. Any plugin accessible by a model agent is susceptible to mis-use — opening you up to remote code execution, data leakage, or privilege escalation. 

Mitigations:

1. Parameterize inputs to plugins as much as possible — similar to protecting against SQLi and XSS.
2. Sanitize and validate any input/output provided to/from a plugin —  especially, if third-party. This can be done manually in the agent used to invoke your model or in Amazon Bedrock Agents using Advanced Prompt Parsing. Follow OWASP ASVS guidelines.

8. Excessive Agency

LLM-based systems may undertake actions leading to unintended consequences. The issue arises from excessive functionality, permissions, or autonomy granted to the LLM-based systems.

The entire value proposition of using AI to streamline production with minimal human interaction. The more you want to off-load to the AI, the more access it requires. This is a balancing act and will certainly become more complicated and more difficult to control as the capabilities of AI grow in the coming years.

Potential Impact:

Excessive agency can result in a wide range of impacts depending on how the AI implementation — more specifically what tools and data sets it has access to.

Mitigations:

Incorporate least privilege at all layers.

• Only grant access to RAG systems or Knowledge Bases required for the specific process that is calling it.
• Do not rely on your LLM for authorization, enforce authentication and authorization on all APIs accessed by LLM — third-party and internal.
• Avoid open-ended/general purpose functions. 
• In Amazon Lex, avoid relying on the FallbackIntent. Codify all approved intents directly. 
• When using Agents for Bedrock, create dedicated API schemas for each Action Group. Create dedicated Lambda functions with dedicated roles for each Action Group.
• Use adequate descriptions in your API schema so your agent can properly identify the proper method to pass the lambda function. 
• Only add tools and plugins that are necessary and 
• Limit what tools your LLM can access. This includes specifying only the required Lambda functions in your Agents service role. Your lambda function should also include execution logic for specific paths. Consider the following example from this AWS blog post:

def lambda_handler(event, context):
    responses = []

    for prediction in event[‘actionGroups’]:
        response_code = …
        action = prediction[‘actionGroup’]        api_path = prediction[‘apiPath’]

        if api_path == ‘/claims’:
            body = open_claims()
        elif api_path == ‘/claims/{claimId}/identify-missing-documents’:
   parameters = prediction[‘parameters’]            body = identify_missing_documents(parameters)
        elif api_path == ‘/send-reminders’:
            body =  send_reminders()
        else:
            body = {“{}::{} is not a valid api, try another one.”.format(action, api_path)}

9. Over-reliance

Systems or people overly depending on LLMs without oversight may face misinformation, miscommunication, legal issues, and security vulnerabilities due to incorrect or inappropriate content generated by LLMs.

Everyone who has played around with GenAI in any capacity has run into times when the AI was just flat out wrong. It does not have the information, tools, or context to provide the correct response so it just makes one up called “hallucinations”. It’s important to provide sanity checks on outputs — This differs from insecure output handling which is mainly about properly typing your outputs and making sure they do not introduce exploits down stream. Over-reliance is focused on validating the actual content generated by the AI.

Potential Impact:

Overly relying on AI increases the likelihood that hallucinations will affect your company or your users. This can include code generated by an AI that is automatically executed or natural language responses being fed to downstream processes in the AI or to users. This is why it is so important for AIs to maintain a “human-in-the-loop” for sanity checks during its use and development.

Mitigations:

1. Regularly review model outputs for consistency and accuracy.
2. Test prompts multiple times to ensure the prompt always gives you the desired response. 
3. When using RAG, use similarity filters.
4. If using Amazon Kendra, add metadata to your documents and tune your searches with metadata weighting and relevancy parameters. Side note: can’t recommend this enough — do not just crawl your docs blindly.
5. If using Amazon Kendra, enrich your documents by removing unnecessary content or adding new content.
6. Always look to optimize your model and app — fine-tuning, prompt engineering, adversarial testing, updating vector databases and external document stores.
7. Break down complex tasks into individual chains or agents. 
8. Always make the user aware they are interacting with an AI — Transparency is core of AI Ethical Use.
9. Optimize randomness parameters like Temperature, Top P, and Top K.
10. Enabling Guardrails for Bedrock can minimize the risk your AI generates inappropriate content. (You do not want to rely on your AI to do this by itself.)
11. Add checks against trusted external sources for LLM outputs.

10. Model Theft

This involves unauthorized access, copying, or exfiltration of proprietary LLM models. The impact includes economic losses, compromised competitive advantage, and potential access to sensitive information.

Protect your Intellectual Property! Model theft is not just ensuring the actual code of your model is kept safe; it also entails sending a high volume of carefully crafted prompts used by an attacker to partially reverse engineer your model — e.g., weights and parameters used, training data sources, foundation models etc.

Potential Impact:

If a competitor gets access to your model, it can potentially put you out of business by either offering the same product for cheaper or impersonating you in some way and cause reputational damage.

Mitigations:

1. Use a centralized ML Model Inventory or Registry for ML models. AWS Sagemaker has a model registry that enforces access control.
2. Develop and deploy models solely using ML Ops Pipelines. This includes training data, testing data, and RAG Source Data. This helps limit the identities that require access. 
3. Apply resource policies on your model registry to prevent access from unauthorized accounts. Monitor updates to these resource policies.
4. If using fine-tuned models Amazon Bedrock, put bucket policies on the training data to ensure only authorized individuals and processes are allowed. Monitor updates to these bucket policies.
5.  Implement a watermark.
6. Monitor query frequency of individual users and alert on anomalies. If using Amazon Lex, make sure you are receiving the userId in the logs. If you are receiving it, it may require enriching with third party API’s when alerting— e.g., userID = 383j10d for an actual user [email protected].
7. Enable Amazon CloudTrail Data Events for Read events on your S3 buckets with proprietary data — training, RAG sources, etc. Ingest these events into your SIEM to identify authorized access.
8. Use Service Control Policies to ensure that Amazon Bedrock is only used in approved accounts in your organization.