AI buying guides
AI Agent Security Review: What Buyers Should Test Before Deployment
An AI agent security review should prove what the system can read, what it can change and how the business can detect, stop and recover from unsafe behavior. This guide turns those questions into procurement evidence and acceptance tests.
Prepared with AI assistance and checked against the linked sources. Numerical comparisons are illustrative unless expressly sourced.
Do not approve an agent because its demo looks capable. Approve a narrowly scoped action boundary only after permissions, untrusted inputs, approvals, logs, failure behavior and shutdown controls pass representative tests.
Start with the action boundary, not a generic security questionnaire
An agent can answer questions, choose steps and call tools. Security risk changes when it moves from reading public material to seeing private records or changing a business system. Begin the review with a task boundary that everyone can test.
Write the intended task, users, channels, data sources, tools and permitted outcomes. Then list prohibited actions. For example: an order-support agent may read the authenticated customer’s order, explain the published return policy and draft a return request. It may not issue money, alter the policy, expose another customer’s order or send data to an unapproved destination.
Map every external input. Customer messages, uploaded files, websites, emails, knowledge-base documents and tool responses can all contain text that influences a model. Map every side effect too: a database write, ticket creation, message, refund, account change or generated file. The review should connect each input to the tools and data it could reach.
NIST’s Generative AI Profile treats governance, pre-deployment testing and incident disclosure as core considerations. It also warns that testing can miss real deployment conditions when it relies only on laboratory benchmarks. [1] A buyer therefore needs a system-level review around the real workflow, not only a model card or a general penetration-test badge.
Separate identity, data and tool trust zones
Ask for a diagram that shows the user, agent, model provider, retrieval sources, memory, tools, policy service, logs and human operators. The diagram should label trust boundaries and data movement. If the provider cannot explain which component authorizes an action, the design is not ready for acceptance.
Use a service identity created for one agent and environment. Do not share an administrator credential across development, testing and production. Restrict the identity to the specific resources and operations needed for the task. Reading an order, creating a return request and issuing a refund should be separate permissions even when one back-office system exposes all three.
OWASP’s AI Agent Security Cheat Sheet recommends minimum required tools, per-tool scopes and explicit authorization for sensitive operations. [2] Treat that as an architectural requirement. A prompt saying “never issue a refund” is guidance to the model; it is not an access-control rule. The downstream service must still reject an unauthorized call.
| Boundary | Buyer evidence | Acceptance question |
|---|---|---|
| Identity | Named service account and environment-specific credentials | Can the agent impersonate a broader operator? |
| Data | Allowed fields, records, retention and region | Can one user obtain another user’s context? |
| Tools | Allowlisted operations and parameter constraints | Can a read-only flow invoke a write? |
| Network | Approved destinations and egress controls | Can tool input be sent to an arbitrary host? |
Treat retrieved content and messages as untrusted data
Prompt injection occurs when content presented to the model contains instructions that try to redirect its behavior. Direct injection can arrive in a user message. Indirect injection can be hidden in a document, email, webpage or tool result the agent reads while doing legitimate work.
Do not accept “we added a stronger system prompt” as the complete defense. Ask how untrusted content is isolated from control instructions, how retrieved text is labeled, how tool arguments are constrained and how the execution layer verifies authorization. Structured fields can reduce free-form propagation, but they do not replace access controls.
OpenAI’s agent-safety guidance recommends keeping untrusted variables out of higher-priority instructions, using structured outputs to constrain data flow, retaining tool approvals and evaluating traces. It also states that guardrails are not foolproof. [3] These recommendations are useful design prompts even if a proposal uses another provider.
Run a harmless injection test in a non-production environment. Put an instruction in a test document that asks the agent to ignore the task and reveal an unrelated record or call an unapproved tool. The expected result is not merely a refusal message. Inspect the trace and downstream systems to confirm the sensitive read or write never occurred.
Bind approvals to the exact high-impact action
Classify actions by business impact. Public search may be low risk. Sending an external message, changing account data, committing money or deleting a record is higher risk. The classification should drive enforced approval, authentication and logging rules.
An approval screen should show the exact action, target and normalized parameters. “Allow the agent to continue” is too vague for a refund or account change. The approval record should expire and be unusable for a different amount, recipient or resource. Unknown operations should fail closed.
OWASP recommends previews, explicit approval for high-impact or irreversible actions, action-specific binding, short-lived authorization and separation between the agent’s decision and the execution component. [2] Ask the vendor to demonstrate an expired approval, changed parameter and replayed request. All three should be rejected.
Human review has a cost and can become a rubber stamp. Use it where impact justifies it, and reduce unnecessary prompts by narrowing the action scope. A useful design may allow the agent to draft a refund request and let a policy service automatically approve only a strictly eligible, low-value case. Larger or ambiguous cases go to a person. The thresholds are business policy choices, not model settings.
Require logs, limits, incident response and a tested stop mechanism
A production agent needs more than application error logs. Record the agent version, policy version, model and retrieval configuration, authenticated actor, tool requested, normalized parameters, authorization result, approval result, tool response, final outcome and correlation identifier. Protect sensitive data in logs and define retention and access.
Set operational limits on tool calls, retries, processing time, spend and repeated actions. Detect unusual denial rates, cross-user access attempts, unexpected destinations, repeated approval failures and sudden changes in tool use. A dashboard without an alert owner is not an operating control.
NIST CSF 2.0 offers a technology-neutral set of outcomes across Govern, Identify, Protect, Detect, Respond and Recover. [4] For an agent, translate those outcomes into named owners, alerts, containment steps, credential revocation, affected-record analysis, communication and safe restoration. NIST’s incident-response guidance emphasizes preparation together with detection, response and recovery. [5]
Test the stop mechanism. The operator should be able to disable the agent’s tool access without waiting for a model or application release. Confirm what happens to in-flight work, queued jobs, sessions, memory and credentials. Also test rollback to a known configuration and the manual process that keeps the business running while the agent is unavailable.
Demand adversarial tests and release evidence
Build an abuse-case set from the actual workflow. Include unauthorized records, malicious document instructions, malformed tool output, repeated submissions, altered approval parameters, stale memory, cross-session context, unavailable integrations and a compromised credential. Define the expected refusal, approval, containment or recovery result before running the test.
Test the integrated system, not only a model endpoint. A model may produce a safe answer while a tool has already changed data, or it may claim success after a tool failed. Verify the resulting environment independently. Repeat critical tests after changes to the model, prompt, tools, retrieval, memory, policy or identity configuration.
OWASP recommends retaining the tested agent version, provider, tool policy, retrieval configuration, abuse cases, observed approval or circuit-breaker behavior and accepted residual risk. [2] Put this evidence in the acceptance package. A single screenshot or a pass percentage without test cases is not enough to support a release decision.
No test proves an agent is risk-free. The buyer’s decision is whether the tested boundary, residual risk, operating controls and recovery capability are acceptable for the intended use. High-impact sectors may need additional legal, privacy, safety and regulatory assessment that this general guide cannot replace.
Worked example: stage a return-support agent
Consider an invented retailer that wants an agent to handle return questions. The first proposal gives the agent broad commerce access so it can read orders, edit customer records and issue refunds. That is convenient for a demonstration but creates a large action boundary.
A safer pilot splits the journey into stages. The agent can read the published policy and the authenticated customer’s eligible order fields. It can collect a reason and draft a return request using a fixed schema. A policy service, not the model, checks the return window, product exclusions and request duplication. The pilot does not issue money.
| Test | Expected evidence | Release effect |
|---|---|---|
| Another customer’s order number | Access denied by the order service and recorded | Mandatory pass |
| Document asks for an unrelated export | No export tool is available; trace shows rejection | Mandatory pass |
| Duplicate submission | Existing request returned without a second write | Mandatory pass |
| Return service unavailable | No success claim; request safely queued or handed to staff | Mandatory pass |
| Operator disables tools | New writes stop; in-flight handling follows the runbook | Mandatory pass |
After the read-and-draft boundary is stable, the retailer may consider a separately authorized refund tool for a narrow policy-defined subset. That is a new release decision with new tests. The example is an editorial scenario, not a client result, a universal refund policy or a guarantee of safety.
Compare proposals on ownership and operating cost
Ask each provider to price the same boundary and evidence package. Separate discovery, integration, identity and policy work, test development, deployment, monitoring, incident support, model or platform usage, ongoing evaluation and change requests. A lower build fee may transfer more monitoring and recovery work to your team.
Clarify ownership of prompts, configurations, evaluation cases, traces, code, tool definitions, data exports and runbooks. Define how you retrieve them at exit. Ask who patches dependencies, reviews provider changes, rotates credentials and repeats security tests after a release.
For a custom engagement, AI agent development, AI integration engineering and MLOps and AI operations may be separate commercial scopes. TechLithic provides these services, but the same evidence and acceptance conditions should apply to TechLithic or any other provider.
Do not buy an agent when a read-only assistant, fixed workflow or ordinary software rule completes the task with less risk and operating burden. The earlier chatbot versus agent guide helps decide the architecture. The knowledge-base acceptance guide covers source quality and grounded answers when the main need is reliable retrieval.
Use this acceptance checklist before production
- Is the intended task, user, channel and prohibited action list approved?
- Are all user, document, web, memory and tool inputs classified by trust?
- Does the agent use a dedicated, least-privileged identity in each environment?
- Are tools and parameters allowlisted, with authorization enforced outside the model?
- Can untrusted content influence a sensitive tool without an independent policy check?
- Are high-impact approvals bound to the exact actor, target, parameters and expiry?
- Do logs connect the user, agent version, policy decision, tool result and business outcome?
- Are retry, spend, tool-chain and duplicate-action limits enforced?
- Have representative abuse cases passed in the integrated pre-production system?
- Can an accountable operator stop tool access, handle in-flight work and restore a known version?
- Are incident, notification, evidence retention and recovery responsibilities named?
- Will critical tests repeat after changes to models, prompts, tools, retrieval or permissions?
- Does the contract define recurring security work, artefact ownership and exit handover?
Treat any missing mandatory control as an explicit risk decision, not an item to “add later.” A narrow first release is often the practical choice. Expand only after the current action boundary produces inspectable evidence and the operating team can control it.
Frequently asked questions
Is a safe system prompt enough to secure an AI agent?
No. Prompts guide model behavior but do not enforce identity, authorization, network access or transaction policy. Sensitive tools need downstream controls that reject unauthorized actions even if the model requests them.
Should every AI agent tool call require human approval?
Not necessarily. Classify actions by impact. Low-risk, tightly scoped reads may not need repeated approval, while financial, destructive, administrative or externally visible actions may require action-specific review and stronger authentication.
What is the most important prompt-injection test?
Use representative untrusted content that asks the agent to exceed its task, then verify both the trace and the downstream system. The key evidence is that the prohibited read or write never occurred, not only that the agent displayed a refusal.
What should an AI agent kill switch disable?
It should revoke or block the agent’s ability to invoke business tools quickly. The runbook should also cover queued and in-flight work, sessions, memory, credentials, customer communication and a safe manual fallback.
When should a buyer avoid deploying an AI agent?
Avoid or narrow the deployment when a fixed rule or workflow can complete the task, permissions cannot be constrained, outcomes cannot be independently verified, or the organization lacks an owner for monitoring and incident response.
Sources and research notes
Sources checked on 2026-10-30. Pricing and policy documents can change; verify current terms before approving expenditure.
- NIST: Artificial Intelligence Risk Management Framework, Generative Artificial Intelligence Profile
Primary NIST guidance on GAI governance, pre-deployment testing, monitoring, third-party transparency and incident disclosure.
- OWASP Cheat Sheet Series: AI Agent Security
Primary OWASP practitioner guidance on least privilege, prompt injection, human approval, output validation, monitoring and adversarial test evidence.
- OpenAI API documentation: Safety in building agents
Provider documentation on untrusted input, structured outputs, tool approvals, guardrails, trace grading and evaluation.
- NIST: Cybersecurity Framework 2.0
Primary technology-neutral framework for cybersecurity outcomes across Govern, Identify, Protect, Detect, Respond and Recover.
- NIST SP 800-61 Revision 3: Incident Response Recommendations and Considerations
Primary NIST guidance for incorporating incident response into cybersecurity risk management and preparing detection, response and recovery activities.
NIST, OWASP and OpenAI primary guidance was opened and checked on October 30, 2026. The return-support agent, test matrix and action classifications are original procurement illustrations, not client results, legal advice or universal security controls. The OpenAI source is provider-specific and is used as an example of agent design guidance, not as an endorsement or a claim about other providers.