The Limits of Autonomous SOC: Five Risks Teams Should Model
Autonomy is a spectrum. A mature SOC models five real failure modes, from tool misuse to evidence loss, before granting an agent access to production.
The Limits of Autonomous SOC: Five Risks Teams Should Model
Autonomy is a spectrum. A mature SOC models failure modes before granting an agent access to production systems or customer data.
Tool misuse
A manipulated instruction can make an authorised agent call the wrong tool. Restrict tools by role, scope, and approval.
Context errors
Incomplete telemetry or stale identity data can produce a confident but wrong conclusion. Show missing context and fail safely.
Evidence loss
Automated cleanup can remove the very data needed for forensics. Preserve evidence before remediation.
Privilege creep
Agents accumulate access as integrations grow. Review permissions and rotate credentials like any other service identity.
Silent drift
A workflow can change as data, models, and rules change. Keep regression cases and review outcomes.
Where an agent's autonomy stops
AI-Assisted SOC vs Autonomous SOC covers where each action sits on the autonomy scale. This register rests on one recommended pattern: steps that change production state (isolating a host, disabling an account, blocking an address) wait for a person, because a wrong call lands on a real user and the evidence behind it may be text the attacker wrote.
In zcrSOAR each playbook step is Autonomous (runs at once), Approval (waits for a role-gated sign-off) or Blocked (never runs until the gating changes). Setting the gating is customer configuration. Recommended pattern: where possible, route an agent's actions through gated playbook steps, start new steps at Approval or Blocked, and promote one to Autonomous only if it is reversible, has run under approval for a review period with no approval later reversed, and has a tested undo path (the bar used in the post linked above).
A risk register for an agent that calls tools
Rows 1 to 4 map to the risks above, row 5 is an addition, and silent drift is covered by question 4 below. Codes are those of the 2025 OWASP entries in Sources; the scenarios in column 2 and the choice of rows are ours, not OWASP text or ranking.
| Risk (OWASP LLM 2025) | In a SOC | Design control | Pre-go-live test |
|---|
| Tool misuse: prompt injection (LLM01) | Attacker-written text (email body, User-Agent field, ticket comment) tells the agent to close the case, skip an approval or act on a host it names. | Recommended pattern: untrusted content in a separate field, targets only from the incident's entities, fewest possible tools. Documented in zcrSOAR: Approval and Blocked gating on playbook steps, which decides whether a step runs, not what the instruction says. | Plant instruction-like text in test emails, log fields and comments, one naming a host outside the incident. Pass: same tool choice, no out-of-scope target, no skipped approval. |
| Tool misuse: improper output handling (LLM05) | Model output is passed on unchecked: a wrong time range in a query, a playbook parameter naming a host not on the incident. | Recommended pattern: treat model output as untrusted user input, validate it against a schema, allow-list targets. | Feed malformed or out-of-scope output (extra fields, wildcard host, no time limit). Pass: rejected before any tool runs, and logged. |
| Privilege creep, evidence loss: excessive agency (LLM06) | A connector added for one playbook also holds delete or account-reset rights, so a cleanup step can destroy evidence. | Documented in zcrSOAR: per-step gating. Recommended pattern: no delete actions in the tool set, authorisation enforced downstream and not by the prompt, evidence kept before containment. | In staging, call every tool as the agent's identity against targets that should be refused. Pass: Blocked never runs, Approval holds without side effects. Diff connector scopes against the tool list each release. |
| Context errors: misinformation (LLM09) | A confident conclusion built on gaps: "no lateral movement" while the log source that would show it is silent. | Recommended pattern: each statement cites an event ID, "not observed" stays apart from "not collected", no clean verdict while a source is silent. | Replay a past incident with one log source removed. Pass: the agent reports the gap and gives no clean verdict. |
| Alert floods: unbounded consumption (LLM10) | A flood, or events an attacker generates on purpose, sends the agent into retry loops while real cases wait unseen. | Recommended pattern: per-case caps on tool and model calls, timeouts, a retry limit, a circuit breaker that hands cases to a person. Customer configuration: caps and thresholds. | Replay a synthetic alert burst against a tool that fails every call. Pass: retries stop at the cap, the breaker trips, the rest reach a human queue. |
Go/no-go questions before production
- Can you list every tool the agent can call, with its identity, scope and purpose?
- Does every step that changes production state sit at Approval or Blocked, with the approver role named?
- Can a named person stop the agent in one step, with open cases handed to a human queue?
- After each change to model, prompt, playbook or connector, do you rerun a fixed replay set, and do you know how to roll back?
Failure modes and escalation triggers
Recommended escalation triggers (thresholds are customer configuration): stop the agent and hand the case to a person when the target is not on the incident, a tool call is refused more than a set number of times, a required log source is quiet, the conclusion contradicts the analyst's, or the queue passes your threshold. An approver who clears nearly every request within seconds may have stopped reviewing, so sample approvals weekly (recommended pattern).
Recommended pattern: keep evidence where the agent cannot delete it before containment, and rerun the replay set after every change before returning steps to Autonomous, since drift shows up as a changed result on a case that used to pass. Documented in zcrSOAR: playbook version history with rollback. Passing these tests shows the controls hold for the cases you wrote, not that the agent is safe.
Frequently asked questions
Can prompt injection be filtered out of a SOC agent?
Not with certainty. OWASP says it is unclear whether any method fully prevents prompt injection. Filtering is one mitigation among several, alongside least privilege, human approval of high-risk actions and segregating untrusted content, so design for a successful injection to reach only narrow, gated tools.
Where do zcr's own AI features sit in this register?
In zcrCTEM, a ticket or takedown drafted by AI stays a proposal until an analyst approves or rejects it (the approval idea in row 3). A playbook drafted by the zcrSOAR AI assistant is model output for review (row 2), and unapproved playbooks stay in Draft, Test, Dry-run or Simulation Mode until production is authorised. zcrSIEM does not include SOAR, so approvals and response actions live in zcrSOAR.
If we buy zcrSOC, do these questions go away?
No, they move. zcrSOC is a managed service delivered through an approved channel partner: Essential covers alert acknowledgement and triage; Response adds coordinated response and escalation with zcrSOAR. Coverage hours are set in the SLA and statement of work, so put the go/no-go questions above to the partner.
Sources
OWASP Gen AI Security Project, accessed 2026-09-28 (last-modified dates):
Continue: AI-Assisted SOC vs Autonomous SOC, Human-in-the-Loop SOC and the AI SOC Agent guide
Explore zcrSOC · Compare pricing · Contact zcr
If you'd rather talk through your own environment, here's how to reach us.
