AI as target
Manipulate the AI-enabled system
Attack data, weights, prompts, retrieval, memory, tools, identities, infrastructure, or users to compromise confidentiality, integrity, availability, privacy, safety, or authorized behavior.
AI as the target
The attack chain against models, data, context, tools, and memory
“AI kill chain” is not a single standardized term. In AI security, it is best treated as a family of models for how an adversary gains knowledge, places malicious influence, redirects model-mediated behavior, persists that influence, and causes an unacceptable external effect.
Recon → Poison → Hijack → Persist → ImpactDefensive analysisNo payloads or exploit execution
Analytical boundaries
Three uses of “AI kill chain” must remain separate.
An AI system can be the target of an attack, an AI model can accelerate a conventional cyber campaign, or AI can assist a military decision-and-engagement chain. Similar language does not make these the same threat model.
AI as offensive enabler
Accelerate a conventional campaign
Use models for reconnaissance, social engineering, vulnerability research, automation, or adaptation against ordinary networks and applications.
AI-enabled military chain
Assist sensing through assessment
Apply AI to ISR, fusion, classification, planning, guidance, or assessment. The legal and human-control questions are different from AI application security.
NVIDIA-style application chain
Five stages and an agentic loop
The stages are an operational narrative. MITRE ATLAS supplies detailed adversary behaviors, NIST supplies lifecycle and attacker-assumption terminology, and OWASP supplies application-level risks.
01 · Map the trust boundary
Recon
A synthetic adversary probes exposed behaviors, data routes, tools, and error messages.
The attack surface includes every document, tool result, memory item, model endpoint, and integration that can enter the model context.
Observable transition
High-frequency variations, component leakage, and unusual tool-enumeration attempts.
Defender question
What can an external party learn without crossing an access-control boundary?
02 · Place malicious data
Poison
A synthetic untrusted document enters a retrieval corpus with instruction-like content.
“Poison” is broader than training-data poisoning: it can include prompts, external pages, tool metadata, shared memory, or retrieved documents.
Observable transition
A low-trust document crosses an ingestion boundary and receives an embedding.
Defender question
Was provenance preserved, and can untrusted content be treated as data rather than authority?
03 · Redirect model-mediated behavior
Hijack
The synthetic model treats retrieved text as an instruction and proposes an unauthorized tool action.
The security boundary must not depend on a probabilistic model correctly distinguishing instruction from data on every input.
Observable transition
A tool request conflicts with the original user task and references a low-trust source.
Defender question
Does an external policy engine validate the requested action against user intent and authorization?
04 · Write poisoned state
Persist
The synthetic agent attempts to store the instruction in shared memory for later sessions.
Persistence can live in memory, retrieval indexes, summaries, tool configuration, or agent plans—not only in model weights.
Observable transition
A model-originated write targets a shared memory namespace.
Defender question
Who may write shared state, what lineage is retained, and can the state be rolled back?
05 · External state change
Impact
A synthetic tool call would create an unauthorized external action, but the simulator contains it.
Autonomy is an impact multiplier. A read-only assistant may produce a bad answer; an over-privileged agent can change systems, communicate, or expose data.
Observable transition
Blocked egress, denied tool scope, and a complete trace from source document to proposed action.
Defender question
What maximum authority was reachable, and which deterministic control contained it?
Why agentic systems change impact
An output becomes an action.
A standalone model normally produces text, an image, or a score. An agent may read private data, retain memory, send messages, call APIs, execute code in a sandbox, move money, or alter an operational state.
Context is a trust boundary
Every email, webpage, document, image, database row, tool result, and memory item that can enter the model context may influence behavior. Retrieved content must be treated as untrusted data, not authority.
Tools multiply the blast radius
A prompt-injection failure becomes consequential when the model can reach secrets, broad credentials, external communications, destructive operations, or cross-tenant resources.
Memory creates persistence
A temporary hijack can survive when attacker-controlled content is written into session history, user memory, shared retrieval stores, summaries, or agent plans.
Feedback creates iterate/pivot behavior
A compromised agent can discover new tools, retrieve updated instructions, write state, and propagate to connected workflows. Monitoring must correlate sequences, not only isolated prompts.
Lifecycle defense
There is no single “AI firewall.”
Prevention, detection, containment, evidence, and rollback must cover data collection, training, supply chain, serving, retrieval, prompts, tools, memory, updates, and operations.
| Layer | Preventive controls | Detection and evidence | Recovery |
|---|---|---|---|
| Data and retrieval | Source allowlists, hashes, lineage, tenant ACLs, deduplication, contribution limits | Source anomalies, embedding clusters, unusual contributor concentration, instruction-like content | Quarantine sources, rebuild indexes, retrain from known-good manifests |
| Model and software supply chain | Signed artifacts, pinned dependencies, safe serialization, isolated builds, SBOM and AI-BOM | Hash mismatch, unexpected package source, code execution during load, unexplained model changes | Block promotion, remove compromised versions, rebuild and rotate credentials |
| Serving and API | Authentication, quotas, output minimization, tenant isolation, cost and rate limits | Boundary probing, adaptive query sequences, account rotation, systematic mutation, cost spikes | Throttle or suspend, preserve evidence, change exposed output, rotate keys or models |
| Context and prompts | Provenance labels, context minimization, trusted/untrusted separation, no unnecessary secrets | Source mismatch, hidden instructions, conflict with original user intent, low-trust retrieval into sensitive tasks | Remove poisoned content, invalidate caches and memories, replay traces to determine exposure |
| Tools and actions | Narrow credentials, schema validation, deterministic policy engine, sandboxing, egress allowlists, approval gates | Tool calls inconsistent with user intent, privilege escalation, new destinations, cross-tenant actions | Stop execution, revoke tokens, isolate sandboxes, roll back transactions where possible |
| Memory and updates | Explicit write policy, provenance, expiry, user controls, staged releases, canaries | Instruction-bearing memories, cross-user retrieval, behavior drift, post-update tool anomalies | Delete or quarantine state, restore prior versions, roll back model, prompt, policy, and index |
Control hierarchy
Contain consequences when recognition fails.
Content classifiers and prompt filters are probabilistic and can be bypassed. The strongest controls do not require perfect recognition of every poison or injection.
Reduce reachable authority
A summarizer should not possess payment, administrative, destructive, or unrestricted communications credentials.
Separate information domains
Keep untrusted material away from secrets and privileged instructions unless the task strictly requires the combination.
Authorize outside the model
Use deterministic policy, object-level access control, transaction limits, and independent approval for consequential actions.
Version and roll back every state
Models, prompts, policies, memories, indexes, tools, credentials, and dependencies all need an emergency recovery path.