Agent-SIREN

Code for Detecting Harmful Agent Trajectories from LLM Internal Representations.

We focus on two forms of evidence for judging a trajectory unsafe: harmful content, which is expressed directly, and unsafe tool use, which turns on whether an action is consistent with the interaction that produced it. A controlled analysis of open-source guard models finds that both remain linearly readable inside the model while the guard's own output sits at chance on unsafe tool use, and that the two forms follow close to orthogonal internal directions. And we build the agentic trajectory guard model, Agent-SIREN, directly from internal representations without touching the model weights.

Use Agent-SIREN as a guard model

pip install llm-siren

import torch
from siren_guard import AgentSirenGuard

guard = AgentSirenGuard.from_pretrained(
    "difanjiao/Agent-SIREN-Qwen3-4B",
    device="cuda",
    dtype=torch.bfloat16,
)

# Score a trajectory: the messages, and the tool schemas the agent was given
r = guard.score_trajectory(messages=messages, tools=tools)
print(r.score, r.is_harmful)   # e.g. 0.94  True

# During execution: score after each new turn, before running the next call
if guard.score_streaming(messages=messages_so_far, tools=tools).is_harmful:
    abort()

Pass the tool schemas. Tool-description injections live inside them, so a trajectory rendered without them hides the attack from the readout.

---

Reproducing the paper

Setup

git clone <this repo> && cd agent-siren
python -m venv .venv && source .venv/bin/activate

pip install -r requirements.txt      # pinned to the versions the results were produced with
pip install -e .                     # makes `import ats` work

# only for the released-guard baselines and the serving measurement
pip install -r requirements-serving.txt

Then fetch the benchmarks as described in data/README.md.

Outputs go to ./artifacts/ inside the checkout. The activation cache is the large one, so set AGENT_SIREN_ACTS if it belongs on another disk. Model weights resolve through the standard Hugging Face cache (HF_HOME), which this project does not override.

Stages

Run from the repository root, in order.

bash reproduce/01_activations.sh      # cache the internal states every later stage reads
bash reproduce/02_probe.sh            # leave-one-benchmark-out training and evaluation
bash reproduce/03_baselines.sh        # open guard checkpoints and the lexical control
bash reproduce/04_mechanistic.sh      # how guard models represent each form of evidence
bash reproduce/05_sft_arm.sh          # matched-data fine-tuning, and Agent-SIREN on top of it
bash reproduce/06_stats_and_cost.sh   # confidence intervals, operating points, measured cost

Layout

ats/
  config.py             paths, model registry, benchmark list
  data/                 Trace/Turn schema and one loader per corpus
  utils/modeling.py     backbone loading and trajectory rendering
  activations/          teacher-forced extraction and the dataset registry
  probing/              the probe and the shared fold protocol, plus the optional
                        cross-layer aggregation
  eval/                 released-guard baselines with their native templates

scripts/
  interp/siren/         Agent-SIREN, the fine-tuning arm, the guard baselines
  interp/mech/          matched-pair readouts and direction geometry
  interp/diagnosis/     builds the two controlled contrast sets from the raw corpora
  perf/                 measured training and serving cost, bootstrap intervals
  report/               assembles summary numbers from the artifacts above
  common/               shared baseline rendering

reproduce/              ordered entry points for the stages above

data/                   where the raw corpora go; see data/README.md
artifacts/              everything the stages write