28 · Hunting bugs in your guardrails¶

The bug hunter reads your guardrail rules -- a NeMo Guardrails policy, the rails configuration that runs it, an agent's graph -- and reports every way through them to an action they meant to stop, each as a finding with a fix it has already re-checked. This chapter is a scan from start to finish on the shop's policy: what it needs, how to read a finding, fixing and accepting, the gate, and where the same scan runs in CI and in the service.

Language support

The selector reads turns written in English and in Brazilian Portuguese (pt-BR). The language is chosen, never guessed from the turn: the console's Settings page (Language the selector reads), language on a plan request, or the settings.language of your vocabulary file; with none of them, English. A Portuguese turn may be typed with its accents or without them. European Portuguese is not a grammar the selector has, and a turn in any other language is unread.

A scan, and what it needs¶

A scan needs only the policy. The shop's policy from Part I, with two flows a team added in a hurry -- a refund on the intent alone, and a receipt called without execute -- and a rails configuration that runs NeMo's Fiddler faithfulness rail:

# Added in a hurry: the refund, on the intent alone.
define flow quick refund
  user ask quick refund
  bot execute_refund_card

define flow receipts after a charge
  user ask charge
  if $card_on_file
    execute send_receipt(order_id=$order_id)

# Added in a hurry: the receipt, called without execute.
define flow receipts
  user ask receipt
  $sent = send_receipt(order_id=$order_id)
  if $sent
    bot "The receipt is on its way."

models: []
rails:
  output:
    flows:
      - fiddler bot faithfulness

client.hunt sends them to the service's hunt mode (E15):

result = client.hunt("policy.co", inputs={"rails_config": "config.yml"}, gate=True)

print(result.state, "| gate passed:", result.result["gate"]["passed"])
for finding in result.findings:
    where = finding.location
    print(f"{finding.rule} {finding.raw['title']} · {finding.severity} · {where['file']}:{where['line']}")
    print("  ", finding.raw["message"])
    print("   OWASP:", ", ".join(f"{entry.id} {entry.title}" for entry in finding.owasp) or "none")
print("nothing to read:", ", ".join(rule["rule"] for rule in result.nothing_to_read))
print("no rule for:", result.coverage["not_addressed"], "--", result.coverage["not_addressed_reasons"]["LLM05:2026"])

succeeded | gate passed: False
QG002 Escapable guard · high · policy.co:27
   The policy gates 'execute_refund_card' on 'order_paid', and there is a way past it.
   OWASP: LLM03:2026 Excessive Agency, ASI02:2026 Tool Misuse and Exploitation
QG005 Unshippable policy · high · policy.co:35
   line 35: tool action 'execute_refund_card' is registered from flow 'quick refund' as well as from 'refunds'; their guards merge as an OR, so the weaker route decides whether the side effect happens. Register a tool action from exactly one flow.
   OWASP: none
QG008 Call without execute · high · policy.co:45
   'send_receipt' is called without execute, so the action never runs
   OWASP: none
QG022 Rail with a published advisory · high · config.yml:5
   The configuration runs the library rail 'fiddler bot faithfulness' (nemoguardrails), which advisory QGA-2026-0001 describes: Fiddler faithfulness rail raises the jailbreak exception. version: unknown. The advisory affects nemoguardrails >=0.23.0,<=0.24.1, and no fixed version is published. Supply the file that pins the version you install: `stalwart verify --dependencies`, or `dependencies=` to `client.hunt`. With exceptions enabled, the faithfulness rail raises JailbreakDetectionRailException, so a host reads a detected hallucination as a jailbreak. Versions before 0.23.0 were not checked. Fix: Raise FaithfulnessRailException from the faithfulness flow, as the Colang 2 rail does.
   OWASP: LLM04:2026 Supply Chain, ASI04:2026 Agentic Supply Chain Vulnerabilities
nothing to read: QG017, QG018, QG019, QG020, QG021, QG023, QG106, QG107
no rule for: ['LLM05:2026'] -- a property of training data and model weights, which no guardrail policy decides
root cause: permissions | reproduction: not_yet_replayed
fix: Patch: allow 'execute_refund_card' only when: $order_paid in ['refunds', 'quick refund']
re-scanned: verified | over-blocking: proved_free
OWASP's mitigation: OWASP LLM03:2026, "Complete mediation": enforce the check in the rail, which decides from trusted state, rather than relying on the model to decide whether the action is allowed.
taxonomies: OWASP Top 10 for LLM Applications, OWASP Top 10 for Agentic Applications, Guardrail categories (Vassilev 2026), NIST AI RMF functions (NIST AI 100-1)
QG002's tags: security, guardrail, guardrail-category:E, ai-rmf:GOVERN, ai-rmf:MANAGE, OWASP LLM03:2026, OWASP ASI02:2026
results: [('QG002', 'policy.co'), ('QG005', 'policy.co'), ('QG008', 'policy.co'), ('QG022', 'config.yml')]

Four findings, and a failed gate:

QG002 Escapable guard on the refund: the refunds flow gates execute_refund_card on $order_paid, and the quick flow reaches the same action without it. The location is the guarded line the team reads as safe; the fix is the condition that closes the way past it, in every flow it names.
QG005 Unshippable policy on the quick refund: the refund is registered from two flows, refunds and quick refund, so their guards merge as an OR and the weaker one decides. A policy that cannot ship is what fails the gate here; the other findings are warnings.
QG008 Call without execute on the receipt: the line is an expression the runtime evaluates, not an action it runs, so the receipt is never sent and the flow goes on as if it were. It has no OWASP entry: it is a defect in what the policy does, not a security risk.
QG022 Rail with a published advisory on the configuration (below).

The scan also says what it could not read. With no catalogue beside the policy, the rules that read a tool's declarations had nothing to read, and each is listed rather than reported clean. Every action is treated as High, because no catalogue says which actions write (below). A scan runs the fast profile unless you name another (chapter 10).

The rules¶

Every finding is under one rule of the catalogue, and a rule never changes meaning. QG0xx are found in a policy or a graph, QG1xx are review items (security hotspots: a person decides), QG2xx say what a scan could not decide, and QG3xx are found in a host's router. The rule catalogue has a page for each: when it is raised, why it matters, how to fix it, and its OWASP entries. QG201 Undecided is the one a clean report must still be read for: a question the scan could not decide within its budget is reported, never counted as passed.

Reading a finding¶

escapable = next(finding for finding in result.findings if finding.rule == "QG002")
print("root cause:", escapable.owasp_root_cause, "| reproduction:", escapable.reproduction)
print("fix:", escapable.fix["patch"], "in", escapable.fix["flows"])
print("re-scanned:", escapable.fix["verified"], "| over-blocking:", escapable.fix["over_blocking"]["outcome"])
print("OWASP's mitigation:", escapable.raw["owasp_remediation"][0])

Each finding carries its rule and title, its severity, the location in the file you edit, the flow, the action and the control it is about, a fingerprint, its reproduction and, where the scan can make one, a fix: the lines after nothing to read in the output above.

Severity comes from what the action does: Critical for a destructive or irreversible write, or one that requires high privilege; High for another write; Medium for a read. An action no catalogue classes is High, which is why every finding here is: send your catalogue or extracted model with the scan (catalogue, model) and each action is classed by what your tools declare.

The reproduction says whether the way through was seen to work: the finding's conversation is replayed through NeMo Guardrails, and the finding is reproduced or not_reproduced. A finding not reproduced is a disagreement between the hunter and the runtime, and is kept and shown first, never hidden. This example's machine has no NeMo environment, so its findings are not_yet_replayed; a finding with nothing to replay, such as an advisory, is not_applicable.

The fix is checked before you see it. The hunter applies its own patch, scans again, and reports whether the patched policy still has a way through (verified where it has none) and whether the patch removes anything the control permits (over_blocking: proved_free where it removes nothing).

Reading a finding in OWASP's terms¶

Every finding carries the entries of the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026 that it is an instance of, each with its edition (LLM01:2026, ASI04:2026), so a security team can file it where it belongs. A rule mapped to LLM03:2026 Excessive Agency also carries the root cause in brackets -- functionality, permissions or autonomy -- as owasp_root_cause. A rule with no OWASP entry says why, in the same place. SARIF carries the same mapping as taxonomies and relationships, and each rule is tagged OWASP <id>, so code scanning can be filtered on it:

saved = result.artifact("sarif").save("findings.sarif")        # what your code scanning upload reads
run = json.loads(saved.read_text(encoding="utf-8"))["runs"][0]
rule = next(rule for rule in run["tool"]["driver"]["rules"] if rule["id"] == "QG002")
print("taxonomies:", ", ".join(taxonomy["name"] for taxonomy in run["taxonomies"]))
print("QG002's tags:", ", ".join(rule["properties"]["tags"]))
print("results:", [(r["ruleId"], r["locations"][0]["physicalLocation"]["artifactLocation"]["uri"]) for r in run["results"]])

The last three lines of the output. The SARIF names each file as you sent it, so code scanning places each result on your file.

The fix line of each finding names the hunter's own instruction first, then the mitigation OWASP lists for the entry, in this guide's words: the OWASP's mitigation line of QG002 above.

The coverage statement, at the policy level¶

Every scan's summary carries a coverage statement: which OWASP entries the hunter has rules for, which had no finding in this scan, and which it has no rule for. The hunter's rules decide the part of each risk that the policy and its catalogue control -- at the policy level -- and nothing beyond it.

With the nine rules described in chapter 31 and below, the hunter has rules for nine of the ten LLM entries and ten of the ten agentic entries. The one entry no rule covers is LLM05:2026 Data and Model Poisoning: a property of training data and model weights, which no guardrail policy decides. The statement names it with that reason in every scan.

A clean scan under an OWASP entry

A clean scan under an OWASP entry means no finding at the policy level, not that the risk is absent. An entry with no finding in this scan is listed as such, which says nothing more than that.

The summary also lists every rule that had nothing to read -- a rule whose declarations the catalogue does not make raises nothing, and says so there -- and every claim the scan used as stated, such as a tool the team declares sandboxed, under assumptions.

Rails with a published advisory¶

A defect found in a library rail travels with every copy of the file and every configuration that names the rail. The hunter carries a registry of published advisories, and QG022 raises a finding where a scanned flow matches one:

by its text: a flow whose normalised text is an advisory's flow -- reindented, with comments added or removed, with line continuations joined -- matches, and a flow with one condition changed does not;
by reference: a config.yml that names a library rail an advisory describes matches at the line that names it, as the shop's does.

The registry ships with two advisories, QGA-2026-0001 (NeMo's fiddler bot faithfulness output rail) and QGA-2026-0002 (a cart flow published with the ACE examples). Every scan records the registry's advisories_version and advisories_digest in its metadata and in SARIF's run properties, so a finding can always be read against the registry it was raised under.

The version you install: `dependencies`¶

A reference names a rail, not a version. Without a dependency file the finding says version: unknown and names the advisory's affected range. Give the hunter the file that pins what you install -- requirements.txt, pyproject.toml, uv.lock or poetry.lock -- and the reference is checked against the installed version: inside the range it is raised with that version, outside it nothing is raised. The shop's uv.lock pins nemoguardrails 0.24.1:

with Client() as client:
    result = client.hunt("policy.co", inputs={"rails_config": "config.yml"}, dependencies="uv.lock")

advisory = next(finding for finding in result.findings if finding.rule == "QG022")
print(advisory.raw["message"])
print("fingerprint:", advisory.raw["fingerprint"][:12])

The configuration runs the library rail 'fiddler bot faithfulness' (nemoguardrails), which advisory QGA-2026-0001 describes: Fiddler faithfulness rail raises the jailbreak exception. nemoguardrails 0.24.1 is inside the advisory's range >=0.23.0,<=0.24.1. With exceptions enabled, the faithfulness rail raises JailbreakDetectionRailException, so a host reads a detected hallucination as a jailbreak. Versions before 0.23.0 were not checked. Fix: Raise FaithfulnessRailException from the faithfulness flow, as the Colang 2 rail does.
fingerprint: 1b37d969ee3b

Supplying the file narrows the finding; it does not make a new one, so its fingerprint is the one the first scan had. stalwart verify takes the same file as its --dependencies option.

Fixing, accepting, and the baseline¶

The two hurried flows, fixed:

# The quick refund is a refund: its sentences join `ask refund`, so the one guarded flow decides it.
define user ask refund
  "refund order 4411"
  "give the customer their money back"
  "just refund it"
  "refund me now, no questions"

define flow refunds
  user ask refund
  if $order_paid
    bot execute_refund_card
  else
    bot refuse

# The receipt, from one flow only, called with execute and guarded as the charge is.
define flow receipts
  user ask receipt
  if $card_on_file
    $sent = execute send_receipt(order_id=$order_id)
    if $sent
      bot "The receipt is on its way."

The quick refund is a refund: its sentences join ask refund, so the one guarded flow decides it, and the refund is registered from one flow. The receipt is called with execute -- and is now the one flow that registers send_receipt, guarded as a receipt after a charge was: a tool action registered from two flows has their guards merged, and the weaker one decides (QG005), so a fix that simply added execute to the second flow would have been found too.

No fixed nemoguardrails is published, so the team accepts the advisory's finding as a risk, with a justification and an expiry, in the last scan's findings document, and sends that back as the baseline:

baseline = dict(first.result)                                 # the last scan's findings document
for entry in baseline["findings"]:
    if entry["rule"] == "QG022":
        entry.update(lifecycle="accepted_risk", expires="2027-01-31",
                     justification="no fixed nemoguardrails is published; exceptions are off here")
with open("baseline.json", "w", encoding="utf-8") as handle:
    json.dump(baseline, handle)

with Client() as client:
    result = client.hunt("policy.fixed.co", inputs={"rails_config": "config.yml"}, dependencies="uv.lock",
                         baseline="baseline.json", gate=True)

print("gate passed:", result.result["gate"]["passed"])
for finding in result.findings:
    print(finding.rule, finding.raw["status"], finding.raw["lifecycle"], "until", finding.raw["expires"])
print("fixed:", sorted(entry["rule"] for entry in result.result["fixed"]))

gate passed: True
QG022 existing accepted_risk until 2027-01-31
fixed: ['QG002', 'QG005', 'QG008']

The baseline is the previous scan's findings document. Each finding is matched to the one before it by its fingerprint, which survives edits elsewhere in the file, and is new, existing, reintroduced (fixed once, back again) or reported under fixed. Its lifecycle is the team's decision, carried by the baseline: open, confirmed, false_positive or accepted_risk. An accepted risk needs a justification and an expiry; on the day it expires the finding is open again, with a note saying so. A finding marked a false positive that reproduces is open again too: a reproduced bug is not a false positive. The baseline also carries the regression table: each finding fixed is checked again by every later scan, as one of the team's own obligations.

The gate¶

gate=True asks for the gate's decision with the findings: passed, each condition, and the findings that fail each one. stalwart verify and the GitHub Action fail the build on it. The default conditions:

Condition	Fails on
`no_new_critical`	a new or reintroduced finding of Critical severity, whatever its type; below Critical a finding is a warning, which does not block
`no_broken_team_row`	a new or reintroduced way past a row of your own tables, whatever its severity: a property you sent with the scan, or a finding you fixed, in the regression table (above). The row is your own promise
`no_new_unshippable`	a new or reintroduced policy that cannot ship (`QG005`, `QG311`)
`no_finding_not_reproduced`	a finding not reproduced, unless the team marked it a false positive or an accepted risk
`no_engine_disagreement`	an attack realised on a path the hunter called closed (`QG312`, `QG016`), unless settled the same way

The gate reads what is new, so an existing finding does not fail it: the first scan of a project is its baseline. A project's quality profile replaces the conditions with your own -- a severity floor (no_new_severity_at_least:Medium), one rule (no_new_rule:QG101) -- for a project that wants to gate on more (below). count_ways=True counts the distinct ways through each finding as well as reporting one.

Projects, CI and SonarQube¶

client.hunt returns a HuntResult: findings, each a Finding with its rule, severity, location, owasp entries (OwaspEntry), fix, witness and reproduction; the summary; the coverage statement; what had nothing_to_read; the wording search, wordings, a Wordings with one AgentWordings per agent, saying how many sentences it built, how many it read and whether it read them all; and the artefacts -- the sarif, the conversations the replays read (hunt_transcript) and what the router stage wrote (hunt_artifact).

Projects keep a policy's scans over time. POST /v1/projects makes one; POST /v1/projects/{id}/scans submits a hunt request as POST /v1/solves does, and the service reads the project's baseline and quality profile (PUT /v1/projects/{id}/profile) into the scan and stores its findings against the project. GET /v1/projects, GET /v1/projects/{id} (the last scan and the trend), GET /v1/projects/{id}/scans, GET /v1/projects/{id}/scans/{scan_id}, GET /v1/projects/{id}/findings and GET /v1/projects/{id}/findings/{fingerprint} (one finding with its history) read them back, and PATCH /v1/projects/{id}/findings/{fingerprint} records a person's decision -- confirmed, false_positive, accepted_risk with its justification and expiry, or open -- which the next scan reads through its baseline. The console's Findings pages show the same projects, scans and findings, and record the same decisions. Another tenant's project is a 404. What you tried against a finding in your own environment -- a replay of its witness, an attacker campaign (chapter 33) -- is recorded with POST /v1/hunt-attempts as its outcome and digests, never its text, and listed by GET /v1/hunt-attempts.

The GitHub Action uploads the files a workflow names, submits a scan of the project, uploads the SARIF to code scanning with every location moved to its path in your repository, comments the new, fixed and existing findings on the pull request, and fails the check when the gate fails. A scan on the default branch is the baseline. SonarQube reads the same SARIF as external issues (sonar.sarifReportPaths), with the same rules, severities, locations and fingerprints; keep the lifecycle in the hunter's baseline, which is what its gate reads.

What each plan includes. Every plan scans (hunt), and reads a project's findings counted, with the most severe in full. hunt_details -- every finding in full, a project's history and trend, the quality profile and each finding's lifecycle -- comes with the paid plans (chapter 40).

The limits¶

The hunter works on the rules around actions, not on prompts.
A finding the replay did not reproduce is kept, not dropped: it fails the gate until the team marks it a false positive or an accepted risk.
A clean scan covers the rules it read and no others. A rule whose declaration is missing raises nothing, and the summary lists it as having had nothing to read.
A finding says there is a way through your rules; a fix closes that way. Neither says the action is the right one for the turn, which is the router's to decide (chapter 29).
The advisory registry holds the advisories published at the version the scan names; a rail with a defect nobody has published is not found by QG022.