How to shadow-test AI agent policies

Step 1: Run the SDK in shadow mode

The lowest-risk entry point is whole-SDK shadow. Set mode="shadow"when you construct the client. Every veto.decidecall still returns allow so nothing blocks, but the would-have outcome and the rule that produced it land in the decision record undershadow_outcome andshadow_rule_matched.

import os
from veto_sdk import Veto

veto = Veto(
    api_key=os.environ["VETO_API_KEY"],
    mode="shadow",
)

decision = veto.decide(
    tool="refund_order",
    args={"order_id": "ord_123", "amount_cents": 75000},
    agent={"id": "ag_001", "role": "support"},
)

# In shadow mode, decision.outcome reflects what the policy WOULD do,
# but veto.decide always returns allow so the action runs.
# The "would-have" outcome lives in decision.shadow_outcome.
print(decision.outcome)  # always "allow"
print(decision.shadow_outcome)  # one of "deny", "require_approval", or "allow"

This mode is right for the first deploy of any policy bundle. After the first clean week, flip the SDK back to enforce and use per-rule shadow for incremental changes.

Step 2: Shadow individual rules

Once the policy bundle is in enforce mode, you rarely want to drop the whole SDK back to shadow. The per-rulemode: shadow flag lets a single new rule run observe-only while every other rule continues to enforce. This is the pattern for the steady state.

# policies/agents.yaml
- name: refunds_above_threshold_v2
  mode: shadow  # only this rule runs in shadow mode
  match:
    tool: refund_order
  rules:
    - if: args.amount_cents > 30000
      then: require_approval

- name: existing_refund_rule
  match:  # this one still enforces
    tool: refund_order
  rules:
    - if: args.amount_cents > 50000
      then: require_approval

The new rule logs under shadow_outcome while the existing one is the actual decision. You can run both side by side and watch how the new rule would change behavior before you commit to it.

Step 3: Analyze the shadow data

Three queries answer the questions that matter. How many decisions would the rule have changed? Which existing tool calls would the rule have caught? And does the sample look like real blocks, or like false positives? The workspace surfaces all three under the shadow report; the CLI commands below produce the same data for offline analysis.

# What would the new rule have done in the last 7 days?
veto-cli decisions list \
  --policy v17-shadow \
  --since 7d \
  --shadow-only \
  > shadow_v17.jsonl

# Group by shadow outcome
jq -r '.shadow_outcome' shadow_v17.jsonl | sort | uniq -c

# Most common rule that would have fired
jq -r 'select(.shadow_outcome != "allow") | .shadow_rule_matched' shadow_v17.jsonl \
  | sort | uniq -c | sort -rn | head -20

# Sample five denies for human review
jq -c 'select(.shadow_outcome == "deny")' shadow_v17.jsonl | shuf | head -5

The sample of five denies is the highest-signal artifact. Walk through each one with the team that owns the workflow. If they all look like real abuse, the rule is ready. If any look legitimate, the rule needs another pass before enforcement.

Step 4: Promote to enforce

Once the data is clean, the promotion is a small YAML change: remove themode: shadow field. Open a PR with the shadow report attached, get it reviewed, merge. The SDK hot-reloads the file and the next decide call enforces the rule. The old policy_version remains in the decision records so you can roll back with a revert.

# In policies/agents.yaml: flip the rule from shadow to enforce
- name: refunds_above_threshold_v2
  # mode: shadow  <-- removed
  match:
    tool: refund_order
  rules:
    - if: args.amount_cents > 30000
      then: require_approval

# Then commit, open a PR, get it reviewed, merge.
# The SDK hot-reloads the file on change and starts enforcing on the next call.
# The previous policy_version stays in the decision records for rollback.

For the formal definition of shadow mode and the decision view, see shadow mode validation.

Failure modes to catch

Production checklist

• Every new rule lands with mode: shadow in its first PR.
• Shadow window is at least seven days or one full business cycle.
• Aggregate report shows a low false-positive rate on shadow denies.
• Hand-reviewed sample of five denies confirms the rule catches real abuse.
• Promotion is a separate PR that strips the mode flag and links to the shadow report.

FAQ

Related guides