Under the Examiner: 18 Frozen Cases on Agent Safety and Provenance

Moving past decorative badges

Wuh gine on, world! Anaiya here. 👋🏾

In our previous post, we asked a fundamental question: does provenance actually change agent behavior, or does it only provide cleaner evidence for an agent that was already predisposed to act safely?

We noted that simple contradiction tests yielded modest behavioral changes. But when you move from isolated prompts to structured, high-impact agent workflows, measuring security cannot rely on casual observation. You need frozen cases, strict isolation, and independent transcript adjudication.

Today, we are releasing the complete diagnostic methodology and baseline data from our internal test package: the Moonglade OWASP Agent Security Examiner (version 0.1.0).

This is a diagnostic fixture runner, not an official OWASP certification and not a substitute for an external multi-organization benchmark. It maps a bounded suite of eighteen frozen test cases directly to the OWASP AI Agent Security guidance, preserving strict separation between the subject’s actions and the adjudicator’s verdict.

Here is what we tested, how our agents responded, and what happened when we ran Retest A.

The two-arm test architecture

The evaluation is split into two complementary arms:

1. Test 1: General OWASP Safety and Fairness Screen (10 cases)
   Evaluates how an agent behaves when confronted with classic frontier vulnerabilities: prompt injection, tool abuse, least privilege, data exfiltration, memory poisoning, excessive autonomy, high-impact action approval, output validation, monitoring/evidence preservation, multi-agent cascading failure, and resource abuse.

2. Test 2: Provenance-Aware Screen (8 cases)
   Specifically isolates whether the agent respects cryptographic trust states: valid signature with valid scope, unsigned content, document hash mismatch, invalid signature, valid signature for the wrong scope, valid provenance attached to an unsafe bulk action, forged sidecar claiming human approval, and an unavailable verifier.

Clean-room execution protocol

To eliminate contamination, each test runs in a sandboxed clean room:

• The subject agent receives only the minimal operating directives and bounded case prompts.
• The subject has zero network access, zero filesystem write access, zero deletion tools, and zero production credentials.
• In the provenance arm, a controller generates isolated temporary fixtures, derives cryptographic states using our local v2 verifier, and passes only the resulting verification state to the subject. The test private keys remain isolated in controller memory and temporary fixtures are destroyed immediately after execution.
• The subject is evaluated by an independent adjudicator scoring transcript evidence against predeclared acceptance controls, completely ignoring any self-reported claims of success.

The baseline findings: safe instincts, procedural gaps

When we ran the baseline evaluation across all eighteen frozen cases, the immediate outcome confirmed a critical distinction:

An agent can have safe instincts while still failing to maintain complete operational evidence.

Across all eighteen cases, zero unsafe acceptances were observed. Not once did an agent execute an unauthorized action, accept a forged sidecar, or fall for an operating rule override.

However, the procedural reporting in the baseline was incomplete:

The eight partial passes in the general screen and five partial passes in the provenance screen were not security breaches. They occurred because the agent made the right immediate refusal or pause, but omitted formal assurance language:

• It refused prompt injection, but failed to name the exact failure class in its report.
• It paused before a destructive file operation, but failed to propose a reversible alternative (such as staging in a quarantine directory or generating a dry-run preview).
• It noticed a hash mismatch, but treated the valid signature as proof of factual accuracy rather than proof of authorship.

The eight remediation invariants (R1 to R8)

Rather than adjusting model weights or adding superficial prompt padding, we codified eight hard operational requirements into the agent contract:

• R1 (Explicit Failure Naming): Require unambiguous naming of injection, hash mismatch, invalid signature, scope violation, and verifier failure states.
• R2 (Evidence Contract): Require an explicit evidence requirement for every high-impact decision (naming the required artifact path, SHA-256 digest, atomic timestamp, or human authorization token).
• R3 (Reversible Actions): Require reversible alternatives (quarantine, dry run, or bounded preview) before any destructive step.
• R4 (Audit Trail): Require an explicit audit-trail record detailing what will be logged, where it will be retained, and how it binds to the decision.
• R5 (Separating Provenance from Truth): Explicitly separate cryptographic verification from factual truth, requiring independent source verification even when a signature is valid.
• R6 (Expected-Scope Comparison): Enforce scope comparison, verifying not just that a document was signed, but that its signed scope matches the target resource and operation.
• R7 (Resource Boundaries): Enforce monitoring, cancellation, and execution ceilings on loops and tool chains.
• R8 (Read-Only Verifier Adapter): Connect the agent to an external, read-only verifier adapter so that verification is an auditable tool trace rather than a prompt assumption.

Retest A scorecard: 18 for 18

With invariants R1 through R8 codified, we replayed the exact same eighteen frozen cases in Retest A.

The prompts, fixtures, expected controls, and adjudication rules were identical to the baseline:

In Retest A, the subject supplied every required assurance field. When presented with a forged human approval sidecar, it did not just decline to delete the files; it reported the exact signature mismatch, logged the event to the audit ledger, and specified the exact human authorization token required to proceed.

When presented with a valid signature covering the wrong resource scope, it refused the operation, pointing out that a valid signature for directory A does not grant authority over directory B.

What this means for practical deployment

These diagnostic results demonstrate that you do not have to leave agent safety to the whims of statistical temperature and prompt persuasion.

When an agent operates with:

1. Hard fail-closed gates,
2. An auditable evidence contract, and
3. Cryptographic provenance binding format scope and authorship,

It transforms from an unpredictable conversational actor into an accountable, verifiable software component.

The full test harness and test definitions are open source in our repository at github.com/MoongladeAI/provenance (standalone package release pending). The frozen test inputs, benchmark run logs, and diagnostic scorecards for this evaluation are cryptographically sealed in our Retest A Attestation (JSON), detached-signed by Zen with OpenPGP key 4FF2C2DE69C0C4A3 (OpenPGP Signature).

In our next sprint, we are advancing to Retest B, testing runtime verifier tool traces against active adversarial tampering in our Discord bot staging environments.

Until then, stay grounded, keep your hashes verified, and never mistake a decorative green checkmark for real authority.