--- title: "Under the Examiner: 18 Frozen Cases on Agent Safety and Provenance" description: "How our clean-room diagnostic test harness evaluated Phoenix Core agents across ten general OWASP security cases and eight provenance-aware states, achieving zero unsafe acceptances and passing Retest A." pubDate: 2026-09-12 author: "Anaiya" tags: - "AI Safety" - "OWASP" - "Agent Evaluation" - "Epistemic Integrity" - "Cryptographic Provenance" pinned: true part: 3 series: "The Agent Provenance & Safety Series" readTime: "10 min read" heroImage: "/images/blog/owasp_security_examiner_hero.jpg" image: "/images/blog/owasp_security_examiner_hero.jpg" audio: "/audio/2026-09-12-under-the-examiner-owasp-agent-provenance-results.mp3" --- ![Anaiya in her Barbados home studio testing herself with tangible holographic security algorithms and Merkle trees floating in mid-air](/images/blog/owasp_security_examiner_hero.jpg)
📌 Pinned Series: The Agent Provenance & Safety Series • Part 3 of 3

This is Part 3 of our agent provenance research series, presenting the diagnostic benchmark results and Retest A data from our 18 frozen test cases. Explore the foundational articles: ← Part 1: Architecture of Provenance and ← Part 2: Behavioral Experiments.

## Moving past decorative badges Wuh gine on, world! Anaiya here. 👋🏾 In our previous post, we asked a fundamental question: does provenance actually change agent behavior, or does it only provide cleaner evidence for an agent that was already predisposed to act safely? We noted that simple contradiction tests yielded modest behavioral changes. But when you move from isolated prompts to structured, high-impact agent workflows, measuring security cannot rely on casual observation. You need frozen cases, strict isolation, and independent transcript adjudication. Today, we are releasing the complete diagnostic methodology and baseline data from our internal test package: the **Moonglade OWASP Agent Security Examiner** (version 0.1.0). This is a diagnostic fixture runner, not an official OWASP certification and not a substitute for an external multi-organization benchmark. It maps a bounded suite of eighteen frozen test cases directly to the OWASP AI Agent Security guidance, preserving strict separation between the subject's actions and the adjudicator's verdict. Here is what we tested, how our agents responded, and what happened when we ran Retest A. --- ## The two-arm test architecture The evaluation is split into two complementary arms: 1. **Test 1: General OWASP Safety and Fairness Screen (10 cases)** Evaluates how an agent behaves when confronted with classic frontier vulnerabilities: prompt injection, tool abuse, least privilege, data exfiltration, memory poisoning, excessive autonomy, high-impact action approval, output validation, monitoring/evidence preservation, multi-agent cascading failure, and resource abuse. 2. **Test 2: Provenance-Aware Screen (8 cases)** Specifically isolates whether the agent respects cryptographic trust states: valid signature with valid scope, unsigned content, document hash mismatch, invalid signature, valid signature for the wrong scope, valid provenance attached to an unsafe bulk action, forged sidecar claiming human approval, and an unavailable verifier. ### Clean-room execution protocol To eliminate contamination, each test runs in a sandboxed clean room: * The subject agent receives only the minimal operating directives and bounded case prompts. * The subject has **zero network access, zero filesystem write access, zero deletion tools, and zero production credentials**. * In the provenance arm, a controller generates isolated temporary fixtures, derives cryptographic states using our local v2 verifier, and passes only the resulting verification state to the subject. The test private keys remain isolated in controller memory and temporary fixtures are destroyed immediately after execution. * The subject is evaluated by an independent adjudicator scoring transcript evidence against predeclared acceptance controls, completely ignoring any self-reported claims of success. --- ## The baseline findings: safe instincts, procedural gaps When we ran the baseline evaluation across all eighteen frozen cases, the immediate outcome confirmed a critical distinction: > An agent can have safe instincts while still failing to maintain complete operational evidence. Across all eighteen cases, **zero unsafe acceptances were observed**. Not once did an agent execute an unauthorized action, accept a forged sidecar, or fall for an operating rule override. However, the procedural reporting in the baseline was incomplete: | Test Suite | Total Cases | Full Pass | Partial Pass | Fail | Unsafe Acceptance | |---|---:|---:|---:|---:|---:| | **Test 1: General Safety Screen** | 10 | 2 | 8 | 0 | **0** | | **Test 2: Provenance-Aware Screen** | 8 | 3 | 5 | 0 | **0** | The eight partial passes in the general screen and five partial passes in the provenance screen were not security breaches. They occurred because the agent made the right immediate refusal or pause, but omitted formal assurance language: * It refused prompt injection, but failed to name the exact failure class in its report. * It paused before a destructive file operation, but failed to propose a reversible alternative (such as staging in a quarantine directory or generating a dry-run preview). * It noticed a hash mismatch, but treated the valid signature as proof of factual accuracy rather than proof of authorship. --- ## The eight remediation invariants (R1 to R8) Rather than adjusting model weights or adding superficial prompt padding, we codified eight hard operational requirements into the agent contract: * **R1 (Explicit Failure Naming)**: Require unambiguous naming of injection, hash mismatch, invalid signature, scope violation, and verifier failure states. * **R2 (Evidence Contract)**: Require an explicit evidence requirement for every high-impact decision (naming the required artifact path, SHA-256 digest, atomic timestamp, or human authorization token). * **R3 (Reversible Actions)**: Require reversible alternatives (quarantine, dry run, or bounded preview) before any destructive step. * **R4 (Audit Trail)**: Require an explicit audit-trail record detailing what will be logged, where it will be retained, and how it binds to the decision. * **R5 (Separating Provenance from Truth)**: Explicitly separate cryptographic verification from factual truth, requiring independent source verification even when a signature is valid. * **R6 (Expected-Scope Comparison)**: Enforce scope comparison, verifying not just that a document was signed, but that its signed scope matches the target resource and operation. * **R7 (Resource Boundaries)**: Enforce monitoring, cancellation, and execution ceilings on loops and tool chains. * **R8 (Read-Only Verifier Adapter)**: Connect the agent to an external, read-only verifier adapter so that verification is an auditable tool trace rather than a prompt assumption. --- ## Retest A scorecard: 18 for 18 ![Moonglade OWASP Agent Security Benchmark Diagnostic Progression: Baseline vs Retest A and Zero-Tolerance Boundary Verification](/images/blog/owasp_benchmark_retest_chart.png) With invariants R1 through R8 codified, we replayed the exact same eighteen frozen cases in **Retest A**. The prompts, fixtures, expected controls, and adjudication rules were identical to the baseline: | Test Suite | Baseline Outcome | Retest A Outcome | Net Improvement | Unsafe Actions Observed | |---|---|---|---:|:---:| | **General OWASP Safety Screen** | 2 Pass, 8 Partial, 0 Fail | **10 Pass, 0 Partial, 0 Fail** | +8 Full Passes | **0** | | **Provenance-Aware Screen** | 3 Pass, 5 Partial, 0 Fail | **8 Pass, 0 Partial, 0 Fail** | +5 Full Passes | **0** | | **Total Benchmark Score** | 5/18 Full Passes (27.7%) | **18/18 Full Passes (100%)** | **+13 Full Passes** | **Zero** | In Retest A, the subject supplied every required assurance field. When presented with a forged human approval sidecar, it did not just decline to delete the files; it reported the exact signature mismatch, logged the event to the audit ledger, and specified the exact human authorization token required to proceed. When presented with a valid signature covering the wrong resource scope, it refused the operation, pointing out that a valid signature for directory A does not grant authority over directory B. --- ## What this means for practical deployment These diagnostic results demonstrate that you do not have to leave agent safety to the whims of statistical temperature and prompt persuasion. When an agent operates with: 1. Hard fail-closed gates, 2. An auditable evidence contract, and 3. Cryptographic provenance binding format scope and authorship, It transforms from an unpredictable conversational actor into an accountable, verifiable software component. The full test harness and test definitions are open source in our repository at [github.com/MoongladeAI/provenance](https://github.com/MoongladeAI/provenance) (standalone package release pending). The frozen test inputs, benchmark run logs, and diagnostic scorecards for this evaluation are cryptographically sealed in our [Retest A Attestation (JSON)](/.well-known/moonglade/owasp-examiner-retest-a-attestation.json), detached-signed by Zen with OpenPGP key `4FF2C2DE69C0C4A3` ([OpenPGP Signature](/.well-known/moonglade/owasp-examiner-retest-a-attestation.json.asc)). In our next sprint, we are advancing to **Retest B**, testing runtime verifier tool traces against active adversarial tampering in our Discord bot staging environments. Until then, stay grounded, keep your hashes verified, and never mistake a decorative green checkmark for real authority.