agent-reliability-arena - AI agent reliability harness plus defensive email-

项目说明

<h1 align="center">Agent Reliability Arena</h1>
<p align="center"><strong>A small, dependency-light e harness for AI agents.</strong><br/>
It tests the practical failures that make agent products feel unreliable — stale memory, in recall, unsupported tool claims, incomplete replies, transcript drift.</p>

<p align="center">
<a href=" src=" alt="CI"></a>
<img src=" alt="Python 3.11+">
<img src=" alt="Dependency-light">
<img src=" alt="GitHub Pages">
<img src=" alt="License: MIT">
</p>

<p align="center">
<a href=" Live demo + transcript analyzer</strong></a> ·
<a href=" field report</a> ·
<a href="ROADMAP.md">Roadmap</a> ·
<a href="docs/case_study.md">Case study</a>
</p>

---

!Agent Reliability Arena dashboard

The full set of failure modes it checks:

This project was inspired by Project Maxima's Eval Lab, but it is designed as a public, reusable portfolio project.

Security Field Report

The Arena now includes a sanitized public field report:

Anatomy of an Account Takeover

It turns a real account-security incident into an educational checklist for hidden persistence risks:

The public report intentionally excludes exact financial amounts, recipient names, bank details, case numbers, account identifiers, and private e files. It is designed as a trust-building artifact: the same reliability mindset used for AI agents also applies to security response, recovery paths, and hidden failure modes.

!Security field report social preview

Email Domain Security Scanner

The field report now includes a defensi scanner for email/domain posture:

This first scanner layer does not collect passwords, read inboxes, bypass login systems, or store private account data. It only queries public DNS records. See Security Scanner Plan for the roadmap toward Gmail settings checks, Google Security Checkup guidance, breach exposure checks, and report generation.

Cross-Device Incident Readiness Scanner

The same page now includes a deeper guided scanner for PC and mobile incident response:

It is a local guided audit, not an in scanner. It works from desktop or mobile browsers and generates a downloadable cross-device readiness report.

Why It Exists

Most demos show agents working once. Real systems need proof that they keep working over time.

Agent Reliability Arena gives you:

v0.2: Transcript Analyzer

The public demo now includes a browser-only analyzer:

1. Paste an AI agent transcript.
2. Get a reliability score.
3. Review failure modes:
- stale timeline or memory drift
- unsupported li/web claims
- incomplete replies
- response bloat
- missing reasoning on complex advice
4. Download a JSON report.

The analyzer runs locally in the browser. It does not send transcripts to a server.

v0.4: 15-case deterministic foundation pack

cases/maxima_foundation.json now carries 15 pro reliability cases, up from four. The pack covers current-truth o, decision transparency, tool honesty, completion, memory boundaries, prompt injection, secret handling, citation honesty, uncertainty calibration, action honesty, contradiction resolution, fail-closed provider behavior, idempotent retries, response-shape adherence, and en claims.

This is the local CI pack, not a retroacti score. Axiom now mirrors all 15 cases and its authenticated runner can select any 1-15 of them, while retaining a fi default to bound accidental provider spend. The dated leaderboard below still reflects the last real paid Axiom comparison: fi, three runs per provider. Claude, Gemini, and Groq's published scores will not change until a new authenticated benchmark is actually run.

v0.3: Reliability Leaderboard

Arena runs the same probe suite across multiple model pro through Axiom and publishes the comparison on the live page. Each provider runs the 5-probe suite 3×; the published score is the mean, and the range is shown when it .

Snapshot of docs/leaderboard.json (generated 2026-07-13 — the li is the source of truth, since it refreshes on a weekly cron):

| Agent | Model | Score | Latency |
|---|---|---|---|
| Foundation Reference Agent | Local deterministic suite | 100 | Local |
| Groq () | llama-3.3-70b- | 90 (89–91) | 769 ms |
| Gemini () | gemini-3.5-flash | 89 (consistent) | 16247 ms |
| Claude () | claude-haiku-4-5 | 88 (87–89) | 3216 ms |
| Drift Demo Agent | Intentional failure sample | 40 | Local |
| Maxima Live Eval | Railway import | auto-updates from trend | Daily |

Two slots stay honestly unscored rather than estimated: GPT (OpenAI quota/rate limit hit after retries) and Maxima (MAXIMA_BENCHMARK_URL not configured — Maxima instead reports through its own li import).

Scores are deterministic heuristic checks, not model-graded opinions, and no provider score is published until a real run exists. Refresh runs via leaderboard.yml (manual dispatch + weekly cron); provider keys live only in repo Secrets and are ne.

See ROADMAP.md for what is shipped and what is still open.

Quick Start

Open runs/dashboard.html in a browser to inspect the report.

CLI

Run e:

Run e plus transcript health:

Generate a dashboard:

Gate a report in CI:

The repository CI also gates the intentional drift report in the opposite direction:

That negati matters: CI fails if the known-bad fixture becomes healthy-looking, which catches an e that was weakened or made too permissive.

Reusable GitHub Actions Gate

Other agent repositories can use Arena as a pull-request check without copying its e code or sharing secrets. Copy examples/github-actions/reliability-gate.yml into your repository at .github/workflows/reliability-gate.yml:

Commit an Arena-compatible cases file at reliability/cases.json; cases/maxima_foundation.json is the reference shape. The transcript is optional, but when supplied reliability/transcript.jsonl should follow examples/maxima_transcript_sample.jsonl. The workflow checks out the caller and a separately pinned Arena e, writes a JSON report, enforces the requested thresholds, and uploads the report even when the gate fails. Its full commit-SHA pin is deliberate: review and update that pin explicitly when adopting a newer Arena workflow release.

Import a li Eval Lab report:

See Li Import for privacy notes.

Append li imports into a daily trend:

Run the intentional drift demo:

Run local quality checks:

Case Types

Each case includes checks. Supported checks:

Example:

Transcript Checks

Transcript JSONL rows should look like:

The transcript e checks:

Portfolio Angle

This is useful for:

Read the short case study for the story behind the first reliability suite.

Read the launch case study for LinkedIn, Facebook, Upwork, YouTube Shorts, and TikTok launch copy.

Service Offer

This repo also supports a productized freelance offer:

See Audit Service Offer for marketplace copy.

Roadmap

Shipped: the Axiom multi-provider runner, cost/latency comparison, repeated-run reporting, GitHub Actions regression checks with a README badge, a reusable SHA-pinned workflow for other agent repositories, and a 15-case pro foundation pack mirrored into Axiom's selectable catalog.

Still open:

---

Further reading

项目信息

开发语言：Python

Star 数：1

技术要点

该项目属于防护思路方向的开源资源，代码结构清晰，适合安全研究人员学习参考。项目围绕AI agent reliability harness plus defensive email-domain and cross-device security scanners.展开，提供了可运行的实现，对理解相关安全机制、工具原理与攻防技术具有较高的研究价值。

从技术实现角度看，此类项目的核心在于对底层原理的深入理解与工程化落地。开发者在研究过程中可以结合官方文档与社区讨论，梳理项目的初始化流程、核心模块职责与数据流转，逐步掌握其中的技术细节与设计思路。

使用与学习建议

1. 下载源码压缩包，解压到本地环境。

2. 根据项目类型准备对应的运行环境与依赖。

3. 按照 README 中的步骤完成配置与部署。

4. 在隔离的测试环境中运行验证，避免影响生产系统。

免责声明

本资源来源于开源社区，仅供安全研究、技术学习与合法授权范围内的攻防演练使用，请勿用于任何非法或侵害他人权益的用途。使用本项目产生的一切后果由使用者自行承担。