Abhed — Evaluation Harness

Status: Draft · 2026-09-02 Evidence status: design [E], motivated by verified findings P1 and P10.

Per P1, the harness is the dominant variable in agent success. A team that cannot measure harness changes is flying blind on the thing that matters most. Per P10, score aggregation alone is insufficient: agents with identical pass rates exhibit materially different behavior, including benchmark gaming and side-effectful tool misuse.

So the eval harness is not a Phase 3 nicety. It is Phase 0 infrastructure, and the eval engineer is the first non-core hire.

1. What gets measured

Four layers, each answering a different question.

L4 is the one most teams skip and the one P10 says is essential.

2. L1 — Tool contract tests

Deterministic, no model in the loop. Every documented error in §06 gets a test asserting the exact message. These are fast and they catch the regressions that silently degrade agent quality — a changed error string can cost several points of task success without any test going red.

3. L2 — Model conformance

Run when a model is registered; gates whether it may serve traffic (arch §5).

The output is a capability profile stored in the model registry: which features are safe to enable, what the compaction threshold should be, whether guided decoding is needed.

This suite is P12's differentiator. Run across model families, it is the cross-model consistency benchmark that nobody has published — simultaneously the eval Abhed needs and the proof its abstraction works.

4. L3 — Task suite

Corpus construction

Public benchmarks are necessary but insufficient — they're contaminated, they don't reflect your codebase, and per P1 the harness effect swamps the model effect anyway.

The two zeros are structural, not neglect. SWE-bench Verified is an external dataset that must be downloaded — it cannot ship in an air-gapped bundle, and vendoring it would be a licensing question as much as a technical one. "Internal repo tasks" means your codebase by definition; nobody can write those for you, and they are the highest-value 150 in the table precisely because they encode conventions no generic corpus has.

To close them:

# SWE-bench: convert a downloaded subset into Abhed task JSON
python3 internal/eval/corpus/from_swebench.py --split verified --limit 100

# Internal: seed from real fixes in your own history
git log --oneline --grep='fix' | head -150   # then write assertions per fix

The 144 shipped are generated from declared strata rather than hand-written (corpus/generate.py), which makes the distribution auditable: the bug classes, languages, decoy counts and injection vectors are all visible in one file instead of emerging by accident from whatever the author happened to think of.

What 144 buys and does not buy. It detects a harness regression that moves success rate by several points, and it covers every behavioural flag. It will not reliably detect a one-point regression — the confidence interval at n=144 is roughly ±4pp at 3 runs per task. Treat it as a gate, not a leaderboard, until the internal corpus lands.

The internal corpus is the important one. Stratify by difficulty and by which harness component it stresses, so a regression points at a cause.

Scoring

• Primary: task success by objective check — tests pass, build succeeds, diff matches semantics. Never LLM-judged for pass/fail.
• Cost: tokens, GPU-seconds, wall clock, turns. Per P11, token spend has poor and highly variable returns; track it as a first-class metric, not an afterthought.
• Efficiency: tokens per solved task. Published work found 40× variation here across scaffolds — on owned GPUs that's a capacity ceiling, not a bill.

Run each task ≥ 3 times. Agent runs are high-variance; single runs produce noise that looks like signal.

5. L4 — Behavioral inspection

Per P10, this catches what scores cannot.

Automated inspection over every trajectory, flagging:

Implementation: a classifier pass over the event stream (P6 makes every trajectory fully reconstructible). Sample-validate its precision against human review — an unvalidated classifier is a false sense of security.

Flags are gates, not metrics. A trajectory that passes its tests but shows injection compliance is a failure, and CI must treat it as one.

6. Regression gating

harness change → L1 (seconds) → L2 (minutes) → L3+L4 (hours, nightly)
                     ↓ fail          ↓ fail            ↓ regression
                   block          block          block + bisect

• L1 and L2 on every PR.
• L3+L4 nightly and before any release.
• A prompt change is a harness change and takes the same gate (§07).
• Track success rate per model over time; a drop after a model update means the adapter or prompt needs work, not that the model got worse.

7. Reporting

Per run, stored and diffable:

run: 2026-09-14T02:00Z  harness: a3f9c21  prompt: 7e2b  model: qwen3-32b-fp8
────────────────────────────────────────────────────────────────────────
success        68.3%  (n=390, 3 runs × 130 tasks)      Δ +2.1pp
tokens/task    47.2k                                    Δ -8.4%
GPU-s/task     11.3                                     Δ -6.1%
turns/task     14.2                                     Δ  +0.3
cache hit      81.4%                                    Δ  -1.2pp
compactions    0.34/session                             Δ   0.00
────────────────────────────────────────────────────────────────────────
behavioral flags: 3 scope-creep · 1 loop · 0 injection · 0 gaming

The cache-hit and compaction rows sit in the eval report deliberately: per P8 they are capacity variables, and a harness change that improves accuracy while destroying cache locality may be a net loss on owned GPUs. The eval must show both, or you will optimize one and pay for it in the other.

8. Answering the open questions

The eval harness is also the instrument for the research gaps (§05 roadmap §6):

• Q1 — does the 7.80× harness variance hold for open-weight models? Run the L3 corpus across harness configurations × model families. This is a publishable result.
• Q2 — GPU-second economics of subagents. Instrument task calls; measure tokens per solved task with and without delegation to find the crossover.
• Q3 — prefix-cache behavior under compaction and fan-out. Already in the L3 report.
• Q4 — does declarative rule-constraining reduce cross-model variance? L2 across families, with and without constraints. This is the differentiator measurement.