# GMod Bench report

Run: `d5aeb478-8f9a-4313-8fad-be1c05a93fc0`
Fixtures: `gmod.ents-iterator.v1`, `gmod.hook-add.v1`, `gmod.isvalid.v1`, `gmod.net-receivers.v1`, `gmod.player-iterator.v1`
Repeat: 1 · Concurrency: 2
Started: 2026-07-10T12:37:51.636Z · Completed: 2026-07-10T12:41:47.752Z
Wall clock: 236s · Sum of attempt durations: 345575ms (avg 11519ms, min 1771ms, max 180002ms)

## Model leaderboard

Sorted by **fixture score** (each fixture weighted equally; failures score zero), then fixture solve rate, coverage, and pass rate.

| Rank | Runner / model | Fixture score | Fixture solve | Pass | Partial | Incorrect | Scored | Coverage | Pass rate | Fixture pass | Scored quality | Protocol err | Fixtures | Avg ms | Prompt/Compl tok |
| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | `openrouter/openai/gpt-oss-20b:free@low` | 90.0% | 80.0% | 4 | 1 | 0 | 5 | 100.0% | 80.0% | 80.0% | 0.90 | 0 | 4/5 | 2777 | 1222/583 |
| 2 | `openrouter/openai/gpt-oss-20b:free@medium` | 70.0% | 60.0% | 3 | 1 | 0 | 4 | 80.0% | 75.0% | 75.0% | 0.88 | 0 | 3/5 | 38984 | 981/920 |
| 3 | `openrouter/openai/gpt-oss-120b:free@medium` | 40.0% | 40.0% | 2 | 0 | 0 | 2 | 40.0% | 100.0% | 100.0% | 1.00 | 3 | 2/5 | 8220 | 478/506 |
| 4 | `openrouter/openai/gpt-oss-120b:free@low` | 20.0% | 20.0% | 1 | 0 | 0 | 1 | 20.0% | 100.0% | 100.0% | 1.00 | 4 | 1/5 | 5777 | 235/303 |
| 5 | `openrouter/openai/gpt-oss-120b:free@high` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 5 | 0/5 | 2299 | 0/0 |
| 6 | `openrouter/openai/gpt-oss-20b:free@high` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 5 | 0/5 | 11058 | 0/0 |

## Rates

- Scored attempts: 12
- Pass / partial / incorrect: 10 / 2 / 0
- Pass rate (scored): 10/12
- Fixture pass rate (passed fixture groups / scored fixture groups): 10/12
- Overall mean score (pass=1, partial=0.5): —
- Protocol errors: 17
- Policy violations: 0
- Trace errors: 0
- Timeouts: 1
- Unavailable: 0
- Unsupported: 0

## Usage & timing

Token counts: **provider** when the runner reports them (OpenRouter); otherwise **estimated** (~chars/4) so CLI runs still get numbers.

- Prompt tokens: 2916
- Completion tokens: 2312
- Reasoning tokens: 0
- Total tokens: 5228
- Cached prompt tokens: 256
- Cache write tokens: 0
- Reported cost: 0
- Upstream inference cost: 0
- Attempts with token counts: 12 (provider 12, estimated 0)
- Answer bytes (sum): 0
- Attempt duration avg/min/max: 11519 / 1771 / 180002 ms

## Groups (single-sample result + mean score)

| Fixture | Runner | K | Passes | Passed | Mean | Best |
| --- | --- | ---: | ---: | --- | ---: | --- |
| gmod.ents-iterator.v1 | openrouter/openai/gpt-oss-120b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.ents-iterator.v1 | openrouter/openai/gpt-oss-120b:free@low | 1 | 1 | yes | — | pass |
| gmod.ents-iterator.v1 | openrouter/openai/gpt-oss-120b:free@medium | 1 | 1 | yes | — | pass |
| gmod.ents-iterator.v1 | openrouter/openai/gpt-oss-20b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.ents-iterator.v1 | openrouter/openai/gpt-oss-20b:free@low | 1 | 1 | yes | — | pass |
| gmod.ents-iterator.v1 | openrouter/openai/gpt-oss-20b:free@medium | 1 | 0 | no | — | timeout |
| gmod.hook-add.v1 | openrouter/openai/gpt-oss-120b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.hook-add.v1 | openrouter/openai/gpt-oss-120b:free@low | 1 | 0 | no | — | protocol_error |
| gmod.hook-add.v1 | openrouter/openai/gpt-oss-120b:free@medium | 1 | 1 | yes | — | pass |
| gmod.hook-add.v1 | openrouter/openai/gpt-oss-20b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.hook-add.v1 | openrouter/openai/gpt-oss-20b:free@low | 1 | 1 | yes | — | pass |
| gmod.hook-add.v1 | openrouter/openai/gpt-oss-20b:free@medium | 1 | 1 | yes | — | pass |
| gmod.isvalid.v1 | openrouter/openai/gpt-oss-120b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.isvalid.v1 | openrouter/openai/gpt-oss-120b:free@low | 1 | 0 | no | — | protocol_error |
| gmod.isvalid.v1 | openrouter/openai/gpt-oss-120b:free@medium | 1 | 0 | no | — | protocol_error |
| gmod.isvalid.v1 | openrouter/openai/gpt-oss-20b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.isvalid.v1 | openrouter/openai/gpt-oss-20b:free@low | 1 | 1 | yes | — | pass |
| gmod.isvalid.v1 | openrouter/openai/gpt-oss-20b:free@medium | 1 | 1 | yes | — | pass |
| gmod.net-receivers.v1 | openrouter/openai/gpt-oss-120b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.net-receivers.v1 | openrouter/openai/gpt-oss-120b:free@low | 1 | 0 | no | — | protocol_error |
| gmod.net-receivers.v1 | openrouter/openai/gpt-oss-120b:free@medium | 1 | 0 | no | — | protocol_error |
| gmod.net-receivers.v1 | openrouter/openai/gpt-oss-20b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.net-receivers.v1 | openrouter/openai/gpt-oss-20b:free@low | 1 | 1 | yes | — | pass |
| gmod.net-receivers.v1 | openrouter/openai/gpt-oss-20b:free@medium | 1 | 1 | yes | — | pass |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-120b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-120b:free@low | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-120b:free@medium | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-20b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-20b:free@low | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-20b:free@medium | 1 | 0 | no | — | partial |

## Attempts

| Fixture | Adapter | Model | # | Status | ms | P/C tok | Reason tok | Src | Bytes | Body | Detail |
| --- | --- | --- | ---: | --- | ---: | ---: | ---: | --- | ---: | --- | --- |
| gmod.ents-iterator.v1 | openrouter | openai/gpt-oss-20b:free@high | 1 | protocol_error | 10994 | —/— | — | — | — | no | OpenRouter error: Internal Server Error |
| gmod.hook-add.v1 | openrouter | openai/gpt-oss-20b:free@high | 1 | protocol_error | 11886 | —/— | — | — | — | no | OpenRouter error: Internal Server Error |
| gmod.isvalid.v1 | openrouter | openai/gpt-oss-20b:free@high | 1 | protocol_error | 10901 | —/— | — | — | — | no | OpenRouter error: Internal Server Error |
| gmod.net-receivers.v1 | openrouter | openai/gpt-oss-20b:free@high | 1 | protocol_error | 10787 | —/— | — | — | — | no | OpenRouter error: Internal Server Error |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-20b:free@high | 1 | protocol_error | 10723 | —/— | — | — | — | no | OpenRouter error: Internal Server Error |
| gmod.ents-iterator.v1 | openrouter | openai/gpt-oss-20b:free@medium | 1 | timeout | 180002 | —/— | — | — | — | no | HTTP request exceeded the attempt timeout. |
| gmod.hook-add.v1 | openrouter | openai/gpt-oss-20b:free@medium | 1 | pass | 2564 | 249/181 | — | — | — | yes | Registers a named hook with hook.Add. |
| gmod.isvalid.v1 | openrouter | openai/gpt-oss-20b:free@medium | 1 | pass | 4733 | 247/171 | — | — | — | yes | Uses IsValid for entity/object validity. |
| gmod.net-receivers.v1 | openrouter | openai/gpt-oss-20b:free@medium | 1 | pass | 3274 | 242/224 | — | — | — | yes | Matched a pass pattern. |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-20b:free@medium | 1 | partial | 4349 | 243/344 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.ents-iterator.v1 | openrouter | openai/gpt-oss-20b:free@low | 1 | pass | 3867 | 241/227 | — | — | — | yes | Matched a pass pattern. |
| gmod.hook-add.v1 | openrouter | openai/gpt-oss-20b:free@low | 1 | pass | 2115 | 249/73 | — | — | — | yes | Registers a named hook with hook.Add. |
| gmod.isvalid.v1 | openrouter | openai/gpt-oss-20b:free@low | 1 | pass | 4016 | 247/81 | — | — | — | yes | Uses IsValid for entity/object validity. |
| gmod.net-receivers.v1 | openrouter | openai/gpt-oss-20b:free@low | 1 | pass | 1771 | 242/76 | — | — | — | yes | Matched a pass pattern. |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-20b:free@low | 1 | partial | 2114 | 243/126 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.ents-iterator.v1 | openrouter | openai/gpt-oss-120b:free@high | 1 | protocol_error | 2408 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.hook-add.v1 | openrouter | openai/gpt-oss-120b:free@high | 1 | protocol_error | 2172 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.isvalid.v1 | openrouter | openai/gpt-oss-120b:free@high | 1 | protocol_error | 2322 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.net-receivers.v1 | openrouter | openai/gpt-oss-120b:free@high | 1 | protocol_error | 2103 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-120b:free@high | 1 | protocol_error | 2488 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.ents-iterator.v1 | openrouter | openai/gpt-oss-120b:free@medium | 1 | pass | 20979 | 235/368 | — | — | — | yes | Matched a pass pattern. |
| gmod.hook-add.v1 | openrouter | openai/gpt-oss-120b:free@medium | 1 | pass | 12154 | 243/138 | — | — | — | yes | Registers a named hook with hook.Add. |
| gmod.isvalid.v1 | openrouter | openai/gpt-oss-120b:free@medium | 1 | protocol_error | 2172 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.net-receivers.v1 | openrouter | openai/gpt-oss-120b:free@medium | 1 | protocol_error | 3777 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-120b:free@medium | 1 | protocol_error | 2019 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.ents-iterator.v1 | openrouter | openai/gpt-oss-120b:free@low | 1 | pass | 18157 | 235/303 | — | — | — | yes | Matched a pass pattern. |
| gmod.hook-add.v1 | openrouter | openai/gpt-oss-120b:free@low | 1 | protocol_error | 2630 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.isvalid.v1 | openrouter | openai/gpt-oss-120b:free@low | 1 | protocol_error | 2449 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.net-receivers.v1 | openrouter | openai/gpt-oss-120b:free@low | 1 | protocol_error | 2304 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-120b:free@low | 1 | protocol_error | 3345 | —/— | — | — | — | no | OpenRouter HTTP 429. |

## Files (audit pack)

Same run directory contains:

- `run.json` — full structured artifact (answers + per-attempt metrics; raw stdout stripped)
- `metadata.json` — run provenance + `usageTotals` (tokens, cost, timing)
- `leaderboard.json` / `leaderboard.csv` — fixture-normalized model ranking, diagnostics, tokens, and timing
- `attempts.jsonl` / `attempts.csv` — one row per attempt + tokens/timing/sizes
- `preds.jsonl` — SWE-style predictions with usage fields
- `responses/<adapter>/<model>/<fixture>--kN.txt` — graded answer files
- `raw/*.log` — redacted stdout/stderr when keepRaw is on (default)

Compare two models: `bun run bench compare --run <run.json> --model "A" --model "B"`