# GMod Bench report

Run: `a9024895-be0b-4ae8-8ce8-146ec0e338f9`
Fixtures: `gmod.player-iterator.v1`
Repeat: 1 · Concurrency: 2
Started: 2026-07-10T12:56:54.758Z · Completed: 2026-07-10T13:32:24.698Z
Wall clock: 2130s · Sum of attempt durations: 451331ms (avg 14559ms, min 804ms, max 120002ms)

## Model leaderboard

Sorted by **fixture score** (each fixture weighted equally; failures score zero), then fixture solve rate, coverage, and pass rate.

| Rank | Runner / model | Fixture score | Fixture solve | Pass | Partial | Incorrect | Scored | Coverage | Pass rate | Fixture pass | Scored quality | Protocol err | Fixtures | Avg ms | Prompt/Compl tok |
| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | `openrouter/nvidia/nemotron-3-super-120b-a12b:free@low` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 3220 | 185/415 |
| 2 | `openrouter/nvidia/nemotron-3-ultra-550b-a55b:free@high` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 5041 | 185/838 |
| 3 | `openrouter/nvidia/nemotron-3-ultra-550b-a55b:free@medium` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 11829 | 185/643 |
| 4 | `openrouter/openai/gpt-oss-120b:free@high` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 60301 | 237/945 |
| 5 | `openrouter/openai/gpt-oss-20b:free@low` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 6209 | 243/126 |
| 6 | `openrouter/openai/gpt-oss-20b:free@medium` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 10723 | 243/508 |
| 7 | `openrouter/poolside/laguna-xs-2.1:free` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 804 | 180/61 |
| 8 | `openrouter/tencent/hy3:free@high` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 42093 | 179/1632 |
| 9 | `openrouter/tencent/hy3:free@low` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 21708 | 179/797 |
| 10 | `openrouter/tencent/hy3:free@none` | 50.0% | 0.0% | 0 | 1 | 0 | 1 | 100.0% | 0.0% | 0.0% | 0.50 | 0 | 0/1 | 3319 | 182/64 |
| 11 | `openrouter/cohere/north-mini-code:free` | 0.0% | 0.0% | 0 | 0 | 1 | 1 | 100.0% | 0.0% | 0.0% | 0.00 | 0 | 0/1 | 11235 | 163/1008 |
| 12 | `openrouter/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 0 | 0/1 | 120002 | 0/0 |
| 13 | `openrouter/cognitivecomputations/dolphin-mistral-24b-venice-edition:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 2013 | 0/0 |
| 14 | `openrouter/google/gemma-4-26b-a4b-it:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 21706 | 169/1021 |
| 15 | `openrouter/google/gemma-4-31b-it:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 4398 | 0/0 |
| 16 | `openrouter/liquid/lfm-2.5-1.2b-instruct:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 2064 | 0/0 |
| 17 | `openrouter/liquid/lfm-2.5-1.2b-thinking:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 1974 | 0/0 |
| 18 | `openrouter/meta-llama/llama-3.2-3b-instruct:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 2035 | 0/0 |
| 19 | `openrouter/meta-llama/llama-3.3-70b-instruct:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 2270 | 0/0 |
| 20 | `openrouter/nousresearch/hermes-3-llama-3.1-405b:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 2024 | 0/0 |
| 21 | `openrouter/nvidia/nemotron-3-nano-30b-a3b:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 11601 | 185/1024 |
| 22 | `openrouter/nvidia/nemotron-3-super-120b-a12b:free@medium` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 4224 | 185/466 |
| 23 | `openrouter/nvidia/nemotron-3.5-content-safety:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 3212 | 529/264 |
| 24 | `openrouter/nvidia/nemotron-nano-12b-v2-vl:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 28755 | 180/1024 |
| 25 | `openrouter/nvidia/nemotron-nano-9b-v2:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 35284 | 180/1024 |
| 26 | `openrouter/openai/gpt-oss-120b:free@low` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 3929 | 0/0 |
| 27 | `openrouter/openai/gpt-oss-120b:free@medium` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 4165 | 0/0 |
| 28 | `openrouter/openai/gpt-oss-20b:free@high` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 11397 | 0/0 |
| 29 | `openrouter/poolside/laguna-m.1:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 8569 | 189/322 |
| 30 | `openrouter/qwen/qwen3-coder:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 2537 | 0/0 |
| 31 | `openrouter/qwen/qwen3-next-80b-a3b-instruct:free` | 0.0% | 0.0% | 0 | 0 | 0 | 0 | 0.0% | 0.0% | 0.0% | 0.00 | 1 | 0/1 | 2690 | 0/0 |

## Rates

- Scored attempts: 11
- Pass / partial / incorrect: 0 / 10 / 1
- Pass rate (scored): 0/11
- Fixture pass rate (passed fixture groups / scored fixture groups): 0/11
- Overall mean score (pass=1, partial=0.5): —
- Protocol errors: 19
- Policy violations: 0
- Trace errors: 0
- Timeouts: 1
- Unavailable: 0
- Unsupported: 0

## Usage & timing

Token counts: **provider** when the runner reports them (OpenRouter); otherwise **estimated** (~chars/4) so CLI runs still get numbers.

- Prompt tokens: 3778
- Completion tokens: 12182
- Reasoning tokens: 0
- Total tokens: 15960
- Cached prompt tokens: 32
- Cache write tokens: 0
- Reported cost: 0
- Upstream inference cost: 0
- Attempts with token counts: 18 (provider 18, estimated 0)
- Answer bytes (sum): 0
- Attempt duration avg/min/max: 14559 / 804 / 120002 ms

## Groups (single-sample result + mean score)

| Fixture | Runner | K | Passes | Passed | Mean | Best |
| --- | --- | ---: | ---: | --- | ---: | --- |
| gmod.player-iterator.v1 | openrouter/cognitivecomputations/dolphin-mistral-24b-venice-edition:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/cohere/north-mini-code:free | 1 | 0 | no | — | incorrect |
| gmod.player-iterator.v1 | openrouter/google/gemma-4-26b-a4b-it:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/google/gemma-4-31b-it:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/liquid/lfm-2.5-1.2b-instruct:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/liquid/lfm-2.5-1.2b-thinking:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/meta-llama/llama-3.2-3b-instruct:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/meta-llama/llama-3.3-70b-instruct:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/nousresearch/hermes-3-llama-3.1-405b:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-3-nano-30b-a3b:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free | 1 | 0 | no | — | timeout |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-3-super-120b-a12b:free@low | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-3-super-120b-a12b:free@medium | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-3-ultra-550b-a55b:free@high | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-3-ultra-550b-a55b:free@medium | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-3.5-content-safety:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-nano-12b-v2-vl:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/nvidia/nemotron-nano-9b-v2:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-120b:free@high | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-120b:free@low | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-120b:free@medium | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-20b:free@high | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-20b:free@low | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/openai/gpt-oss-20b:free@medium | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/poolside/laguna-m.1:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/poolside/laguna-xs-2.1:free | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/qwen/qwen3-coder:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/qwen/qwen3-next-80b-a3b-instruct:free | 1 | 0 | no | — | protocol_error |
| gmod.player-iterator.v1 | openrouter/tencent/hy3:free@high | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/tencent/hy3:free@low | 1 | 0 | no | — | partial |
| gmod.player-iterator.v1 | openrouter/tencent/hy3:free@none | 1 | 0 | no | — | partial |

## Attempts

| Fixture | Adapter | Model | # | Status | ms | P/C tok | Reason tok | Src | Bytes | Body | Detail |
| --- | --- | --- | ---: | --- | ---: | ---: | ---: | --- | ---: | --- | --- |
| gmod.player-iterator.v1 | openrouter | cognitivecomputations/dolphin-mistral-24b-venice-edition:free | 1 | protocol_error | 2013 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | cohere/north-mini-code:free | 1 | incorrect | 11235 | 163/1008 | — | — | — | yes | Keeps pairs() rather than using the direct iterator. |
| gmod.player-iterator.v1 | openrouter | google/gemma-4-26b-a4b-it:free | 1 | protocol_error | 21706 | 169/1021 | — | — | — | no | OpenRouter message content was empty or missing. |
| gmod.player-iterator.v1 | openrouter | google/gemma-4-31b-it:free | 1 | protocol_error | 4398 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | liquid/lfm-2.5-1.2b-instruct:free | 1 | protocol_error | 2064 | —/— | — | — | — | no | OpenRouter HTTP 502. |
| gmod.player-iterator.v1 | openrouter | liquid/lfm-2.5-1.2b-thinking:free | 1 | protocol_error | 1974 | —/— | — | — | — | no | OpenRouter HTTP 502. |
| gmod.player-iterator.v1 | openrouter | meta-llama/llama-3.2-3b-instruct:free | 1 | protocol_error | 2035 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | meta-llama/llama-3.3-70b-instruct:free | 1 | protocol_error | 2270 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | nousresearch/hermes-3-llama-3.1-405b:free | 1 | protocol_error | 2024 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-3-nano-30b-a3b:free | 1 | protocol_error | 11601 | 185/1024 | — | — | — | no | Final response exceeds fixture byte cap of 2048. |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free | 1 | timeout | 120002 | —/— | — | — | — | no | HTTP request exceeded the attempt timeout. |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-3-super-120b-a12b:free@medium | 1 | protocol_error | 4224 | 185/466 | — | — | — | yes | Answer must contain exactly one fenced code block followed by a reason. |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-3-super-120b-a12b:free@low | 1 | partial | 3220 | 185/415 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-3-ultra-550b-a55b:free@high | 1 | partial | 5041 | 185/838 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-3-ultra-550b-a55b:free@medium | 1 | partial | 11829 | 185/643 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-3.5-content-safety:free | 1 | protocol_error | 3212 | 529/264 | — | — | — | yes | Answer must contain exactly one fenced code block followed by a reason. |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-nano-12b-v2-vl:free | 1 | protocol_error | 28755 | 180/1024 | — | — | — | no | OpenRouter message content was empty or missing. |
| gmod.player-iterator.v1 | openrouter | nvidia/nemotron-nano-9b-v2:free | 1 | protocol_error | 35284 | 180/1024 | — | — | — | no | OpenRouter message content was empty or missing. |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-120b:free@high | 1 | partial | 60301 | 237/945 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-120b:free@medium | 1 | protocol_error | 4165 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-120b:free@low | 1 | protocol_error | 3929 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-20b:free@high | 1 | protocol_error | 11397 | —/— | — | — | — | no | OpenRouter error: Internal Server Error (code 500) |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-20b:free@medium | 1 | partial | 10723 | 243/508 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | openai/gpt-oss-20b:free@low | 1 | partial | 6209 | 243/126 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | poolside/laguna-m.1:free | 1 | protocol_error | 8569 | 189/322 | — | — | — | yes | Answer must contain exactly one fenced code block followed by a reason. |
| gmod.player-iterator.v1 | openrouter | poolside/laguna-xs-2.1:free | 1 | partial | 804 | 180/61 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | qwen/qwen3-coder:free | 1 | protocol_error | 2537 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | qwen/qwen3-next-80b-a3b-instruct:free | 1 | protocol_error | 2690 | —/— | — | — | — | no | OpenRouter HTTP 429. |
| gmod.player-iterator.v1 | openrouter | tencent/hy3:free@high | 1 | partial | 42093 | 179/1632 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | tencent/hy3:free@low | 1 | partial | 21708 | 179/797 | — | — | — | yes | Still materializes player.GetAll(). |
| gmod.player-iterator.v1 | openrouter | tencent/hy3:free@none | 1 | partial | 3319 | 182/64 | — | — | — | yes | Still materializes player.GetAll(). |

## Files (audit pack)

Same run directory contains:

- `run.json` — full structured artifact (answers + per-attempt metrics; raw stdout stripped)
- `metadata.json` — run provenance + `usageTotals` (tokens, cost, timing)
- `leaderboard.json` / `leaderboard.csv` — fixture-normalized model ranking, diagnostics, tokens, and timing
- `attempts.jsonl` / `attempts.csv` — one row per attempt + tokens/timing/sizes
- `preds.jsonl` — SWE-style predictions with usage fields
- `responses/<adapter>/<model>/<fixture>--kN.txt` — graded answer files
- `raw/*.log` — redacted stdout/stderr when keepRaw is on (default)

Compare two models: `bun run bench compare --run <run.json> --model "A" --model "B"`