gmod-bench

Docs

Run the same GMod coding benchmark locally, compare your preferred models, and inspect every answer behind the scores.

Install

Requires Bun. Clone the monorepo and install dependencies:

shell
git clone https://github.com/Iydah/gmod-bench.git
cd gmod-bench
bun install

Optional for OpenRouter: copy .env.example to .env and set OPENROUTER_API_KEY. Bun loads .env automatically. Never commit secrets.

Doctor & list

doctor checks installed CLIs and OpenRouter key readiness (including free-quota plan). It never sends a model prompt.

shell
bun run bench doctor
bun run bench list
bun run bench list-models --free

Free OpenRouter

Uses the live free catalog (:free text chat models). Official free cap is 20 RPM; daily requests are 50 or1000 depending on whether the key has ≥$10 credits.

powershell
# Free OpenRouter (:free) — rate-limited
# requires OPENROUTER_API_KEY in .env
bun run bench run --fixture all --openrouter-free --concurrency 2

Free concurrency is clamped (typically ≤4). Dead free endpoints that never return text can enter a short quarantine so they stop burning budget:

shell
bun run bench quarantine
bun run bench quarantine --clear
bun run bench quarantine --clear provider/model:free

Uses explicit paid model ids. The free RPM/RPD limiter does not apply; providers may still 429. Same fixtures, scorers, and reports as free mode.

powershell
# Paid OpenRouter — no free RPM/RPD limiter
bun run bench run --fixture all --runners openrouter --concurrency 8 `
  --model openrouter=openai/gpt-4o-mini `
  --model openrouter=anthropic/claude-sonnet-4 `
  --model openrouter=google/gemini-2.5-flash

You can mix free and paid ids in one run — the limiter only applies to free-tier slots.

AntiGravity

Display name is always AntiGravity. The CLI binary and--runners id are still agy. When the CLI is ≥1.1.x and strict-capable, the harness stays unsupported for strict scoring until a reviewed deny-all tools policy and structured trace contract are available.

shell
# AntiGravity (CLI id: agy) when installed & strict-capable
bun run bench run --fixture all --runners agy --repeat 5 --concurrency 6

Without an explicit --concurrency, AntiGravity defaults high for paid Gemini tiers (~25). If you hit 429s on free Gemini API, drop to 2–4.

Thinking levels

Some models expose a reasoning / thinking effort (None, Low, Medium, High, Extra High, Max, Ultra). On the leaderboard that lives in its ownThink column. Use the Think filter to compare efforts.

  • AntiGravity Gemini rows often look like Gemini 3.1 Pro (Low)in raw logs — the site shows model + thinking apart.
  • OpenRouter free slots may append @low /@medium / @high /@none to the model id.
  • Rows with no thinking signal show a dash in Think.

Concurrency notes

  • Bounds: --repeat 1–20,--timeout-seconds 1–600,--concurrency 1–32
  • OpenRouter free: clamp ≤4 (20 RPM free cap)
  • OpenRouter paid / explicit flag: whatever you pass (≤32)
  • AntiGravity default (no flag): auto ~25 for paid Gemini-class throughput

Concurrency is for rate limits, not a RAM guess. Prefer lower values when debugging format errors under load.

Resume, report, compare

powershell
bun run bench run --fixture all --openrouter-free --concurrency 2 `
  --resume-from .gmod-bench/runs/<checkpoint-or-run-id>

bun run bench report --run .gmod-bench/runs/<run-id>/run.json
bun run bench compare --run .gmod-bench/runs/<id> `
  --model "Model A" --model "Model B"

--resume-from keeps prior attempts and only runs remaining slots after a crash or interrupt.

Artifacts

Each run writes an audit pack under.gmod-bench/runs/<run-id>/ (gitignored):

  • report.md — human leaderboard + attempts
  • run.json — structured artifact (schema v3) with graded answers
  • leaderboard.json / .csv
  • attempts.jsonl, preds.jsonl,responses/, optional raw/

Quality gate in the monorepo: bun run check (typecheck + tests). Site-only work lives under website/ and does not affect the harness.

Source:https://github.com/Iydah/gmod-bench· Scoring: methodology· Results: leaderboard