gmod-bench

Methodology

How we test AI models on real GMod Lua knowledge and turn their answers into a trustworthy leaderboard.

What we measure

gmod-bench measures how well AI models handle the GMod Lua questions addon developers care about: correct APIs, realms, security, lifecycle, networking, and performance. Every challenge and scoring rule is public, so results can be inspected instead of taken on trust.

The leaderboard helps you compare:

  • GMod API and realm knowledge
  • Safe, efficient addon coding choices
  • Consistency across model versions and reasoning levels

It is a practical comparison tool, not a claim that one score predicts every possible addon task.

Response contract

Before any rubric runs, shared validation requires exactly one final response of this form:

answer shape
```lua
<one code body>
```
Reason: <one line>
  • Fence language must be lua
  • Exactly one fenced code block, then reason text
  • Reason starts with Reason: and stays within the max line count
  • Loop-count and byte-cap constraints from the fixture contract
  • No tool calls — tool_calls become policy violations

CLI attempts use a fresh empty temp workspace and isolated profile directories. They require native non-interactive mode, deny-all agent tools (where supported), and structured traces.

Status taxonomy

Wrong GMod content and protocol failures are different things. Transport and format issues never get relabeled as “incorrect.”

pass

Scored answer matched a pass pattern or full plugin credit.

partial

Scored answer earned partial credit (plugin or graded partial).

incorrect

Well-formed answer that failed the oracle / rubric.

protocol_error

Format / contract failure (bad fence, missing Reason line, tools, etc.) or transport failure before fair grading. Shown as “format” on the leaderboard.

policy_violation

Tool or agent use detected when the contract forbids it.

trace_error

Wire format not understood — not the same as “CLI unsupported.”

timeout

Deadline exceeded for the attempt.

unavailable

Binary or API key missing; not graded as incorrect.

unsupported

Adapter not eligible for strict scoring in the reviewed configuration.

Metrics

Pass rate

pass / scored, where scored means pass + partial + incorrect. Format errors do not count as wrong answers in this rate.

Coverage

Fraction of attempts that produced a scorable answer. High pass rate with low coverage usually means the model often fails the contract or transport.

Fixture score (ranking metric)

Each attempt scores pass=1, partial=0.5, and every incorrect, format, policy, timeout, or transport failure=0. Repeated attempts are averaged inside their fixture first, then fixtures are averaged equally. Ranks compare only rows with the same fixture set and repeat count.

pass@k

With --repeat K, a fixture is solved if any of the K attempts is a pass. Reported as pass@k rate over fixtures. Use this for consistency, not luck of a single sample.

Scored quality (diagnostic)

Mean of pass=1, partial=0.5, incorrect=0 over scored attempts. Complements raw pass rate when partial credit is common.

Tokens & cost

When a provider reports token usage, the harness records prompt/completion totals and estimated cost on the leaderboard. Some adapters (notably certain CLI paths) may report zeros — that means “not available from the provider,” not “free.” Prefer provider-reported figures when present; treat missing usage as unknown.

OpenRouter free (:free) slots still consume free-tier RPM/RPD quota. Paid model ids skip the free limiter and may still hit provider 429s.

Runners

OpenRouter (HTTP)

  • Strict when OPENROUTER_API_KEY is present
  • Free mode: live catalog of pure text→text chat models at $0; RPM/RPD limiter from key tier
  • Paid mode: explicit model ids, no free RPM/RPD gate
  • Sampling fixed for fairness: temperature 0, top_p 1, fixed seed; tools never requested

CLI adapters

  • claude, gemini — strict when help/policy surface qualifies
  • AntiGravity — unsupported for strict scoring until the CLI exposes a reviewed deny-all tool policy and structured trace contract.
  • codex — strict only when the reviewed read-only, no-network, non-interactive controls are present.
  • grok, cursor, devin — unsupported until deny-all contracts are reviewed.

Fairness notes

  • Public fixtures — core GMod questions plus perf items with a clear winner from Facepunch wiki / gmod-lua-performance. Myths and ties are omitted.
  • No silent demotions — unavailable or unsupported runners are never treated as incorrect answers.
  • Free-model quarantine — endpoints that never return answer text can be skipped temporarily so dead free models do not burn the whole RPM budget.
  • Reproducibility — every run writes an audit pack (report, run.json with graded answers, attempts, responses, optional raw logs).

Full CLI details live in the repoREADME. Install steps are condensed on the docs page.