OpenRelay
I built this to find out if I could make an AI coding harness use fewer tokens and finish faster without the quality dropping. The goal was simple: get the most useful finished work per premium-model token.
I used the GLM Coding Plan as the workhorse and a ChatGPT subscription as the premium tier. OAuth only, never API or pay-as-you-go. The plan was to ship it all as a plugin for OpenCode.
The short version
- Output filtering was the only clear win. 72% fewer input tokens on a noisy fixture, with no recovery calls.
- Everything else was marginal or lost. Context retrieval helped a little. Memory, escalation and repo maps did not.
- Saving tokens isn’t enough. Extra turns and latency often erased the saving.
What it does
The repo holds two things:
- The runtime: an OpenCode plugin with telemetry, output filtering, context selection and routing. Routine work goes to the cheaper model (GLM). The premium model (ChatGPT) is kept for the hard parts.
- A benchmark system: a reproducible runner that compares each change against independent task verifiers.
The design
Four parts, each answering one question. This is the first sketch:
You
|
Observability v
- tokens/task OpenCode orchestration
- quality/task |
- allowance v
pressure Task controller
- retrieval - classify workflow
efficiency - select agent
- spend - manage handoff
- force/deep
/ \
v v
Context engine <------> Tool layer
- repo manager - git / rg / LSP
- context compiler - AST / tree-sitter
- retrieval - tests / lint / compiler
- filtering - format
- compression
- project memory
|
| compact, reversible context
v
Agent layer ---------> LiteLLM proxy ---> OpenRouter PAYG
- Claude - model gateway (last resort)
- Codex - routing/feedback
- GLM - accounting
/ | \
Claude Codex GLM| Part | Question | What’s inside |
|---|---|---|
| Task controller | Which model does the work? | Classifier (routine or complex, simple if/else rules), router, monitor (progress, failures, retries) and escalator (GLM stuck, send it to ChatGPT). |
| Context engine | What does the model need? | Retriever, budgeter, packet builder, handoff builder, expander (fetch more if something’s missing) and a trust layer that separates exact code and requirements from stale summaries. |
| Tool layer | What can tools find without tokens? | ripgrep, LSP, Git, OpenCode’s own edit tools, lint, type checks and tests. Plus tool-output filtering: send only the failing test, the stack trace, the pass and fail count and the exit code. |
| Observability | Did it work? | Tokens and quality per task, allowance pressure, retrieval efficiency, spend. |
Project memory stores the architecture, decisions and task progress, so the model doesn’t rediscover everything each time.
Most of this is a plan, not what shipped. Only filtering, the context engine and routing shipped. See “What shipped” below.
How I tested it
Every idea had to pass the same gate before it went into the plugin:
- Fixture corpus: the same set of tasks for every change.
- Independent verifier: checks the quality of the result, not the agent.
- Telemetry: every run recorded and joined by session ID.
- Pre-registered gates: set before the run, so the result is Pass, Fail or Inconclusive. No moving the goalposts.
Token use and total task time are measured separately. An inconclusive result does not get added to the runtime.
What the evidence says
| Stage | What I tried | Result | Status |
|---|---|---|---|
| 0 to 1. Measurement | Capability map, telemetry, runner | Built. A/A noise floor never run. | Infrastructure |
| 2. Output filtering | Strip noise from tool output | Pass. 72% fewer input plus cache-read tokens on the noisy fixture, no recovery calls. Re-run showed 49%. | In the plugin |
| 3. Output discipline | Make the model write less | Fail or inconclusive. | Removed |
| 4. Task controller | GLM-first routing | Quality kept, zero premium calls. Escalation never exercised. | Partial pass |
| 5. Context retrieval | Pre-select relevant code | 15.2% saving on UI tasks, short of the 25% gate. Verifiers passed 36 of 36 and 12 of 12. Latency mixed. | Limited rollout |
| 5. Broader retrieval | Wider file and span retrieval | Good file ranking, missed the correct-span and abstention gates. | Not integrated |
| 6. Memory and handoffs | Carry context between sessions | All 29 runs passed. Handoff changed tokens by -15%, +23% and +6% depending on the task. | Off |
| 7. Premium escalation | Hand hard tasks up to the premium model | All 12 passed, but escalation took about 2x as long as premium-first (74 to 75 s vs 36 s). | Stopped |
| 8. Repository maps | Give the model a map of the repo | Live run used more tokens and time. | Not integrated |
| Pi harness | Same ideas on a second harness | Test was invalid, so no result. | Stopped |
These are small samples. Treat them as notes from my own runs, not guarantees.
What shipped
- Output filtering: failure-card filtering, collapsing passing results and cleaning metadata. On by default.
- Context engine: on in my daily setup, with a switch to turn it off and a rollback build. That was my call, not a passed gate.
- GLM-first routing: deterministic, from Stage 4.
Everything else is off or never integrated.
What went wrong
Only output filtering was a clear win. It scales with how noisy the tool output is. Everything else was marginal or lost.
Saving tokens isn’t enough. Extra turns and latency wiped out the token savings in filtering variants, escalation and repo maps. A change that saves tokens but slows the task down is not a win, so I measure both.
Context helps only where retrieval is cheap and clear. UI and debugging tasks gained. Git tasks and trivial tasks lost, so the selector skips them.
Small samples are noise. A single run per cell told me nothing. I needed repeated, interleaved A/B pairs with A/A controls to say anything, and I never ran the A/A calibration.
My harness broke two runs.
- Stage 6: sessions shared state and contaminated each other.
- Pi: the test workspace sat inside the repo, so the agent could see the verifier.
Lesson: keep verifiers out of any path the agent can reach.
Known limits
- Title calls are invisible: usage from them can’t be seen on the supported surface, so all-call savings are partial.
- Latency monitoring is still open for the daily rollout.
- Escalation recovery was never validated live.
- Some holdouts got reused as development data.
What I’d do differently
- Run the A/A calibration first, so I know the noise floor before reading any result.
- Isolate the verifier from day one.
- Use fresh holdouts before adding any retrieval change.
- Start with output filtering and stop earlier on ideas that only save tokens.
Where it stands
It didn’t make a dramatic difference, but I got a clear answer: filter the noise out of tool output and be sceptical of the rest. The rigour was the useful part. Gates set in advance and an independent verifier stopped me from shipping things that only looked good.
The project is still going. The filtering and context engine run in my daily setup, and the open items above are what’s left to look at.