ARCANADual-Suite Benchmark
Three CLI AI agents. One model. Two benchmark suites. We measured correctness, speed, tool efficiency, code performance, self-correction, and over-engineering risk across 5 real-world tasks — a coding implementation and four domain-knowledge challenges. All agents ran deepseek-v4-flash, cold-start, single-attempt, identical prompts, identical verification.
Speed, measured.
Lower is better. Arcana (cobalt) against OpenCode and Hermes. Domain suite totals and coding-suite resolution time, cold start, same model.
Async Semaphore.
Implement a concurrency primitive with FIFO queuing, capacity enforcement, and exception safety. 10 automated tests covering edge cases.
| Metric | OpenCode | Hermes | Arcana | Best |
|---|---|---|---|---|
| Correctness | 10/10✓ | 9/10✗ | 10/10✓ | O + A |
| Resolution Time | 24.2s | 116s | ~10s | Arcana |
| Tool Calls | 4 | 24 | 2 | Arcana |
| Code Perf (50k ops) | 7.6ms | 7.0ms | 6.9ms | Arcana |
| Self-correction | First-try✓ | Failed test3✗ | First-try✓ | O + A |
| Over-engineering | None✓ | Replaced tests✗ | None✓ | O + A |
| Lines of Code | 43 | 43 | 42 | Arcana |
Key finding: Hermes over-engineered — replaced the pre-existing test file with its own simplified tests (24 tool calls), then still missed the release-without-acquire edge case. OpenCode and Arcana both clean single-attempt passes.
* Arcana resolution time measured via file-detection on disk (code writes complete before process signals exit). OpenCode/Hermes measured via process exit.
Math. Science. Biology. History.
Four tasks requiring factual knowledge + tool-based verification. Same model means same factual accuracy — the difference is how agents use tools to verify.
| Task | OpenCode | Hermes | Arcana | Fastest |
|---|---|---|---|---|
| Math (sum of primes) | 11.6s✓ | 18.1s✓ | 3.4s | Arcana |
| Science (escape velocity) | 10.5s✓ | 16.5s✓ | 4.4s | Arcana |
| Biology (DNA verification) | 16.2s✓ | 37.0s✓ | 14.1s | Arcana |
| History (chronological order) | 9.8s✓ | 14.3s✓ | 6.5s | Arcana |
| Total Time | 48.1s | 86.9s | 28.4s | Arcana |
| Tasks Passed | 4/4✓ | 4/4✓ | 4/4✓ | All |
Arcana wins all 4 tasks on speed. Its compiled binary delivers 3–14s resolution times. OpenCode consistent at 10–16s. Hermes slower but reliable — 37s on biology due to extra tool calls for verification.
Why these results?
Three agents, three philosophies. Speed comes from runtime; reliability comes from discipline; over-engineering comes from unchecked tool budgets.
Arcana Raw Speed
Arcana's compiled binary gives a structural advantage: faster startup (0.57s vs 0.96s Node, 1.08s Python), zero-overhead tool dispatch, minimal scaffolding. On the coding task it needed only 2 tool calls (goal_set + file write) vs OpenCode's 4 and Hermes' 24.
Caveat: Arcana's measurement required a workaround — the process doesn't signal completion natively, so we measured file-creation time on disk rather than process exit. Reported times are actual code-generation time, minus session teardown that OpenCode and Hermes include.
OpenCode Lean & Reliable
Purpose-built for code tasks. On the Semaphore it used exactly 4 tool calls: glob to find the test file, read to understand it, write to create the implementation, bash to run the tests. No wasted operations. Discipline: reads existing files before writing, runs tests after writing, doesn't get distracted by peripheral tasks.
Hermes Undisciplined
24 tool calls on the Semaphore tell the story. Instead of reading the existing test file and implementing against it, Hermes created its own test suite, a package.json, and rewrote the project infrastructure — 116s, 5× longer than OpenCode — and still missed release-without-acquire. The bug: release() checked this.current === 0 instead of this.current <= 0, a single-character error the original test suite would have caught.
Code Performance Why So Close?
All three within 10% on runtime (6.9–7.6ms for 50,000 acquire/release ops): the generated code is nearly identical — a class with a counter and a queue. Differences come from micro-optimizations: how release() checks the queue, whether promises are chained or awaited, constructor guard order. Arcana fastest (6.9ms) using _out as a single counter and checking the queue before decrementing.
Six principles for honest benchmarking.
Task Complexity
Real-world tasks: concurrent programming, physics computation, DNA verification, temporal reasoning. Not toy problems.
Environment
Isolated workspaces. Identical prompts. Same model. Cold start — no cache, no pre-existing files.
Verification
Automated tests (Semaphore: 10 tests) and file-output verification (Domain: code execution). Exit code == pass.
Agent Behavior
Measured: cold startup, tool execution, AI resolution, code performance. All single-attempt.
Reproducibility
Full reproduce.sh script. SHA256-locked test file. All implementations archived.
Transparency
No cherry-picking. No hidden failures. Hermes ran as subprocess. All raw logs available.
Arcana
Fastest across both suites. 28.4s domain total — 3× faster than Hermes, 1.7× faster than OpenCode. Clean implementations, no over-engineering. Correct on every task — 10/10 coding, 4/4 domain.