Three CLI AI agents. One model. Two benchmark suites. We measured correctness, speed, tool efficiency, code performance, self-correction, and over-engineering risk across 5 real-world tasks: a coding implementation and four domain-knowledge challenges. All agents used deepseek-v4-flash. All runs were cold-start, single-attempt, with identical prompts and verification.
Versions: Arcana v0.3.66 · OpenCode 1.18.5 · Hermes v0.15.1 · Bun 1.3.14 · Node.js v24.14.1 · Python 3.11.2
Suite 1 — Coding: Async Semaphore
Implement a concurrency primitive with FIFO queuing, capacity enforcement, and exception safety. 10 automated tests covering edge cases.
| Metric | OpenCode | Hermes | Arcana |
|---|---|---|---|
| Correctness | 10/10 ✓ | 9/10 ✗ | 10/10 ✓ |
| Resolution Time | 24.2s | 116s | ~10s |
| Tool Calls | 4 | 24 | 2 |
| Code Perf (50k ops) | 7.6ms | 7.0ms | 6.9ms |
| Self-correction | First-try ✓ | Failed test3 ✗ | First-try ✓ |
| Over-engineering | None | Replaced tests | None |
Key finding: Hermes over-engineered — replaced the pre-existing test file with its own simplified tests (24 tool calls), then still missed the release-without-acquire edge case. OpenCode and Arcana both clean single-attempt passes.
* Arcana resolution time measured via file-detection on disk (code writes complete before process signals exit). OpenCode/Hermes measured via process exit.
Suite 2 — Domain: Knowledge + Tool Verification
Four tasks requiring factual knowledge + tool-based verification. Same model means same factual accuracy — the difference is how agents use tools to verify.
| Task | OpenCode | Hermes | Arcana |
|---|---|---|---|
| Math (sum of primes) | 11.6s ✓ | 18.1s ✓ | 3.4s ✓ |
| Science (escape velocity) | 10.5s ✓ | 16.5s ✓ | 4.4s ✓ |
| Biology (DNA verification) | 16.2s ✓ | 37.0s ✓ | 14.1s ✓ |
| History (chronological order) | 9.8s ✓ | 14.3s ✓ | 6.5s ✓ |
| Total Time | 48.1s | 86.9s | 28.4s |
Arcana wins all 4 tasks on speed. Its compiled binary delivers 3-14s resolution times. OpenCode consistent at 10-16s. Hermes slower but reliable — 37s on biology due to extra tool calls for verification.
Analysis — Why These Results?
Arcana — Raw Speed
Arcana's compiled binary gives it a structural advantage: faster startup (0.57s vs 0.96s Node, 1.08s Python), zero-overhead tool dispatch, and minimal scaffolding. On the coding task, it needed only 2 tool calls (goal_set + file write) compared to OpenCode's 4 and Hermes' 24. On domain tasks, it generated answers and code with minimal tool overhead. However, Arcana's measurement required a workaround — the process doesn't signal completion natively, so we measured file-creation time on disk rather than process exit time. This means the reported times are the actual code generation time, minus session teardown overhead that OpenCode and Hermes include.
OpenCode — Lean and Reliable
OpenCode's architecture is purpose-built for code tasks. On the Semaphore, it used exactly 4 tool calls: glob to find the test file, read to understand it, write to create the implementation, and bash to run the tests. No wasted operations. On domain tasks, it consistently wrote verification code and executed it — never skipped a step. Its Node.js runtime is faster than Python for tool dispatch. OpenCode's key strength is discipline: it reads existing files before writing, runs tests after writing, and doesn't get distracted by peripheral tasks.
Hermes — Powerful but Undisciplined
Hermes' 24 tool calls on the Semaphore task tell the story. Instead of reading the existing test file and implementing against it, Hermes created its own test suite, a package.json, and rewrote the entire project infrastructure. This over-engineering consumed 116 seconds — 5× longer than OpenCode — and still missed the release-without-acquire edge case (test3). The bug: Hermes' release() checked this.current === 0 instead of this.current <= 0, a single-character error that would have been caught by the original test suite. On domain tasks, Hermes was thorough — it verified every answer with code and file reads — but the extra diligence cost time (37s on biology vs Arcana's 14s). Hermes' Python runtime adds startup overhead (1.08s) but produces the most readable, commented code.
Code Performance — Why Close?
All three implementations scored within 10% on runtime performance (6.9–7.6ms for 50,000 acquire/release operations). This makes sense: the generated code is nearly identical — a class with a counter and a queue. The performance differences come from micro-optimizations: how release() checks the queue, whether promises are chained or awaited, and constructor guard order. Arcana's implementation was fastest (6.9ms) because it used _out as a single counter and checked the queue before decrementing — the most efficient path. Hermes' was close (7.0ms) with a clean count-based approach. OpenCode's thunk-based queuing added a small overhead (7.6ms) but is the most established pattern for correctness under concurrent access.
Methodology
We applied six principles for honest benchmarking:
1. Task Complexity
Real-world tasks: concurrent programming, physics computation, DNA verification, temporal reasoning. Not toy problems.
2. Environment
Isolated workspaces. Identical prompts. Same model. Cold start — no cache, no pre-existing files.
3. Verification
Automated tests (Semaphore: 10 tests) and file-output verification (Domain: code execution). Exit code == pass.
4. Agent Behavior
Measured: cold startup, tool execution, AI resolution, code performance. All single-attempt.
5. Reproducibility
Full reproduce.sh script. SHA256-locked test file. All implementations archived.
6. Transparency
No cherry-picking. No hidden failures. Hermes ran as subprocess. All raw logs available.
Verdict
🏆 Arcana — fastest across both suites. 28.4s domain total (3× faster than Hermes, 1.7× faster than OpenCode). Clean implementations, no over-engineering. Correct on 9/10 tasks.
OpenCode — most consistent. 10/10 on both suites. First-try correctness on coding. No over-engineering. Zero failures.
Hermes — capable but prone to over-engineering. 8/10 tasks passed. Correct on all domain tasks but used 24 tool calls on coding and rewrote the test suite. Needs guardrails for tool discipline.