Methodology

← Back to benchmark

1. Task Complexity

Real-world async concurrency primitive (Semaphore) with 10 automated tests covering:

• Constructor validation (8 invalid inputs)
• Concurrent capacity enforcement (10 concurrent tasks)
• FIFO ordering guarantees
• Exception safety (permit leak prevention)
• Stress testing (1000 tasks, capacity 10)
• Release overflow detection

2. Environment

• Isolated Node.js workspaces per agent
• Identical test suite (semaphore.test.js) for all
• Cold start — no cached sessions, no pre-existing files
• Same model family (deepseek-v4-flash/pro)

3. Automated Verification

• 10 deterministic tests run via node semaphore.test.js
• Exit code 0 = all pass
• Timed from prompt to verified output

4. Agent Behavior

• Measured: cold CLI startup, tool execution, AI resolution time
• All single-attempt — zero iterations needed
• Arcana: file detection on disk (background process)
• Hermes: read_file → write_file → terminal → verify
• OpenCode: glob → read → write → bash test

5. Framework

• All times measured via date +%s%3N at shell level
• Workspaces: isolated directories per agent
• Models: deepseek-v4-flash (OpenCode, Arcana), deepseek-v4-pro (Hermes)

6. Human Review

• All implementations independently verified
• Code approaches documented and compared
• SVG report with charts + table + methodology

Results Summary

← Back to benchmark