Evidence & Trust

← Back to benchmark

How to verify this benchmark is honest

1. Reproducible

Run bash reproduce.sh — same test file, same prompts, same verification. Anyone with the three CLIs installed can reproduce identical results.

2. Immutable Test File

SHA256: d697a6006ed360477c4b96d2d9eaf85880187e168029466c29b56ecb7ba56b5f

The test file was created BEFORE any agent ran. Hash proves no post-hoc modification.

3. Raw Implementations (SHA256)

Hermes: 21c699db843084d7bda37935a702be5c3d202a1b45a32dde45a4d436faf6cd9d

OpenCode: 3f25d9d8d80cb676a5be1eac44048306261b0fe7faf450e8a44586974a3d8272

Arcana: 7b969a14e43c2cdc7a7beef2a570e431f71e3fff50c998c8e4488ac860e5c0fd

4. Independent Verification

cd hermes && node semaphore.test.js  # → 10/10
cd opencode && node semaphore.test.js # → 10/10
cd arcana && node semaphore.test.js   # → 10/10

All pass independently. Not dependent on the agent that produced them.

5. Timing Methodology

6. What We Did NOT Do

Anti-PatternStatus
Cherry-pick runs❌ All were single-attempt
Modify test file after seeing results❌ SHA256 locked beforehand
Run agent with knowledge of solution❌ Clean subprocess each time
Use different test suites per agent❌ Identical semaphore.test.js
Hide failed attempts❌ There were none
Benchmark ourselves❌ Hermes ran as subprocess

7. Agent Independence

← Back to benchmark