Voice Input in Arcana: Fully Local Speech-to-Text

Arcana v0.4.0 introduced voice input via Ctrl+X V. The goal was simple: talk to your agent like you talk to a colleague, without shipping your voice to someone else's server.

Voice input is one of those features that sounds trivial until you think about the implications. Every cloud-based speech-to-text service requires your audio to leave your machine. That audio might contain proprietary code being dictated, sensitive data being discussed, or internal project names that should not exist on someone else's servers. Arcana takes a different approach: the entire pipeline runs locally, on your hardware, with zero network dependencies.

This is not a compromise for the sake of ideology. Local speech-to-text has reached a point where it is genuinely useful for developer workflows. The accuracy is good enough for most dictation, the latency is acceptable on modern hardware, and the privacy guarantees are absolute. Your voice data never touches a network socket.

The Pipeline

Voice input in Arcana is a two-stage local pipeline:

Capture and transcribe. Audio goes to whisper.cpp, a C/C++ implementation of OpenAI's Whisper model. It runs entirely on your CPU. No GPU required, no API calls, no network traffic. The model loads once and stays resident for the session.
Clean up with Ollama. Raw transcripts from Whisper are often rough: filler words, false starts, run-on sentences. Arcana passes the raw text to a local Ollama model (you pick which one) that normalizes it into a clean, actionable prompt. The cleanup step is optional but recommended.

Model Sizes and Tradeoffs

Whisper ships in several model sizes, each with a different accuracy/speed profile:

tiny (~39 MB). Fastest. Accuracy is rough but usable for short, clear utterances. Good for quick commands.
base (~74 MB). A reasonable default. Handles most English speech well with minimal latency.
small (~244 MB). Noticeably more accurate, especially with accents and background noise. The sweet spot for most users.
medium (~769 MB). High accuracy but slower. Worth it if you dictate long passages or work in noisy environments.
large (~1.5 GB). The most accurate but slowest. Only recommended if you have substantial RAM and patience.

Arcana defaults to the small model on first setup. This gives a good balance of accuracy and speed for most developer workflows. You can change this at any time with the voice.model config key.

Why Local?

Most voice-to-text in developer tools routes through cloud APIs: Google STT, Azure, Deepgram. That works until you're working on proprietary code, regulated data, or anything you'd rather not have transcribed on someone else's infrastructure.

Consider the compliance implications. If you are working on HIPAA-regulated healthcare software, your dictated notes about patient data cannot leave your machine. If you are working on defense contracts, your voice recordings about classified specifications cannot touch external servers. Even if you are working on a startup's unreleased product, your voice discussing feature plans before launch is a leak vector.

Arcana's approach trades a small amount of transcription quality (Whisper is slightly less accurate than cloud APIs on noisy audio) for complete data sovereignty. Your voice never leaves the machine. Your prompts never touch a third-party server.

The quality gap is narrowing with each Whisper release. The large-v3 model approaches cloud API accuracy on clean audio, and the Ollama cleanup step compensates for many of the remaining weaknesses by normalizing punctuation, capitalization, and sentence structure.

Setup

Voice input requires two things: a Whisper model file and a running Ollama instance.

# Start Ollama (if not already running)
ollama serve

# Use voice input in the TUI
# Press Ctrl+X, then V

Voice input requires a running Ollama instance with a Whisper model. The model is downloaded automatically on first use.

Configuration

Voice behavior is controlled through a few config keys in your config.json:

{
  "voice": {
    "model": "small",
    "cleanup": true,
    "cleanupModel": "llama3.2",
    "language": "en"
  }
}

Each key controls a specific part of the pipeline:

voice.model which Whisper model variant to use (tiny, base, small, medium, large). The model is downloaded automatically on first use and cached locally.
voice.cleanup whether to run the Ollama cleanup step (default: true). Disabling this gives you raw Whisper output without normalization, which can be useful if you prefer to edit the transcript yourself.
voice.cleanupModel which Ollama model to use for normalization. Smaller models like llama3.2 or qwen2.5 work well for this task because the cleanup is a simple formatting job, not a complex reasoning task.
voice.language hint for Whisper about the input language. This improves accuracy when you know the language in advance. If unset, Whisper auto-detects the language, which adds a small amount of latency.

You can also set these via environment variables if you prefer not to edit the config file directly.

Linux / macOS:

export ARCANA_VOICE_MODEL=small
export ARCANA_VOICE_CLEANUP=true
export ARCANA_VOICE_LANGUAGE=en

Windows (cmd):

set ARCANA_VOICE_MODEL=small
set ARCANA_VOICE_CLEANUP=true
set ARCANA_VOICE_LANGUAGE=en

Windows (PowerShell):

$env:ARCANA_VOICE_MODEL="small"
$env:ARCANA_VOICE_CLEANUP="true"
$env:ARCANA_VOICE_LANGUAGE="en"

What It Feels Like

Press Ctrl+X V. A small indicator appears in the TUI. Speak. Press Ctrl+X V again to stop. The transcribed and cleaned text appears in the input field, ready to send. You can edit it before sending, or press Enter to fire immediately.

The latency depends on your hardware. On a MacBook Pro M2, transcription takes 1-3 seconds for a 10-second utterance. Cleanup adds another 1-2 seconds. Total round-trip from speech to prompt is typically under 5 seconds.

Practical Tips

After using voice input extensively, here are patterns that improve the experience:

Speak in complete sentences. Whisper works best with natural, flowing speech. Single words or fragmented phrases are more likely to be misheard than full sentences with context.
Pause briefly before technical terms. A short pause before a function name or variable gives Whisper a better chance of capturing it correctly. "Call the function pause process data" works better than "callthefunctionprocessdata."
Use the cleanup step. The Ollama normalization is remarkably good at fixing Whisper's punctuation and capitalization quirks. Raw Whisper output often has no periods or commas; the cleanup step adds them based on natural sentence boundaries.
Edit before sending. The transcript appears in the input field before you send it. Take a moment to scan for obvious errors, especially around technical terms and proper nouns.
Start with the small model. If you are unsure which model to use, start with small. It runs fast on most hardware and handles the vast majority of developer dictation well.

Limitations

Local speech-to-text is not perfect. Whisper struggles with heavy accents, technical jargon, and very fast speech. The Ollama cleanup step helps with punctuation and formatting but cannot fix fundamentally wrong transcriptions.

The latency is also a factor. Cloud APIs typically return results in under a second. Local Whisper, especially on CPU-only hardware, can take several seconds for longer utterances. This is fine for conversational dictation but can feel sluggish if you are used to instant cloud feedback.

Memory usage is another consideration. The large model requires roughly 1.5 GB of RAM while loaded. On machines with limited memory, this can pressure other applications. The small model uses approximately 500 MB, which is a more comfortable fit for most development environments.

For most use cases, the accuracy is more than sufficient. For edge cases, the transcript is always editable before sending. The combination of local transcription, local cleanup, and manual review gives you a reliable voice input system that never compromises on privacy.