Voice is where a personal system feels either natural or intrusive. JARVIS’s voice loop is designed to be local-first, to hear the owner without hearing itself, to stay silent when unsure whether it was addressed, and to stop the moment the owner speaks over it. It is also designed to be honest about which voice is speaking.
The pipeline
The loop runs in this order:
- Capture. Microphone audio (WASAPI) goes into a 30-second pre-roll ring buffer, so the first syllable survives while the system decides whether speech started.
- Echo cancellation and self-speech gate. An adaptive echo canceller removes JARVIS’s own playback, so JARVIS never hears itself as the owner.
- Speech segmentation. Silero voice-activity detection does this. A simpler energy detector exists as a fallback, but it must be chosen deliberately. The code’s own words are “Refusing to fall back to EnergyVAD silently.”
- Wake word. An openWakeWord gate is available, but it is off by default, so the session is open-mic.
- Transcription. faster-whisper large-v3 runs locally on the GPU. Cloud transcription is used only when explicitly requested.
- End of turn. A semantic model decides whether the owner has finished or is pausing mid-thought.
- Speaker verification. Input is classified OWNER, NOT_OWNER, UNCERTAIN or UNAVAILABLE, and only OWNER is admitted. The owner’s voice embedding is kept, and the audio samples are dropped.
- Addressee detection. “Was that said to me?” Uncertainty resolves toward silence.
- Cognition. The request is handled as on the how it works page.
- Clause chunking. The reply is split into clauses, with a pronunciation dictionary and prosody intent.
- Speech and playback. Playback feeds the echo canceller’s reference signal.
A runtime map generated from the source code traces this path across 14 stages and 24 code anchors, with every anchor resolving to real code.
Why local recognition
A measurement recorded in the speech module’s own documentation (July 2026) gives 165 ms for local transcription against 2,479 ms for the hosted transcription API. The local latency stayed flat as utterances got longer. This is a code-docstring measurement from one point in time, not a current benchmark. Local recognition also keeps the owner’s raw speech on the machine.
GPU memory is the constraint. A screen on 21 September 2026 found that running the recogniser at int8_float16 precision saves about 1.9 GiB of VRAM, with no accuracy regression on a 14-case public-synthetic test set. That room matters on a 12 GB card shared with the voice, the HUD and language-model work.
End-of-turn detection has its own recorded figure: the text model reaches 94.6% balanced accuracy on the owner’s own end-of-utterance set. In that set, its remaining errors were all cases of waiting too long. None cut the owner off mid-thought. That residual is why an MIT-licensed voice-activity-projection model was approved. The licence register records that weights from a different turn-taking repository were refused: the repository ships an MIT licence file, but its README restricts the trained weights to academic use. “A LICENSE file is not the licence.”
Interruption
The turn manager handles barge-in against a 120 ms budget. In tests it measured 14.8 to 27.6 ms. When the owner interrupts, unheard clauses are cancelled and only what the owner actually heard stays in context. After a correction, the conversation continues from the correction rather than from the abandoned answer. A plan that was mid-flight is abandoned at the next step boundary, so no effect is left half-applied.
The speaking voice: C013
C013 is the identifier of the owner-selected speaking voice. JARVIS speaks through a locally run, open-source text-to-speech model, with no fine-tuning, conditioned on a short reference clip. The owner chose the voice through scored and listened comparisons. The project’s rule is that owner preference alone never grants qualification, and its technical qualification is still pending. Two technical checks stand out: a long-form non-regression check failed, and time to first audio has not been measured.
The voice runs on the owner’s GPU from an immutable, hash-checked package. At load, the package is checked against its manifest digest and fails closed on any mismatch. It “never falls back to a different checkpoint”. No pitch, formant, rate, equaliser or effect processing is applied. Nothing in the loop writes audio to disk.
If C013 cannot speak, a local Piper voice takes over, and the switch is announced. The project describes this as being honest about who spoke. An earlier hosted voice remains as a rollback path. That hosted voice’s streaming mode failed its latency budget in August (p95 367.5 ms against a 220 ms target).
We don’t publish voice samples.
Latency, honestly
The project sets first-audio budgets per lane: reflex p95 ≤ 300 ms, and cognition p50 ≤ 600 ms and p95 ≤ 900 ms. One recorded reflex first-audio reading is 398.5 ms, over its budget, and a fresh owner-present reading is owed. The only recorded audible first-audio reading, 403.5 ms p50 in a 17 September handoff, used the older Piper voice, not C013.
A probe now exists to measure C013 specifically. It rejects any fallback voice and binds to the exact package. The project labels it an “IMPLEMENTED and TESTED instrument, NOT audible qualification”. The audible measurement needs the owner present.
On 22 September 2026 the voice worker was loaded alongside a local language-model recipe, speech recognition, the HUD and body services in one clean run. The whole-GPU peak was 11.03 GB. That shows the stack fits in memory. It says nothing about latency or audio quality.
What is not proven yet
- C013 is not technically qualified. Long-form non-regression failed, and first audio, streaming, acronym pronunciation, audible cancellation, restart recovery and fallback have not been measured for it.
- Full-duplex voice has not been proven live. It needs an owner-present sitting, and the interrupted-speech journey is fixture-tested only.
- Reflex first audio is over budget on its recorded reading, and no C013 audible reading exists.
- Recorded figures are dated and limited. The ASR comparison is a July 2026 code-docstring measurement. The barge-in times come from tests. The coexistence run is a single observation.
- No voice samples are published.