Voice evaluation experiments

These helpers evaluate candidates without changing live recognition or the selected synthesis engine. They do not open a microphone or upload audio. The wake helper calls acc stt wake, which can download missing local models through its ordinary first-run setup; prepare assets before offline evaluation. Turn and Piper helpers require separately supplied models/environments. Use recordings you deliberately collected; keep private audio outside Git.

Wake detection

acc config set wake-attention true requires an attention word before the numeric code, including during playback wake probes. Try it before replacing the local Canary detector. It reduces opportunities for an ordinary spoken number to wake Accessor; it is not authentication or a measured false-wake improvement.

Create non-overlapping 16 kHz mono PCM16 windows, at most eight seconds each:

[
  {"file":"quiet-wake.wav","wake":true,"condition":"quiet-near"},
  {"file":"music-wake.wav","wake":true,"condition":"music","playback":true},
  {"file":"television-negative.wav","wake":false,"condition":"television"}
]
python3 scripts/evaluate_wake.py corpus/manifest.json --acc target/release/acc \
  --output /tmp/wake-report.json

The report includes wake recall, false detections per negative corpus hour, condition errors, inference median/P95, and audio hashes. Duplicate files are rejected to avoid inflating the exposure. --candidate candidate.json compares previously collected candidate results on the same files/order using the acc stt wake --json results shape. This is offline comparison, not a live shadow detector. The helper reports inference timing, not end-to-end or acoustic latency.

Collect separate conditions for fan/TV/music, room distance, the user’s accent, near-miss numbers, assistant speech, and speech while assistant/music output is playing. Short negative recordings cannot establish a room false-accept rate. Measure idle CPU, power and long negative exposure separately on the target machine. Keep the existing scripts/test_room_wake.py procedure for deliberate room recordings and do not infer real-room accuracy from synthetic clean clips.

For candidate research, sherpa-onnx KWS supports configurable keywords. openWakeWord is another candidate; its code and pretrained-model licences differ. Neither is enabled by this change.

Semantic turn endpoints

Draft mode (“let me explain … send that”) is the implemented way to prevent hesitations and corrections from submitting a partial request. The ordinary silence cutoff remains configurable with stt.endpoint-ms.

scripts/evaluate_turns.py evaluates a separately provided Smart Turn v3 ONNX model on labelled utterances. Its preprocessing follows the upstream eight-second audio tail, left padding and normalized Whisper features. Use a separate Python environment with numpy, onnxruntime, and transformers; provide the model explicitly.

[
  {"file":"hesitation.wav","complete":false},
  {"file":"finished-request.wav","complete":true}
]
python3 scripts/evaluate_turns.py corpus/turns.json --model /path/to/model.onnx \
  --threads 1 --output /tmp/turn-report.json

The report includes model hash, load time, feature-plus-inference time and counts of premature/delayed endpoints. Include silence, background noise, corrections, backchannels, accents and deliberately unfinished phrases. Verify preprocessing against the exact model revision and benchmark on the old/slowest supported CPU before integrating it into the capture loop. Upstream speed claims are not Accessor measurements. No semantic endpoint model is loaded during normal use.

Piper migration

The pinned legacy binary remains the default. The OHF-Voice successor provides a chunked Python API and libpiper C API. Its engine is GPL-3.0-or-later; voice licences need separate review. The legacy persistent-WAV protocol cannot simply be substituted with the successor’s CLI, which reports generated paths differently.

If you have installed the successor in a separate environment, measure it:

python3 scripts/evaluate_piper.py --python /path/to/piper-env/bin/python \
  --voice /path/to/voice.onnx --output /tmp/piper-successor.json
acc tts benchmark --provider piper --runs 5 "The task is running."

Use the same text and voice for comparison. The successor helper reports engine version, voice hash, load time, first synthesized chunk, total synthesis time and audio length while keeping the model warm between runs. It does not play audio. First chunk availability is not audible latency. Cancellation, buffering, resampling, AEC references and libpiper ABI compatibility still need validation before a live-engine migration.