Streaming Speech Inference & Endpointing

What I implemented

• Implemented a 20-ms streaming VAD state machine with configurable energy threshold, speech confirmation and silence hangover.
• Built deterministic packet arrival jitter/loss schedules, simple telephony-style band filtering, and audio frame processing.
• Measured per-frame events, algorithmic hangover and local compute real-time factor, separating them from actual network and model latency.

Data and execution path

1. 01

   8-kHz mono PCM frame ingestion

2. 02

   20-ms framing and signal-energy calculation

3. 03

   Multi-frame speech confirmation

4. 04

   Speaking/idle state transition with hangover

5. 05

   Simulated packet loss, jitter and channel filtering

6. 06

   Endpoint event times and locally measured compute cost

Experiment and results

On the included 1.25-second synthetic sample, the detector processed 63 20-ms frames and emitted a speech-start and speech-end event. The configured 8-frame hangover adds 160 ms of algorithmic trailing-silence delay. These are local signal-path observations, not end-to-end live-call latency or streaming-ASR results.

These results are from the specified local dataset or offline rendering pipeline. I keep them separate from real speech-corpus and live-call metrics because the populations and failure modes differ.

System-level constraints

• A live media session introduces microphone/PSTN capture, RTP jitter buffering, sample-rate negotiation and bounded work queues beyond this signal model.
• Turn-level latency combines ASR partial stability, endpoint delay, first audio, barge-in and tail response time; local VAD compute time is only one component.
• Quantized inference and batching behave differently across CPU/GPU/ANE, thermal states, memory limits and frame deadlines.
• Backpressure, cancellation, timeouts and fallback are system reliability properties; they cannot be inferred from model accuracy.