Audio ML engineering: what the numbers actually mean

I built the audio ML experiments around results that can be reproduced and challenged: ASR alignment counts, codec reconstruction error, streaming endpoint delays, offline TTS output and classifier ablations. A benchmark number without its dataset, denominator and runtime conditions is decoration.

Is that word error rate measured on actual recognizer output? Is the compression ratio including model weights? Does the TTS model really control pitch, or is there just an attractive curve next to the audio? And when someone says "real-time," do they mean 20-millisecond frames or a real caller waiting for a response?

I built five audio engineering projects around those questions. They cover ASR transcript evaluation, a small trained neural vector-quantized codec, a prosody-and-synthesis pipeline, streaming speech endpointing, and a controlled audio-classification ablation. Each has an executable offline Python implementation and tests. The point is to connect modeling choices with the constraints they'd face in a voice system—not to pretend a MacBook experiment is a distributed research cluster.

In the ASR evaluation project, I implemented word error rate (WER) and character error rate (CER) from sequence alignment, including substitutions, deletions and insertions. The reference length is the denominator; for a corpus, I sum errors and reference words before dividing. Averaging individual utterance WERs gives a different, often misleading result.

I also split results by condition: clean audio, noise, telephony-style degradation and code switching. A recognizer can look good overall while consistently dropping the one field the downstream workflow needs—an account number, a confirmation phrase or a date. In an appointment workflow, intent accuracy and transcription accuracy are related but not identical questions.

The first local corpus uses four hand-authored reference/hypothesis pairs, producing four word errors across 27 reference words (WER 14.8%). That verifies the evaluation code; it is not a measured ASR model result. Feeding it real model hypotheses from licensed, speaker-separated recordings is the next boundary. Accent coverage and multilingual quality are properties of those real datasets and models, not of the evaluator itself.

I implemented a small neural audio codec experiment in NumPy. The encoder and decoder have learned nonlinear weights. Each 64-sample frame maps to a continuous embedding, then to the closest entry in a 32-vector codebook. The decoder reconstructs the frame from that discrete code. I train the weights with a straight-through gradient approximation, and the codebook moves toward assigned encoder embeddings.

PCM frame (64 samples)
    -> nonlinear encoder
    -> 12-dimensional latent vector
    -> nearest of 32 trained codewords
    -> 5-bit code index
    -> nonlinear decoder
    -> reconstructed PCM frame

I measured reconstruction MSE, signal-to-noise ratio and code utilization. On my deterministic synthetic voiced waveform sample, 280 optimization steps reduced MSE from 0.04164 to 0.01217. That produced 5.24 dB reconstruction SNR. The index stream alone amounts to 625 bits/second at 8 kHz.

That bitrate sounds astonishing until you listen to the quality. The comparison 8-bit μ-law sample compander has 37.89 dB SNR at 64 kbit/s on the same signal. It uses much more bandwidth and reconstructs far better. The VQ experiment's index bitrate also excludes headers, model parameters, packetization and real transport. There is no magic codec result hiding in the ratio.

The codec comparison exposes the next engineering constraint: representative held-out speech, listening tests, packet-loss behavior, latency and actual inference memory determine whether a compressed representation is useful in production. Residual vector quantization, temporal modeling and discrete or continuous latents become interesting after the baseline is measured.

In my TTS and prosody pipeline, I separate sentence segmentation and acoustic intent from the renderer. A statement, a question and an emphatic reply receive different rate decisions and target pitch contours.

The offline implementation invokes macOS say and writes actual AIFF speech. I can set rate per chunk. But that engine does not accept the detailed F0 contour I represented as control metadata. I'm deliberately keeping the requested prosody metadata separate rather than implying the output voice follows it.

A fully controllable neural TTS adapter would need an explicit conditioning interface, model-specific acoustic targets and perceptual tests. I'd measure pronunciation, phrasing, question inflection, speaker consistency, time-to-first-audio, chunk boundaries and interruption handling independently. Otherwise, "expressive" is just something a demo voice says about itself.

My streaming audio project processes 20-ms frames, uses multiple frames to confirm speech and waits through a configurable hangover before declaring the turn finished. I added deterministic jitter, packet loss, noise and a simple telephony-style channel filter to make the control path observable.

The local 1.25-second synthetic sample processes as 63 frames and emits both speech start and speech end. Eight silent hangover frames introduce 160 ms of algorithmic trailing delay. That's before a model thinks, a tool responds or the TTS stream begins.

Processing real-time factor is one number. End-to-end perceived latency is another. At system scale I'd want time-to-first-audio, ASR partial stability, turn-end delay, interruption response, P50/P95 turn latency, CPU/GPU scheduling and queue backpressure in separate traces. A fast model behind a slow queue isn't fast to the person on the phone.

For audio ML ablations, I trained a small logistic activity classifier using four acoustic features: RMS, zero-crossing rate, spectral centroid and spectral flatness. I kept fixed, generated training and held-out samples, with the noisy split evaluated separately.

Three variants isolate what matters: an RMS-only model, a multi-feature model trained on clean waveforms, and the same feature set trained with synthetic noise augmentation.

On the seeded synthetic noisy holdout, F1 measured 0.689 for RMS-only, 0.361 for clean-trained multi-feature, and 0.867 for the noise-augmented model. Clean-trained multi-feature performance collapsed under the changed noise distribution; augmentation helped on this specific synthetic distribution. It would be irresponsible to turn that into a claim about real-world ASR accuracy, accents or call-center performance.

That's the value of an ablation: it separates the thing you changed from the story you'd like to tell about it. There's a reason I store seeds, feature definitions, dataset conditions and confusion counts, not just a line chart that goes up and to the right.

The local suite uses Python, NumPy and the system speech engine. It doesn't include self-supervised foundation-model pretraining, large multilingual corpora, accelerator kernels, distributed GPU training, production RTP or customer traffic.

Those are very different engineering problems: data licensing and consent, speaker-separated validation, weak-label filtering, checkpoint recovery, distributed optimizer state, mixed precision, kernel profiling, model serving, streaming deadlines and privacy-aware observability. The baseline implementation gives me contracts and experiments to extend, but each of those claims needs its own implementation and evidence.

I enjoy that boundary between models and systems. A speech model can be mathematically interesting and operationally terrible. The fun part is finding out why before a customer does.

Related projects: Speech & Audio ML · Voice Agent Engineering · AI & ML · Mobile voice-agent integration.