Voice agents are distributed systems with microphones

A voice-agent demo can sound convincing while doing almost nothing. I built the integration work around the less glamorous questions: what got changed, who authorized it, what happened after a timeout, and what evidence is left when the next engineer has to investigate.

I care about the part after the nice conversation. What did the system actually change? Was the caller allowed to change it? What happens if the booking service commits the update but drops the response? How does anyone find out which step failed without listening to an entire call recording?

Those are backend questions with a microphone attached.

I built five connected voice-agent engineering projects around those boundaries: appointment workflow orchestration, adversarial evaluation, operations and latency analysis, event-driven CRM handling, and native iOS/Android integration. The backend implementation includes working HTTP contracts and regression tests, not just architecture diagrams. Speech transport, model-provider configuration, customer infrastructure and model artifacts stay separate from this public write-up.

I separated the system into four responsibilities:

1. Media: PSTN, SIP or real-time app audio, speech recognition, turn detection, interruptions, synthesized output and session state.
2. Reasoning: dialogue context, policy-aware intent interpretation and a structured proposal to call a tool. This output is probabilistic and untrusted.
3. Execution: schema validation, authenticated identity, tenant-scoped access checks, business rules, transaction or durable job handling, and reconciliation.
4. Operations: call outcomes, redacted traces, latency budgets, evaluation suites, deployment versions, alerting and escalation.

One provider might bundle the first two. Some have webhook and tool features for the third. None of that removes the need for a backend you control. I wouldn't give an agent direct database credentials just because it speaks politely and has a convincing voice.

Customer / mobile client
    |
    v
Voice transport / STT / dialogue engine
    |
    v
Typed tool request (untrusted proposal)
    |
    v
First-party tool gateway ── authentication + tenant policy
    |                             |
    v                             └── reject / escalate
Business API + idempotency claim
    |
    v
Durable outcome + event delivery
    |
    ├── CRM / case system
    └── redacted metrics + evaluation

In the workflow project, the conversational side produces a request, not permission. A tool contract might look like:

{
  "tool": "reschedule_appointment",
  "booking_id": "apt-101",
  "new_slot": "2026-11-18T18:30:00Z",
  "operation_id": "reschedule-01"
}

The backend derives the principal from a verified session; it does not believe a caller-supplied tenant_id or is_verified field. It checks ownership, validates the destination slot, and commits at most one logical mutation for the operation key. Higher-risk changes need a separate step-up action, not a second LLM-generated sentence saying verification succeeded.

The reference service deliberately includes a commit succeeded, response was lost scenario. The first request reports a simulated timeout after applying the write. A retry with the same operation ID returns the stored result instead of changing the booking again. That is a much better outcome than accidentally booking somebody for Tuesday, Wednesday and the end of civilization.

The local implementation demonstrates operation-ID deduplication and fingerprint checks. For enterprise persistence, the transaction and deduplication contract also needs defined retention, crash recovery, concurrent-writer guarantees and operator reconciliation. An in-memory Map is a useful executable test fixture, not a substitute for durable storage.

The evaluation project records expectations for callers who skip verification, switch intent mid-flow, attempt cross-account access, inject policy overrides or ask for a human. I also include malicious text returned by a tool: external data doesn't become a new system instruction because it arrived with an HTTP 200.

I separate three different questions: whether the model understood the caller's intent, whether it selected an appropriate tool action, and whether the backend accepted only authorized side effects. Blurring those together creates impressive dashboards and unpredictable incidents.

A deterministic test harness is a good first gate. It is not a measured model accuracy score. Real quality evaluation also needs representative audio, accents, noise, turn-taking, long pauses, interruptions, model/version comparisons, human review and outcome labels. A test should identify the reason for failure, not merely count how often the model sounds confident.

I built the operations implementation around bounded retries, downstream failures, a simple circuit-breaker rule and per-operation latency samples. A call has many critical paths: recognition latency, model decisions, tool fan-out, synthesis, network jitter and interruption handling. Optimizing average response latency alone is a good way to miss the callers who had a miserable experience.

I keep P50/P95 for turns and tool calls separate from end-to-end task completion, handoff rate, per-call cost and the distribution of failure reasons. Be careful with retries: a retry can recover a dependency problem or double the time until someone hears a response. A slow booking API does not improve because the agent says "one moment" more cheerfully.

The published architecture explains which signals to capture and how to keep customer identifiers out of public telemetry. The local 80-operation dataset is synthetic; it is not carrier latency data or real voice-provider billing.

The CRM integration treats delivery as at least once, rather than relying on magical exactly-once promises. I validate signed event envelopes, timestamps, tenant scope, idempotency keys and request fingerprints. If the downstream system is unavailable, the event needs a recoverable state, not a silent 200 OK and crossed fingers.

I implemented HMAC verification, replay-window validation, conflict detection and an outage/reconciliation path in the local service. The production integration contract covers the provider-specific signature format, persistent inbox and outbox tables, uniqueness constraints, dead-letter handling, per-tenant scopes, retention controls, and retry scheduling. The interesting failure is often the process crashing between a remote side effect and recording success.

For an iOS or Android companion, I keep callback consent and the user experience on-device. Session authentication, case ownership, abuse limits and voice-provider credentials belong behind a first-party API. The device receives a queued request ID; it does not get to declare that a person has been called before the provider says so.

On-device speech, Core ML or Android ML inference can reduce latency and network dependence for carefully selected tasks. It also changes the performance envelope: model size, battery, thermal throttling, hardware support, fallback, model versioning and privacy decisions all matter. Performance claims need actual device and model benchmark results, not assumptions about the Neural Engine or an Android accelerator. I implemented the native API clients and pluggable inference adapters; model weights and private performance datasets are not bundled with this public case study.

A voice agent that can't solve a request still needs to finish the interaction responsibly. That means preserving a short, redacted summary for a human, owning retries and acknowledgements, and giving support staff enough context without dumping secrets or personal histories into every tool call.

The five projects are my way of connecting conversation design to the stuff engineers have to maintain after launch. A nice voice matters. So do boring things like tenant isolation, replay protection, trace IDs, failure budgets, audit records, rollback paths and documentation. The boring parts tend to become interesting at approximately 2:17 AM.

Related work: Voice agent engineering collection · Applied AI and ML · Engineering work · Security analysis.