What building AI voice agents on real Indian phone lines taught me

There's a category difference between a voice AI demo in a browser and an agent on an actual phone number. I crossed it building a calling platform for a real-estate business: bulk outbound campaigns and 24/7 inbound support, one agent, real telephony. Most of what I learned had nothing to do with the LLM.

Telephony is the hard dependency

In India, Twilio isn't the default answer; regulations and cost push you toward Airtel IQ or Exotel for local numbers. The practical consequence: your SIP layer must be provider-abstracted from day one, because you will switch providers, and mid-project. I kept a comparison doc (per-minute cost, trunk setup, DTMF quirks) that ended up being one of the most-read files in the repo.

One agent, two jobs

Outbound sales and inbound support want opposite personalities. Outbound is proactive, script-driven, allowed to steer; inbound must listen first, answer precisely, and never sell to someone calling about a problem.

Running two deployments doubles your infra and splits your call logs. Instead, the agent reads call direction from LiveKit room metadata and selects its persona and prompt accordingly. One codebase, one deployment, two behaviors. And the direction check is one branch, not an architecture.

The pipeline is a latency budget

A voice turn is STT → LLM → TTS, with voice-activity detection (Silero VAD) deciding when the user has finished speaking. Each stage gets measured separately, because "the agent feels slow" has four different fixes depending on which stage is slow:

• VAD too patient → agent feels deaf; too eager → it interrupts
• STT slow → everything downstream starts late
• LLM slow → cap output length, stream, pick a faster model for telephony
• TTS slow → first-sentence latency matters more than total

Per-stage metrics turned provider selection from vibes into data.

Calls are data, not just conversations

Every call gets transcribed into Firebase, then post-processed: sentiment analysis and a Hot/Warm/Cold lead score that lands in a dashboard the sales team actually opens. This is the part that made the system valuable: the calls were always going to happen; the structured record of every call is what a human team can't produce at 2am.

If you're starting a voice-agent project, budget your effort roughly like this: a third on telephony plumbing, a third on turn-taking latency, a sixth on post-call data, and whatever's left on prompts. It's the inverse of how every tutorial allocates it, and the inverse is correct.