A voice agent has more moving parts than a chat box: audio transport, speech recognition, model turns, knowledge retrieval, playback, interruptions, call limits, and sometimes recordings. The important design decision is where business control lives.
It should live on the server.
Keep business access out of the browser
In SyntaxLab's calling demo, the browser opens a relay endpoint rather than receiving an upstream provider key. The relay establishes the provider connection server-side, performs knowledge-base tool calls there, and enforces session limits independently of what the browser does.
That gives the application a clear place to enforce identity, access scope, rate limits, quotas, and logging. It also means the same knowledge search can serve text chat and voice, avoiding two inconsistent answers to the same customer question.
Treat a tool result as data, not permission
When the voice agent needs a fact, it calls a narrowly named search_knowledge_base tool. The server searches only that caller's permitted material and returns passages. It does not grant a general connection to the database.
The application remains responsible for validation and authorisation on every tool call. Function calling is intended for exactly this division of responsibility: a model asks for a tool, and application code executes and returns the result. OpenAI function-calling guide
Put guardrails in the session layer
The demo signs short-lived relay tickets, prevents simultaneous calls for the same visitor, begins charging time only after the upstream agent connects, and stops the call when its allowance is exhausted. These are operational controls, not prompt suggestions.
For a production deployment, add business-specific escalation rules, consent and recording notices where needed, retention policies, monitoring, and staff handoff paths. Voice agents should always have a safe way to say: “I can't complete that here; let me connect you with the team.”
Project evidence
This draft is based on the implemented calling-demo relay in syntaxlab-backend/src/agents/demo-agent/relay.ts, session/quota code in calls.ts, and the shared retrieval implementation in src/kb.ts. The repository supports short-lived demo data and recordings when configured; it does not prove call quality, conversion, or a production contact-centre deployment.