I built a Voice AI agent that handles most customer calls on its own, including answering routine questions, logging requests, and scheduling visits. But occasionally, the agent reaches a question that requires a human specialist. The obvious approach is to transfer the caller to that specialist. But, I wanted to avoid that entirely.
Instead, the AI agent keeps ownership of the conversation. It places the caller on hold, reaches out to a specialist in the background, gets the answer, and then returns to the caller with that answer. The customer never leaves the original call.
That changes the role of the human specialist. In this flow, the AI agent treats human specialists as a callable tool, removing the need to hand off the call to a specialist. They are simply helping the AI answer a specific question.
Here is the high-level architecture of the workflow:
Consulting the Specialist by Phone
My first version used a separate phone call.
When the AI needed help:
- The caller stayed on the original call and was placed on hold.
- The system started a separate call with the specialist.
- The AI asked the specialist the customer’s question.
- The specialist provided the answer.
- The answer was sent back to the original AI conversation.
- The AI resumed the call with the customer.
I built this using a Squad of multiple assistants, with each assistant handling a different part of the workflow.
The customer never had to speak with the specialist, repeat the problem, or leave the AI conversation.
For a small number of escalations, this worked well.
The Scaling Problem
The problem appeared when several customers needed specialist help at the same time.
A phone consultation still requires one specialist to stay on one call until that consultation is complete.
If three callers need help but only one specialist is available, the consultations become a queue.
Adding more specialists increases capacity, but it does not remove the underlying limitation: voice is a one-conversation-at-a-time communication channel.
The AI workflow could scale, but the human consultation channel could not.
Consulting the Specialist by Chat
I kept the same core architecture and changed only how the AI reached the specialist.
Instead of starting a phone call, each escalation opens a separate chat conversation. The consultation is assigned to an available specialist, and a chat window appears on their screen where they can respond.
The customer experience stays the same.
The caller remains on the original Voice AI call. The AI consults the specialist in the background, receives the answer, and then continues the conversation with the caller.
The only difference is the communication channel between the AI and the specialist: chat instead of voice.
One Pattern, Two Consultation Modes
Both approaches follow the same model:
Customer ↔ Voice AI ↔ Human Specialist
The specialist supports the AI rather than replacing it.
A phone consultation works well when escalations are infrequent or require a more detailed conversation. Chat works better when specialists need to handle several consultations at the same time.
The important architectural decision is not whether the specialist is contacted by phone or chat.
It is that the customer call is never transferred.
The AI remains the primary interface, while human expertise is brought into the conversation only when needed.