Building a Customer Support AI Agent: Sources, Actions, and Human Handoff
A customer support AI agent has three different jobs: explain what a business knows, help a customer take an allowed next step, and recognize when a person needs to take over. A fluent answer is only one part of that workflow.
Make Agent Fast is the product I built around this problem. Its setup starts with company material, a reviewable agent draft, and approved knowledge. The same agent can then serve a website widget and connected messaging channels. This article explains the design choices behind that workflow and a practical way to evaluate an agent before publishing it.
Start with a narrow job
“Answer everything about our company” is difficult to test. A clearer first job is “explain our services and collect enquiries that need a reply.” The business owner can then decide which information is public, which questions require a person, and what the agent may collect.
For a booking assistant, distinguish a request from a confirmed appointment. If the system cannot check availability and complete the reservation, its reply should say that it has collected a request. The wording must match the action that actually happened.
A useful configuration records four things:
- The customer's goal, such as finding a service or asking for a quote.
- The sources the agent may use to answer.
- The actions it may take, including the fields each action needs.
- The conditions that send the conversation to a person.
Review knowledge before publishing
Crawling a website is a convenient way to collect material, but a crawled page is not automatically a current business policy. Old pricing, duplicate pages, draft content, and missing exceptions can all enter the knowledge base.
In Make Agent Fast, crawling and source ingestion lead into a review-before-publish workflow. Retrieval uses PostgreSQL and pgvector to find relevant material for the runtime. Retrieval gives the model useful context; it does not make an outdated source correct.
For each important source, check its owner, intended audience, and current accuracy. Keep a small set of approved answers for questions that often cause confusion. If two sources disagree about a refund deadline, resolve that conflict before asking the model to answer refund questions.
Separate answers from actions
Explaining how a booking works and creating a booking have different consequences. Treat them as separate capabilities with explicit permissions.
For example, a customer might ask, “Can someone call me tomorrow?” The agent may need a name, a contact method, a preferred time, and enough context for the team to respond. It should not request unrelated personal information just because the chat interface makes it easy to ask.
The owner inbox is part of the product, not an afterthought. A useful handoff includes the request, relevant conversation context, and the status of any attempted action. A team member should be able to see whether something was submitted, failed, or still needs confirmation.
Keep channel behavior consistent
A website conversation and a messaging conversation should use the same approved knowledge and action boundaries. Otherwise a policy update can leave one channel giving an old answer while another gives a new one.
The Make Agent Fast case study describes a shared agent identity, knowledge, conversations, and handoff model across the widget and messaging connectors. This is an architectural choice: channel-specific delivery can vary while the business rules remain consistent.
Voice adds another interface to that model. Check how consent, interruptions, names, and contact details behave in the actual voice experience. A text transcript alone cannot prove that the spoken interaction worked well.
Test failure paths before launch
A short evaluation set should include ordinary questions and situations where the agent should stop or ask for help:
| Situation | What to check |
|---|---|
| A question with an approved answer | The reply matches the source and preserves its conditions. |
| A question absent from the knowledge base | The agent explains the gap and offers an appropriate next step. |
| Two conflicting sources | The conflict is resolved or escalated instead of silently guessed. |
| An incomplete booking request | Only the missing information is requested; no confirmation is invented. |
| A failed external action | The customer sees an accurate status and a recovery path. |
| A request for a human | The request reaches the inbox with useful context. |
| A policy update | Every connected channel uses the revised material. |
Run these checks through the real customer interface and the receiving inbox. Seeing a successful model response does not prove that a request reached the team.
Measure useful outcomes
Message count alone says little about support quality. More messages may mean an engaged customer, or they may mean the agent failed to understand a simple request.
Start with outcomes you can inspect: correct answers against approved sources, completed request capture, successful handoffs, and unresolved questions that deserve better documentation. Review representative conversations before changing prompts. Sometimes the fix belongs in the source material or the action workflow.
The goal is an agent that knows its job, works within clear boundaries, and makes the next step easier for the customer and the team.
Read the Make Agent Fast case study for the implemented product and stack, browse other projects, or contact me about an AI product workflow.
Related case study
Make Agent Fast
Built and launched a company-facing platform that turns approved business knowledge into customer agents for websites and messaging channels, with grounded answers, voice, structured actions, and human handoff.
- Shipped one agent workflow across the website widget, Telegram, WhatsApp, Messenger, Instagram, Discord, and KakaoTalk.
- Turned a pasted company URL into a reviewable agent draft, approved knowledge base, embeddable widget, and owner inbox.
Start a project
Need this level of product thinking in your build?
Reach out if you need someone who can shape the product and ship the interface, system, and iteration loop.
Start a project