Worked Example: Planning an AI Customer Support Agent

This is a worked example, not a client case study. The company is fictional and no results are claimed: the point is to show, step by step, how we plan an AI agent project and the decisions involved. The process, the checks, and the cost estimate are the ones we use; the scenario is representative of the requests we see most often.

The scenario

A mid-size online retailer runs customer support by email and website chat. Most contacts are repetitive — where is my order, can I return this, does this product fit or work with that — and peak seasons create long queues. The company wants an AI agent that resolves the routine questions itself and hands everything else to its support team with the context already gathered.

Step 1: Decide what the agent does — and doesn't

• Handles: order status (by looking up the order system), return eligibility (by applying the published returns policy), and product questions (from the product catalogue and help centre).
• Prepares but does not send: refund and replacement requests, which a person approves with one click.
• Always hands to a person: complaints, damaged-goods claims, anything legal or safety-related, and any conversation where the customer asks for a human.

Step 2: Check the data

• Help-centre articles and the returns policy — available, but some articles are out of date, so a content review is scheduled before launch.
• Product catalogue — available by export; specifications are inconsistent across older products.
• Order system — has an API; read-only access is enough for status lookups.
• Past support tickets — thousands exist but none are labelled, so support staff label a representative sample of around 200 to build the evaluation set.
• Personal data — customer names, addresses, and order details are masked in logs and never used for training.

Step 3: Choose RAG, not fine-tuning

Every answer the agent gives depends on facts that change — order status, policy wording, stock and product details — and customers and staff need to know where an answer came from. That points to retrieval-augmented generation (RAG) over the help centre and catalogue, plus live tool calls to the order system. Fine-tuning is not planned for launch: there is no consistent-format or tone problem that good prompting cannot solve, and it would not keep the facts current. It stays an option if evaluation later shows a behaviour gap.

Step 4: Agree how accuracy is measured

Before building, the retailer's support leads and our team agree what a correct answer looks like for each type of question, and set the thresholds the agent must meet before launch. In a plan like this, the thresholds might be: answers grounded in a source for nearly all policy and product questions; correct handover on every complaint and safety case in the test set; no refund actions without approval; and response times within a few seconds. The exact numbers are the retailer's decision — what matters is that they are written down first.

• Evaluation set: around 200 labelled real tickets, including edge cases and requests the agent must refuse or hand over.
• Metrics reported separately: task success, groundedness, handover accuracy, format validity, latency, and cost per conversation.
• Every change to prompts, retrieval, or the model is re-scored, and releases below threshold don't ship.

Step 5: Build in guardrails

• Read-only access to the order system; refunds and replacements are drafted for staff approval, never executed by the agent.
• Protection against prompt injection in customer messages, such as instructions to ignore policy.
• A confidence threshold below which the agent hands over instead of guessing.
• A full log of each conversation's sources and tool calls, so any answer can be traced and reviewed.

Step 6: Estimate the cost

Delivered as a new web application — the customer chat, a staff dashboard for handovers and approvals, an admin portal, several integrations (order system, helpdesk, catalogue), a few user roles, and advanced AI — the model behind our free Software Cost Calculator estimates about 1,930 hours, $59,000–$88,000, roughly 15 weeks, and a team of seven: a part-time project manager, a designer, three software engineers, a QA engineer, and an AI/ML engineer. It rates the risk Moderate because custom AI work carries research uncertainty. That is an indicative range, not a quote; a short discovery would firm it up.

Step 7: Roll out in stages

• Shadow mode — the agent drafts replies that staff review and send, which builds the evaluation set and shows real accuracy without customer risk.
• Limited autonomy — the agent answers order-status questions on its own, where answers come straight from the order system.
• Wider autonomy — returns and product questions are added one at a time, each only after it meets its threshold.
• Ongoing review — a weekly sample of conversations is checked by people, and every new failure joins the evaluation set.

What success would be measured by

The retailer would track the share of conversations resolved without a person, time to resolution, customer satisfaction on agent-handled conversations compared with staff-handled ones, handover accuracy, and cost per resolved conversation — against a baseline taken before launch. Those measurements, not the demo, decide whether to expand the agent's scope.