Case study

AI-assisted customer support

I built a tool that searches past answers and product data, drafts replies, and gives experts the evidence to review them.

Client details are anonymized. Camera gear stands in for the original domain.

A specialist company’s product knowledge was scattered across emails, chats, and product documentation. I built a support tool that brings that knowledge together and drafts replies for expert review, preserving their judgment and voice.

I structured the knowledge so the agent could retrieve it, designed tools for investigating each question, and built an interface where experts could check and refine the advice.

I haven't covered the evaluation system here. I'll write about that in a separate post.

1. Expert review

Keep experts in the loop

The right advice depended on the customer’s setup, experience, and conditions. The client wanted to preserve the personal connection and the experts’ judgment. I kept an expert responsible for every response, with the agent handling search and the first draft.

From question to reviewed answer

1
Question enters the queue
Support adds the customer’s question.
2
An expert claims it
The right person picks it up.
3
The agent drafts
It searches the knowledge base and product catalog.
4
The expert refines
They edit and approve the response.

Save the answer and the correction

Approval saves the final response to the searchable archive. The expert's edits and revision instructions become context for later questions, so the agent can retrieve their corrections alongside past answers.

2. Data structure

Give the knowledge structure

The system uses two sources: expert-approved Q&As and a product catalog with specifications, price ranges, and compatibility notes.

Budget, experience, and use case all mattered to a recommendation. I made those explicit fields in the records. For an archive of hundreds of Q&As, I started with model-based selection from summaries rather than a vector database.

Sending the entire archive with every question would add a lot of irrelevant material. Large context windows do not remove that problem: performance can degrade as context grows.

The context engineering work was mostly in the data: consistent records, short summaries for finding candidates, and full answers available when needed.

Turn the archive into records

I converted 500+ email Q&As and product documents into structured records. Coding agents processed the archive in parallel batches of 20–30 emails and flagged ambiguous answers, malformed data, and possible errors for review.

Resolving those flags helped improve the next extraction round. Each saved record received an ID and a short retrieval summary, alongside the extracted fields.

Extraction prompt

Extract Q&As from this chunk. For each:
- question: the customer's actual question
- answer: the expert's response
- budget_range: if mentioned (e.g., "$1500-2000")
- experience_level: if mentioned (beginner/enthusiast/pro)
- products_mentioned: array of product names
- use_case: what they're trying to shoot

Return as JSON array. Skip email headers and signatures.

Saved Q&A record

{
  "id": "qa_127",
  "question": "I'm an enthusiast with a $2k budget. What camera would you recommend for low light?",
  "answer": "I'd consider the Alpha 200 for its full-frame sensor and low-light performance.",
  "budget_range": "Around $2000",
  "experience_level": "enthusiast",
  "products_mentioned": ["Alpha 200"],
  "use_case": "low-light photography",
  "summary": "Enthusiast seeking a low-light camera around $2000"
}

Filter summaries, then retrieve the detail

In early 2025, the first version gave a model all the summaries and selected fields. It chose relevant Q&A IDs by customer profile, product, and use case. A separate call drafted the answer using the full text of those records.

3. Agent architecture

From workflows to agents

The original workflow

The first version used three sequential model calls: select Q&As, draft a response, then extract metadata after expert approval. Expert feedback could trigger another draft, but retrieval followed the same sequence every time.

The initial pipeline

1
Filter
Read the summaries and select up to 15 relevant Q&As.
2
Generate
Draft from the selected Q&As, summarized catalog, and system prompt. Model choice scales with complexity.
3
Extract
After expert approval, save metadata, expert edits, and correction patterns to the knowledge base.

This worked for straightforward questions. When a question had several parts or the first search missed useful evidence, the workflow had no way to change its search strategy.

The agent version

I replaced the fixed pipeline with an agent using the same data. It could search summaries, fetch a full answer, try keywords, or look up a product, then choose another tool if it needed more evidence.

The catalog became searchable on demand, so the agent could retrieve specific product details instead of receiving the whole catalog with every question.

Illustrative retrieval trace“Best camera for a beginner, low light, around $2k?”

A scripted example: the first answer is relevant to low light, but comes from a more experienced customer. The agent searches for a comparison before drafting.

search_qas
beginner camera for low light
Found related answers, including Q&A #127.
get_qa_by_id
qa_127
Similar budget and low-light needs, but this customer was an enthusiast. Beginner suitability is still unresolved.
search_qas
Alpha 200 vs Alpha 100 for low light
Refocused on product tradeoffs. Found comparison Q&A #203.
get_qa_by_id
qa_203
The comparison covers specifications, but does not establish how easy either camera is to learn.
search_products
Alpha 200 and Alpha 100
Checked catalog prices and specifications: $1,199 for the full-frame Alpha 200; $699 for the APS-C Alpha 100.
submit_answer
structured response
Returned a provisional recommendation, source caveats, and a question about the customer’s needs for the expert to review.

Follow-up questions

An expert can add a budget constraint or ask for a comparison without restating the original question. The agent keeps the products and customer requirements in context across turns.

One conversation, three turns

A scripted example using the anonymized camera catalog.

Expert

What's the best camera for a beginner?

Tool activity (3 calls)

search_qas
Searching for beginner recommendations...
Found 18 Q&As about beginners
search_products
Checking entry-level camera options...
4 cameras tagged as beginner-friendly
get_qa_by_id
Reading past advice for beginners...
Retrieved Q&A #89, #234

Agent draft

The Alpha 100 costs $699; the Alpha 200 costs $1,199 and has a higher low-light rating in the catalog. I’d ask about budget and what they want to photograph before recommending either. Prior beginner Q&As give us examples to discuss, but they don’t tell us this customer’s priorities.

Sources: Q&A #89, Q&A #234, Product Catalog

Agent confidence: medium

Alternatives: Alpha 100 ($699; lower catalog low-light rating); Alpha 300 ($2,400; ask about budget before considering)

4. UI design

Make judgment easy to apply

Design the UI for expert review

I designed the response schema and review interface together. The draft is editable, its supporting Q&As are linked, and each source includes why it applies and where the comparison falls short.

Alternative products appear with their tradeoffs. An expert can choose one, request a revised draft, or edit the response directly before approving it. The agent's confidence assessment is shown alongside the evidence.

Each field has a job in the UI: source IDs open evidence, caveats flag differences between customers, and alternatives give the expert a specific change to request.

Inspect the structured response behind the interface

{
  "response": "For low light, I’d consider the Alpha 200 at $1,199. The Alpha 100 costs $699, but the catalog rates its low-light performance lower. The closest past recommendation was for an enthusiast. What will you be photographing, and how comfortable are you with camera settings? That would help me judge whether it’s a good fit for you.",
  "confidence": "medium",
  "confidence_explanation": "Sources support the low-light comparison, but beginner suitability needs clarification",
  "sources": [
    { "id": "qa_127", "relevance": "Same budget, same low-light priority",
      "caveat": "Customer was enthusiast-level, not beginner" },
    { "id": "qa_203", "relevance": "Direct comparison of Alpha 100 and Alpha 200",
      "caveat": "Focused on specs, not beginner-friendliness" }
  ],
  "alternatives": [
    { "product": "Alpha 100", "considered": "Under budget, beginner-friendly",
      "why_not": "Lower low-light rating in the sample catalog" },
    { "product": "Alpha 300", "considered": "Best low-light in lineup",
      "why_not": "Over budget at $2,400" }
  ]
}

The expert's review desk

Illustrative interface

Customer question

Best camera for a beginner, low light, around $2k?

Draft response. Edit before approving.

Agent confidence: medium. Sources support the comparison; beginner suitability needs clarification.

View reasoning & sources

Agent assessment: medium confidence. The sources support considering the Alpha 200 for low light within a $2,000 budget. Neither source establishes its suitability for this beginner; the draft asks about intended use and comfort with camera settings.

Q&A #127

An expert recommended the Alpha 200 for a customer with a similar budget and low-light needs.

Caveat: An enthusiast, rather than a beginner. Explain the learning curve.

Q&A #203

A direct comparison of Alpha 100 and Alpha 200 specifications and prices.

Caveat: Focused on specifications. Supplement with beginner-friendly guidance.

Try a different product instead

5. Tool design

Build tools that carry forward

The tools determine what the agent can retrieve and what it must return. I gave it separate operations for finding candidates, checking exact terms, and reading full records.

I refine those operations through evaluations and real usage. Most have been rewritten several times as new edge cases exposed gaps.

Give each search tool a clear purpose

search_qas

Find relevant examples

The model reads all summaries and searches by meaning, including the customer’s constraints.

keyword_search

Find an exact match

Direct text search across normalized records. Consistent formatting makes specific terms easier to retrieve.

get_qa_by_id

Read the full context

Fetch the complete conversation once a promising example has been found.

Make the output a contract

The submit_answer tool requires a customer response, a confidence assessment and explanation, sources with relevance and caveats, and alternatives with the reasons they were considered. The review UI depends on those fields, so I require them at the tool boundary.

Reuse the tools in a research agent

I reused the Q&A search, product lookup, and pattern-analysis tools in a second agent for brand research. It combines the same customer and product knowledge with web research, document tools, and spreadsheet analysis.

Reused from support

Q&A search and pattern analysis
Product catalog and specifications
Brand knowledge base

Added for research

Web search for competitive intelligence
Google Docs reading and writing
Spreadsheet analysis and editing

Because the support team adds reviewed answers to the same knowledge base, the research agent can use what the company has learned from more recent customer questions.

More work & writing