Case study · B2B software
Answers from your documentation, with sources
A support assistant is only useful if its answers are right and customers can check them. This is how we built one grounded in the client's own documentation and past tickets, with citations, a clean handover to people and an evaluation set that proves quality before launch.
Project overview
A retrieval-augmented support assistant.
- Industry
- B2B software & services
- Client
- Confidential
- Solution
- AI support assistant
- Platform
- Azure OpenAI + AI Search
- Timeline
- 8–12 weeks
- Team
- 4 people
- Sources
- Docs, knowledge base, resolved tickets
- Channels
- Website, portal, in-app
- Safety
- Citations and human handover
Expected business outcomes
What the client set out to achieve
Quality is measured on a test set before launch and on real conversations after it.
-
20–40%
Of common questions resolved without an agent
The assistant answers routine how-to questions from documentation.
Target range; measured on real conversations after launch
-
90%+
Answer accuracy on the test set
Answers checked against an agreed set of real questions before go-live.
Launch threshold agreed with the support team
-
Always
Source links on answers
Every answer links to the documents it used.
Answers without a source are not shown
Business challenges
Where the client started
The knowledge exists; customers just can't find it.
Documentation
Hundreds of articles, hard to search.
Tickets
Many tickets ask questions the docs already answer.
Agents
Senior agents repeat the same explanations every day.
Chatbot
A previous rule-based bot frustrated customers.
Concern
Leadership worries an AI will invent answers.
Proposed solution
What the assistant build covers
Retrieval, grounded answers and safety measures, built and tested together.
-
Content pipeline
Documentation and resolved tickets indexed, chunked and kept in sync.
-
Grounded answers
Answers generated only from retrieved content, with citations.
-
Human handover
Conversation passed to an agent with context when the assistant isn't sure.
-
Guardrails
Topic limits, refusal of unsupported questions and prompt-injection defences.
-
Evaluation set
Real questions with approved answers used to test every change.
-
Monitoring
Usage, unanswered questions and feedback reviewed weekly.
Project requirements
What the client asked us for
The request was fewer tickets. The condition is trust: the assistant must say 'I don't know' rather than guess, and show where each answer came from.
We build the evaluation set with the support team before the assistant, so quality has a definition everyone agreed.
What the project had to deliver
- 01Answers only from the client's content
- 02Sources on every answer
- 03Smooth handover to people
- 04Quality measured before launch
Key challenges & considerations
What made it harder, and what we did
-
Outdated or conflicting documentation
What we did
Conflicting articles surfaced by the evaluation set were sent to the documentation team to fix first.
-
Customer-specific configurations
What we did
Answers were limited to content that matched the customer's product and version.
-
Questions that need account data
What we did
Questions that needed account data were handed to an agent with the conversation attached.
-
Regulated or safety-critical topics
What we did
Regulated and safety-critical topics were excluded and always routed to a person.
What we watched for
- Build the evaluation set first
- Never show answers without sources
- Fix documentation gaps it reveals
- Review real conversations weekly
Development process
Evaluate first, then launch
The launch decision is based on the evaluation set, not a demo.
-
Question set
Real questions and approved answers collected from past tickets.
-
Content pipeline
Documentation indexed and gaps listed for the docs team.
-
Assistant
Retrieval, answers, citations and handover built.
-
Evaluation
Tested against the question set until it meets the agreed threshold.
-
Launch and review
Released to a customer group with weekly quality reviews.
Technology stack
Retrieval-augmented generation on Azure
Models run in the client's Azure tenant; content stays under their control.
AI
- Azure OpenAI
- Retrieval-augmented generation
- Prompt and output guardrails
Data
- Azure AI Search
- Document pipelines
- Vector index
Application
- Python / .NET services
- Web or Teams interface
Evaluation
- Test question sets
- Quality and safety metrics
- Usage monitoring
Quality assurance & deployment
Measured against a test set before launch
AI quality was measured, not judged from a demo. A test set of real examples with agreed answers decided whether the system was ready, and the same set was re-run after every change.
- A test set of real examples with agreed answers
- Accuracy thresholds agreed with the business before launch
- Safety checks: refusals, permissions and prompt-injection tests
- A person reviewing low-confidence output
- Cost and response time measured at real volumes
Deployment
- 01EvaluationTested against the question set until it meets the agreed threshold.
- 02Launch and reviewReleased to a customer group with weekly quality reviews.
- 03Pilot, then staged rolloutA small group of real users first, with results reviewed weekly before widening access.
- 04Monitoring and warrantyAccuracy, drift, usage and cost tracked after launch, and six months of free bug fixing.
Final deliverables
An assistant customers can trust
Customers get quick answers to routine questions with links to the source, and anything uncertain goes to a person with the conversation attached.
Unanswered questions show the docs team exactly which articles to write next.
- Grounded, cited answers
- Human handover
- Tested quality
- A feedback loop for documentation
Handed over to the client
- 01Production AI serviceDeployed in the client's own cloud tenant
- 02Source code and modelsIn a repository the client can access
- 03Test set and resultsThe evidence behind the launch decision
- 04MonitoringAccuracy, usage and cost
- 05Six-month warrantyFree bug fixing after delivery
Success metrics
How success is measured
| Measure | Outcome | How it is measured |
|---|---|---|
| Of common questions resolved without an agent | 20–40% | Target range; measured on real conversations after launch |
| Answer accuracy on the test set | 90%+ | Launch threshold agreed with the support team |
| Source links on answers | Always | Answers without a source are not shown |
Verified reviews
What clients say about our custom software work
Verified reviews from our clients on the same kind of work, published on Clutch.
-
Custom Software Development
Web-Based Coupon App Development
The work produced by the team was of exceptional quality.
Jon Thies
CTO, Model Rocket Feb 2022 – Mar 2023
-
Custom Software Development
Mobile & Web Platform Development & Design
Silver Scintilla was responsive, respectful, and enjoyable to work with, quickly addressing any questions or requests.
DeShawn Brown
CEO & Founder, Lithios Jun 2021 – Nov 2022
-
Custom Software Development
Web-Based HRM System Development
Silver Scintilla provided the best solution with scalable functionality, personalized features, and robust security.
Suheb Khan
Director, Collaborative Insight Technologies Jul 2020 – Jun 2022
Free discovery call
Have an idea? Let's turn it into AI-powered software.
Book a free discovery call with our experts. Share your idea and we will help you shape the scope, timeline and budget, under NDA.
- Free consultation
- NDA before we talk
- Transparent estimate