Case study · Public sector
Answers from your documents, respecting who can see what
Staff in large organisations spend hours looking for policies, procedures and past decisions. This is how we built AI search over SharePoint and document stores that answers questions with sources and only uses documents each person is allowed to see.
Project overview
Permission-aware AI search.
- Industry
- Public sector
- Client
- Confidential
- Solution
- Knowledge search
- Platform
- Azure OpenAI + AI Search
- Timeline
- 8–12 weeks
- Team
- 3 people
- Sources
- SharePoint, policies, procedures
- Security
- Existing permissions enforced
- Interface
- Teams and intranet
Expected business outcomes
What the client set out to achieve
Measured by answer quality on a test set and time saved in a pilot.
-
30–50%
Less time searching for information
Staff ask a question instead of browsing folders.
Target range; measured in a pilot survey
-
Enforced
Document permissions
Answers only use documents the user can already open.
Tested with permission test cases
-
90%+
Grounded answers on the test set
Answers supported by cited documents.
Launch threshold agreed with the information manager
Business challenges
Where the client started
Information exists but can't be found.
Documents
Thousands of files across SharePoint sites and drives.
Search
Keyword search returns long lists of old versions.
Policies
Staff ask colleagues instead of reading policies.
Versions
Outdated documents are found before current ones.
Risk
Concern that AI might reveal documents to the wrong people.
Proposed solution
What the build covers
Search that respects the organisation's existing access controls.
-
Indexing
Selected libraries indexed with metadata and versions.
-
Permission trimming
Results filtered by the user's existing permissions.
-
Answers with sources
Answers generated from retrieved documents with citations.
-
Freshness rules
Current versions preferred; archived content excluded.
-
Teams and intranet
Available where staff already work.
-
Evaluation and feedback
Test questions and user feedback monitored.
Project requirements
What the client asked us for
The request was 'ChatGPT for our documents'. The essential requirement is permissions: nobody should see content they couldn't open themselves.
Content clean-up is part of the project, because AI makes outdated documents easier to find too.
What the project had to deliver
- 01Answers with sources
- 02Permissions enforced
- 03Current content preferred
- 04Available in Teams
Key challenges & considerations
What made it harder, and what we did
-
Inconsistent permissions on old sites
What we did
Permissions were tested explicitly, and sites with broken permissions were excluded until fixed.
-
Duplicate and outdated documents
What we did
Indexing began with maintained content, and outdated documents were flagged to their owners.
-
Sensitive content categories
What we did
Sensitive categories were excluded by rule.
-
Records management requirements
What we did
Retention labels were respected, and archived material was left out.
What we watched for
- Test permissions explicitly
- Start with maintained content
- Exclude archived material
- Use feedback to improve content
Development process
Scope, secure, evaluate
Start with well-maintained content, then expand.
-
Content scope
High-value, well-maintained libraries chosen first.
-
Permissions
Permission model reviewed and enforced in search.
-
Build
Indexing, answers and Teams interface.
-
Evaluation
Test questions and permission tests passed.
-
Expand
More content added after clean-up.
Technology stack
Permission-aware RAG
Microsoft's search and AI services within the tenant.
AI
- Azure OpenAI
- Retrieval-augmented generation
- Prompt and output guardrails
Data
- Azure AI Search
- Document pipelines
- Vector index
Application
- Python / .NET services
- Web or Teams interface
Evaluation
- Test question sets
- Quality and safety metrics
- Usage monitoring
Quality assurance & deployment
Measured against a test set before launch
AI quality was measured, not judged from a demo. A test set of real examples with agreed answers decided whether the system was ready, and the same set was re-run after every change.
- A test set of real examples with agreed answers
- Accuracy thresholds agreed with the business before launch
- Safety checks: refusals, permissions and prompt-injection tests
- A person reviewing low-confidence output
- Cost and response time measured at real volumes
Deployment
- 01EvaluationTest questions and permission tests passed.
- 02ExpandMore content added after clean-up.
- 03Pilot, then staged rolloutA small group of real users first, with results reviewed weekly before widening access.
- 04Monitoring and warrantyAccuracy, drift, usage and cost tracked after launch, and six months of free bug fixing.
Final deliverables
Knowledge that's easy to find, and safe
Staff ask questions in Teams and get answers with links to current documents they are allowed to see.
Information managers see which questions go unanswered and which documents need updating.
- Answers with sources
- Enforced permissions
- Current content first
- Content gap insight
Handed over to the client
- 01Production AI serviceDeployed in the client's own cloud tenant
- 02Source code and modelsIn a repository the client can access
- 03Test set and resultsThe evidence behind the launch decision
- 04MonitoringAccuracy, usage and cost
- 05Six-month warrantyFree bug fixing after delivery
Success metrics
How success is measured
| Measure | Outcome | How it is measured |
|---|---|---|
| Less time searching for information | 30–50% | Target range; measured in a pilot survey |
| Document permissions | Enforced | Tested with permission test cases |
| Grounded answers on the test set | 90%+ | Launch threshold agreed with the information manager |
Verified reviews
What clients say about our custom software work
Verified reviews from our clients on the same kind of work, published on Clutch.
-
Custom Software Development
Web-Based Coupon App Development
The work produced by the team was of exceptional quality.
Jon Thies
CTO, Model Rocket Feb 2022 – Mar 2023
-
Custom Software Development
Mobile & Web Platform Development & Design
Silver Scintilla was responsive, respectful, and enjoyable to work with, quickly addressing any questions or requests.
DeShawn Brown
CEO & Founder, Lithios Jun 2021 – Nov 2022
-
Custom Software Development
Web-Based HRM System Development
Silver Scintilla provided the best solution with scalable functionality, personalized features, and robust security.
Suheb Khan
Director, Collaborative Insight Technologies Jul 2020 – Jun 2022
Free discovery call
Have an idea? Let's turn it into AI-powered software.
Book a free discovery call with our experts. Share your idea and we will help you shape the scope, timeline and budget, under NDA.
- Free consultation
- NDA before we talk
- Transparent estimate