Case study · eCommerce
Recommendations that earn their place on the page
Many online shops show 'you may also like' boxes that nobody clicks. This is how we built recommendations from browsing and purchase behaviour for site and email, and prove with A/B tests that they beat simple rules before they replace them.
Project overview
Personalised recommendations with A/B testing.
- Industry
- eCommerce
- Client
- Confidential
- Solution
- Recommendations
- Platform
- Azure ML + storefront
- Timeline
- 8–12 weeks
- Team
- 3 people
- Placements
- Product page, basket, email
- Signals
- Views, purchases, attributes
- Measurement
- A/B tests
Expected business outcomes
What the client set out to achieve
Only A/B-tested uplift counts.
-
3–10%
Higher revenue per session in test groups
Personalised suggestions compared with existing rules.
Target range, measured by A/B test
-
Tested
Every placement
Each recommendation box tested before full rollout.
A/B framework
-
Daily
Model refresh
Recommendations updated as behaviour changes.
Batch retraining
Business challenges
Where the client started
Recommendations exist but aren't measured.
Boxes
'Related products' chosen by category rules.
The same products sent to everyone.
Data
Browsing and purchase data unused.
Measurement
No idea whether recommendations help.
Catalogue
New products rarely recommended at all.
Proposed solution
What the build covers
Models, placements and the testing framework together.
-
Behaviour data
Views, basket adds and purchases collected consistently.
-
Models
Collaborative and content-based models, including for new products.
-
Site placements
Recommendations on product, basket and home pages.
-
Email recommendations
Personalised products in emails.
-
A/B testing
Every placement tested against current rules.
-
Business rules
Out-of-stock, margin and brand rules applied.
Project requirements
What the client asked us for
The request was personalisation. The discipline is testing: a model replaces a rule only when it wins an A/B test.
Business rules stay in control, so recommendations never push out-of-stock or unwanted products.
What the project had to deliver
- 01Personalised suggestions
- 02Site and email
- 03Business rules respected
- 04A/B-proven uplift
Key challenges & considerations
What made it harder, and what we did
-
Sparse data for new customers
What we did
New customers saw popular and contextual picks until enough behaviour existed.
-
Frequent catalogue changes
What we did
The catalogue synced daily, so new and discontinued products were handled.
-
Platform limits on placements
What we did
Placements were chosen within the platform's limits, and each one was tested.
-
Consent for personalisation
What we did
Personalisation ran only with consent, with a non-personalised fallback.
What we watched for
- Fix tracking first
- Test every placement
- Handle new products and customers
- Respect consent
Development process
Test before you switch
Rules remain until a model beats them.
-
Tracking
Behaviour tracking checked and fixed.
-
Models
Recommendation models built offline.
-
First placement
One placement A/B tested.
-
Expand
More placements and email tested.
-
Operate
Daily refresh with ongoing tests.
Technology stack
Practical recommendations
Proven methods served where customers shop.
Machine learning
- Python
- scikit-learn / LightGBM
- Azure Machine Learning
Data
- Microsoft Fabric or Azure SQL
- Feature pipelines
Serving
- Batch scoring
- REST API
- ERP/CRM integration
Monitoring
- Accuracy tracking
- Drift alerts
- Retraining schedule
Quality assurance & deployment
Measured against a test set before launch
AI quality was measured, not judged from a demo. A test set of real examples with agreed answers decided whether the system was ready, and the same set was re-run after every change.
- A test set of real examples with agreed answers
- Accuracy thresholds agreed with the business before launch
- Safety checks: refusals, permissions and prompt-injection tests
- A person reviewing low-confidence output
- Cost and response time measured at real volumes
Deployment
- 01ExpandMore placements and email tested.
- 02OperateDaily refresh with ongoing tests.
- 03Pilot, then staged rolloutA small group of real users first, with results reviewed weekly before widening access.
- 04Monitoring and warrantyAccuracy, drift, usage and cost tracked after launch, and six months of free bug fixing.
Final deliverables
Recommendations with proof
Customers see relevant products on site and in email, and every placement has proven its value in an A/B test.
New products get recommended from day one using their attributes.
- Personalised placements
- Email recommendations
- A/B-tested uplift
- Business rules applied
Handed over to the client
- 01Production AI serviceDeployed in the client's own cloud tenant
- 02Source code and modelsIn a repository the client can access
- 03Test set and resultsThe evidence behind the launch decision
- 04MonitoringAccuracy, usage and cost
- 05Six-month warrantyFree bug fixing after delivery
Success metrics
How success is measured
| Measure | Outcome | How it is measured |
|---|---|---|
| Higher revenue per session in test groups | 3–10% | Target range, measured by A/B test |
| Every placement | Tested | A/B framework |
| Model refresh | Daily | Batch retraining |
Verified reviews
What clients say about our custom software work
Verified reviews from our clients on the same kind of work, published on Clutch.
-
Custom Software Development
Web-Based Coupon App Development
The work produced by the team was of exceptional quality.
Jon Thies
CTO, Model Rocket Feb 2022 – Mar 2023
-
Custom Software Development
Mobile & Web Platform Development & Design
Silver Scintilla was responsive, respectful, and enjoyable to work with, quickly addressing any questions or requests.
DeShawn Brown
CEO & Founder, Lithios Jun 2021 – Nov 2022
-
Custom Software Development
Web-Based HRM System Development
Silver Scintilla provided the best solution with scalable functionality, personalized features, and robust security.
Suheb Khan
Director, Collaborative Insight Technologies Jul 2020 – Jun 2022
Free discovery call
Have an idea? Let's turn it into AI-powered software.
Book a free discovery call with our experts. Share your idea and we will help you shape the scope, timeline and budget, under NDA.
- Free consultation
- NDA before we talk
- Transparent estimate