Case study · Healthcare
Know about problems before patients do
Healthcare platforms can't afford long outages or lost data, yet many find out about problems from users. This is how we added monitoring with meaningful alerts, service level objectives, tested backups, disaster recovery in a second region and runbooks the team can follow at 3am.
Project overview
Reliability engineering for a critical platform.
- Industry
- Healthcare
- Client
- Confidential
- Solution
- Reliability and disaster recovery
- Platform
- Azure or AWS
- Timeline
- 10–16 weeks
- Team
- 3 people
- Recovery
- Second region
- Targets
- Agreed RPO and RTO
- Compliance
- Health data regulations
Expected business outcomes
What the client set out to achieve
Reliability is measured with agreed objectives and tested, not assumed.
-
Agreed
Recovery time and data loss objectives
RTO and RPO set per service with the business.
Proven in a disaster recovery test
-
Tested
Restores and failover
Backups restored and failover rehearsed on a schedule.
Test results recorded
-
Faster
Detection of incidents
Alerts on user-facing symptoms reach the right person first.
Measured as time to detect
Business challenges
Where the client started
The platform works; nobody is sure what happens when it doesn't.
Monitoring
Server metrics collected, few meaningful alerts.
Incidents
Users report problems before the team notices.
Backups
Backups configured, restores never tested.
Recovery
No tested plan if the primary region fails.
Knowledge
Incident handling depends on the most experienced engineer.
Proposed solution
What the reliability work covers
From knowing something is wrong to recovering from the worst case.
-
Observability
Logs, metrics and traces with dashboards for each service.
-
Service level objectives
Targets for availability and latency that matter to users.
-
Alerting
Alerts on symptoms users feel, routed to on-call engineers.
-
Backups
Automated backups with scheduled restore tests.
-
Disaster recovery
A second region with replication and rehearsed failover.
-
Runbooks
Step-by-step guides for common incidents and recovery.
Project requirements
What the client asked us for
The request was 'make sure we're safe'. That starts with agreeing what 'safe' means: how much data loss and downtime each service can tolerate.
Disaster recovery is only real once it has been tested, so a full failover rehearsal is part of the project.
What the project had to deliver
- 01Problems detected before users notice
- 02Recovery objectives agreed
- 03Backups and failover proven
- 04Runbooks anyone on call can follow
Key challenges & considerations
What made it harder, and what we did
-
Health data residency rules
What we did
The second region was chosen inside the permitted residency boundary.
-
Stateful services that are hard to replicate
What we did
Stateful services were replicated, and recovery was rehearsed.
-
Cost of a second region
What we did
Recovery objectives were agreed with the business to size the standby cost.
-
Small on-call teams
What we did
Alerts were reduced to symptoms that needed action, so a small on-call team was enough.
What we watched for
- Alert on symptoms, not every metric
- Agree recovery objectives with the business
- Test restores regularly
- Rehearse failover before you need it
Development process
Observe, protect, rehearse
The project ends with a real recovery rehearsal.
-
Objectives
RPO, RTO and SLOs agreed per service.
-
Observability
Monitoring, dashboards and alerts implemented.
-
Backups
Backup coverage checked and restore tests automated.
-
Disaster recovery
Second region set up with replication.
-
Rehearsal
Failover rehearsed, runbooks refined, on-call trained.
Technology stack
Reliability on the major clouds
Native services with open standards for telemetry.
Observability
- Azure Monitor / CloudWatch
- OpenTelemetry
- Dashboards
Alerting
- On-call rotation
- Alert routing
Data protection
- Automated backups
- Geo-replication
Recovery
- Second-region infrastructure as code
- Failover runbooks
Quality assurance & deployment
Rehearsed before production, monitored after
Infrastructure changes were proved before they touched production. Every change was made as code, tested in a copy of the environment and rehearsed with a rollback.
- Infrastructure defined as code and reviewed like application code
- Changes tested in a non-production copy first
- Load tests before any resize or cut-over
- Backup restores and failover rehearsed, not assumed
- Security baselines checked automatically
Deployment
- 01Disaster recoverySecond region set up with replication.
- 02RehearsalFailover rehearsed, runbooks refined, on-call trained.
- 03Rehearsed cut-oversEach change window was planned, rehearsed and reversible.
- 04Monitoring and handoverDashboards, alerts and runbooks handed to the client's team, and six months of free bug fixing.
Final deliverables
Reliability you've actually tested
The team hears about problems from alerts, not users, and every service has agreed recovery objectives.
Backups and failover have been proven in a rehearsal, and runbooks mean recovery doesn't depend on one person.
- Meaningful monitoring
- Agreed SLOs, RPO and RTO
- Tested backups and failover
- Incident runbooks
Handed over to the client
- 01Infrastructure as codeIn the client's repository
- 02PipelinesBuild, test and deploy
- 03Monitoring and alertsDashboards and on-call routing
- 04RunbooksFor releases, recovery and failover
- 05Six-month warrantyFree bug fixing after delivery
Success metrics
How success is measured
| Measure | Outcome | How it is measured |
|---|---|---|
| Recovery time and data loss objectives | Agreed | Proven in a disaster recovery test |
| Restores and failover | Tested | Test results recorded |
| Detection of incidents | Faster | Measured as time to detect |
Verified reviews
What clients say about our custom software work
Verified reviews from our clients on the same kind of work, published on Clutch.
-
Custom Software Development
Web-Based Coupon App Development
The work produced by the team was of exceptional quality.
Jon Thies
CTO, Model Rocket Feb 2022 – Mar 2023
-
Custom Software Development
Mobile & Web Platform Development & Design
Silver Scintilla was responsive, respectful, and enjoyable to work with, quickly addressing any questions or requests.
DeShawn Brown
CEO & Founder, Lithios Jun 2021 – Nov 2022
-
Custom Software Development
Web-Based HRM System Development
Silver Scintilla provided the best solution with scalable functionality, personalized features, and robust security.
Suheb Khan
Director, Collaborative Insight Technologies Jul 2020 – Jun 2022
Free discovery call
Have an idea? Let's turn it into AI-powered software.
Book a free discovery call with our experts. Share your idea and we will help you shape the scope, timeline and budget, under NDA.
- Free consultation
- NDA before we talk
- Transparent estimate