Testing resilience: failover tests and chaos engineering
In short: Redundancy that has never been tested is only a hope. Failover tests and DR drills check that planned recovery works; chaos engineering goes further, injecting controlled faults to find weaknesses before real incidents do. Start small, define normal behaviour first and always have a way to stop.
Why test resilience
Standby servers drift out of date, certificates expire, runbooks go stale and failover scripts break silently. Many outages happen not because there was no redundancy, but because nobody had checked that it worked. Testing turns assumptions into evidence and gives teams practice so real incidents are calmer.
Levels of testing
| Method | What it involves | Good for |
|---|---|---|
| Tabletop exercise | Talk through a scenario with the team | Checking roles, contacts and runbooks |
| Component failover test | Planned switch of a pair, such as firewalls or database | Proving each HA pair works |
| DR drill | Run the service from the DR site | Measuring real RTO and RPO |
| Game day | Scheduled, realistic failure with the team watching | Testing people, tools and process together |
| Chaos engineering | Controlled, often automated fault injection | Finding unknown weaknesses continuously |
Chaos engineering
Chaos engineering is the practice of running careful experiments on a system to build confidence that it can handle turbulent conditions. It was popularised by Netflix with tools that randomly terminated instances. The method is scientific:
- Define steady state using user-facing metrics, such as successful orders per minute.
- Form a hypothesis, for example "if one web instance dies, the success rate stays above 99.9%".
- Inject a fault with a small blast radius: one instance, a small share of traffic, a test environment first.
- Observe metrics, alerts and logs. Did the system and the on-call team behave as expected?
- Learn and fix, then repeat or widen the experiment.
Tools include open-source projects such as Chaos Mesh and LitmusChaos for Kubernetes, and managed services from cloud providers such as AWS Fault Injection Service and Azure Chaos Studio. Simple scripts work too. On a lab Linux host you can add network delay to test timeouts:
sudo tc qdisc add dev eth0 root netem delay 200ms
sudo tc qdisc del dev eth0 rootDoing it safely
- Get approval from service owners and run experiments only on systems you are authorised to change.
- Have a clear abort switch and automatic stop conditions if key metrics drop.
- Start in test environments, then production at quiet times with a small blast radius.
- Tell the support desk and on-call team, unless the aim is to test detection, in which case have a senior observer ready to stop it.
- Record findings and track fixes like any other bug.
Tip: chaos experiments also test security monitoring. Check whether your SIEM alerts when a host disappears or logging stops, since attackers often disable logs first.
For exams and interviews, stress that chaos engineering is controlled and hypothesis-driven, and that it supports SLOs and error budgets (see availability and SLOs).
Practice
What is meant by "blast radius" in a chaos experiment?
Why define steady state before injecting a fault?
Which test would best measure the real RTO of a DR site?
Key takeaways
- Untested redundancy is only an assumption.
- Tabletops, failover tests, DR drills and game days build confidence step by step.
- Chaos engineering is a controlled, hypothesis-driven experiment.
- Keep the blast radius small and always have an abort switch.
- Only test systems you are authorised to change; track fixes.
Original content written by Mohamed Fayas for FM TecHub. © 2026 FM TecHub. You're welcome to share a link to this page; please don't copy the text elsewhere without permission.