Cloud alerting and on-call: thresholds, noise reduction and runbooks

What makes a good alert

An alert interrupts a person, sometimes at three in the morning. It should therefore be actionable (someone needs to do something), urgent enough for the channel used, and clear about what is wrong and where. Anything that fails those tests belongs on a dashboard, in a ticket queue or in a daily report instead.

Choosing what and when to alert

• Alert on symptoms, not every cause. Users notice slow pages and errors, not CPU percentages. "Checkout error rate above 2 percent for 5 minutes" is more meaningful than "CPU above 80 percent".
• Use duration windows. Requiring a condition to last several minutes avoids paging for brief spikes.
• Use service level objectives (SLOs). If the target is 99.9 percent successful requests over 30 days, alert when the error budget is burning too fast, rather than on every error.
• Use anomaly detection carefully. Cloud services offer it for metrics with daily patterns, but test it before trusting it with pages.
• Security alerts follow the same rules: root or break-glass sign-in, logging disabled, new admin role assignments and public exposure of sensitive data are high-value, low-volume alerts.

Reducing noise

Alert fatigue happens when people see so many alerts that they start ignoring them, and real incidents slip through. Practical fixes:

• Deduplicate and group related alerts, so a failed region produces one incident, not 400 messages.
• Suppress dependent alerts during known outages and planned maintenance windows.
• Delete or downgrade any alert that nobody acted on in the last month.
• Track numbers: pages per on-call shift, out-of-hours pages, and how many were actionable.

Runbooks and on-call

A runbook is a short guide linked from the alert. It explains what the alert means, how to confirm impact, first steps to mitigate, who to escalate to and how to roll back. Keep runbooks next to the code or in a shared wiki and update them after incidents.

1. Give every alert an owning team and a runbook link.
2. Use an on-call tool for schedules, escalation and acknowledgement.
3. Keep rotas fair, with handovers and enough people to rest.
4. After incidents, hold blameless reviews and fix the alert rules too.

Tip: try a budget alert in your own cloud account as a first, low-risk example; see the budget alerts lab.