The Best Practices Guide Sent You Here. Now Build the Runbooks.

Ten pre-built sections cover the full incident lifecycle: triggers tied to specific alert names and thresholds, numbered diagnostic steps with decision points and expected command outputs, a 4-step rollback procedure (Stop → Restore → Verify Stability → Document), escalation contacts organized by tier, and copy-paste communication templates for both initial notification and resolution. Every section includes a prompt so no engineer ever faces a blank page at 2 AM.

Triggers on HTTP_500_Error_Rate_Critical when error rate exceeds 0.5% of requests for 3+ minutes. Step 1 verifies scope with a curl test. Step 2 inspects logs — exact grep commands for both Apache (/var/log/apache2/error.log) and Nginx. Step 3 checks infrastructure health. Resolution branches into four paths: database connections, memory exhaustion, application code exceptions, and disk space, each with specific commands and time estimates. If a recent deployment is the cause, kubectl rollout undo triggers after 15 minutes with no resolution.

The service outage runbook assigns severity by impact (SEV1 = complete outage, immediate; SEV2 = >50% affected, 15 minutes) and runs a 5-minute immediate response checklist: acknowledge in PagerDuty, run a health check, open an incident Slack channel, page the Incident Manager for SEV1. Diagnosis routes through infrastructure, network, and application layers with a decision tree to crash loop, capacity, network, or deployment rollback procedures. The database failover runbook covers AWS RDS, PostgreSQL manual promotion, and Patroni clusters — and checks replica lag before every failover to flag data loss risk if lag exceeds 5 minutes.

Defines four severity levels with concrete criteria (SEV1 = complete outage; SEV4 = UI glitch, no user impact) and sets the escalation clock from declaration: L2 at 15 minutes for SEV1/2, Incident Manager at 30 minutes, Engineering Leadership at 45, Executive Team at 60. A separate condition table handles non-time triggers — code changes, database access, customer data, vendor escalations. Includes a 3-tier contact directory with PagerDuty routes and response SLAs, plus a structured escalation message template requiring incident summary, severity, impact, duration, actions taken, and specific ask.

Organized into 12 sections: document metadata, overview, triggers, diagnostic steps, resolution actions, rollback, escalation, communication, references, formatting, testing, and version control. Checks whether alert names and thresholds are quantified, whether commands are copy-pasteable, whether escalation contacts have been verified in the last 30 days, and whether the runbook has been tested by someone unfamiliar with the system. Scoring: 50–55 = ready for production, 40–49 = minor improvements, 30–39 = significant gaps, below 30 = major revision required.

Covers four integration points: trigger sections (document which SLO alerts invoke the runbook), decision points (use error budget remaining to guide next steps), escalation criteria (replace arbitrary time limits with burn rate — 10x demands immediate escalation, 2x allows investigation time), and resolution verification (confirm fixes by checking SLO recovery). Includes the full burn rate action table: burn rate >5x = begin investigation, error budget <50% = notify Incident Manager, error budget <10% = page Engineering Lead.