Developer Knowledge
How to write runbooks that people read — a practical guide for engineers and SREs who want operational runbooks that actually get used during incidents, not documentation that gets discovered after the incident is resolved and is outdated by the time it's needed.
An on-call engineer receives an alert at 2:47 AM. A database connection pool is exhausted. The system is degraded. The engineer needs to act in under three minutes before the degradation becomes an outage.
There's a runbook for this. It's 1,400 words long. It covers the history of the connection pool design, explains how PostgreSQL connection management works, and has a section on long-term remediation strategies. The immediate action the engineer needs — "increase the connection pool size in the environment variable and restart the service" — appears on page three after the background section.
The engineer doesn't read the runbook. They search Slack for previous discussions about connection pool issues, find a thread from eight months ago, and piece together the steps from memory and context. The incident resolves in 22 minutes instead of 6.
This is the core failure mode of most runbooks: they're written by engineers deep in a system with time to be thorough, and read by engineers who are already in an incident with no time for anything but the action they need to take right now.
Good runbooks invert this. They put the action first. They assume the reader is under pressure. They're short enough to be read in under a minute. And they're accurate enough to be trusted.
Runbook: A procedure for a specific, anticipated operational event — usually a specific alert or system state. "Connection pool exhausted runbook." "Service health check failing runbook." One alert or event, one runbook.
Playbook: A higher-level procedure for a class of incidents or situations — "database incident playbook," "security breach playbook." Playbooks reference runbooks as steps; they don't replace them.
This guide focuses on runbooks — the specific, action-oriented documents that correspond to specific alerts and operational events.
1. Action first. The first thing in a runbook is the immediate action required — not the background, not the history, not the explanation of why this alert exists. If the immediate action is "restart the service," that's the first line.
2. Short enough to read under pressure. A runbook that takes more than five minutes to read will not be read during an incident. Target 200-400 words for most runbooks. Background and explanation belong in a separate wiki article that the runbook links to; not in the runbook itself.
3. Structured as a decision tree, not prose. Incidents are not linear. "If step 3 reveals X, go to step 5a. If step 3 reveals Y, go to step 5b." A numbered list with conditional branches matches how incidents actually unfold.
4. Linked from the alert. A runbook no one can find during an incident is useless. Every alert annotation should include a link to the relevant runbook. Finding the runbook should take five seconds, not five minutes.
5. Maintained as part of incident close. A runbook updated only at creation time becomes outdated within months. The process discipline: closing an incident includes verifying that the runbook accurately reflected what worked and updating it if not.
# [System Name] — [Alert Name]
**Severity:** [P1 / P2 / P3]
**On-call contact:** [Team Slack channel]
**Escalation:** [Secondary contact if unresolved in 30 min]
---
## Immediate action
[1-3 lines: what to do right now to stop the bleeding /
restore service / prevent escalation.
This is step zero before any investigation.]
Example:
"1. In the k8s dashboard, restart the affected pods: `kubectl rollout restart deployment/payments-service`
2. Watch pod status: `kubectl get pods -l app=payments-service -w`
3. Verify health check passes before continuing investigation."
---
## Symptoms
What this alert looks like in practice:
- [Specific observable behavior 1]
- [Specific observable behavior 2]
- [Related alerts that may fire simultaneously]
---
## Diagnosis
Questions to answer to identify the root cause:
**Check 1: [What to check]**
Command / dashboard: `[exact command or dashboard link]`
Healthy: `[what healthy looks like]`
Unhealthy: `[what unhealthy looks like]` → Go to Step [N]
**Check 2: [Next check]**
[Same format]
---
## Resolution
**If [diagnosis result A]:**
[Specific remediation steps, with exact commands]
Expected outcome: [what you should see when this works]
**If [diagnosis result B]:**
[Specific remediation steps for this case]
Expected outcome: [what you should see when this works]
---
## Escalation
If service is not restored within 30 minutes, or if the root cause
cannot be identified from the above:
- Escalate to: [Name / team / PagerDuty escalation policy]
- Context to provide: [What information the escalated contact will need]
---
## Post-resolution
- [ ] Verify all alerts have cleared
- [ ] Check for cascading failures in dependent services
- [ ] File incident report within 24 hours
- [ ] Update this runbook if the steps above did not reflect what actually worked
---
**Last updated:** [Date] by [Author]
**Related:** [Link to system architecture | Link to relevant ADR | Link to postmortem from last occurrence]
Background and history: Why the system works this way, the architectural decisions behind the alert threshold, the history of this specific failure mode. This belongs in a linked wiki article, not in the runbook.
Long explanations of how the system works: An on-call engineer who doesn't know how connection pooling works should not be learning it from a runbook during an incident. Link to the explanation; don't embed it.
Comprehensive coverage of every possible scenario: A runbook that tries to cover every possible state produces a document that's too long to read during an incident. Cover the common cases well; escalate the uncommon ones.
FAQs and best practices: Operational best practices belong in engineering standards documents. A runbook is a specific procedure for a specific operational event.
A runbook linked from the alert annotation is found in five seconds. A runbook in a wiki that requires navigation to find is often not found until the incident is over.
Most alerting platforms support annotation on alerts:
PagerDuty: Add a runbook URL in the service configuration or in the alert annotation field.
Grafana: Add alert annotations with runbookUrl field in the alert rule definition.
Prometheus/Alertmanager: Add runbook_url as an alert label or annotation.
OpsGenie: Add alert details with a link to the runbook URL.
The implementation takes five minutes per alert and is the most high-leverage thing you can do to improve runbook utilization during incidents.
Runbooks become inaccurate because they're updated at creation time and rarely afterward. The system changes; the runbook doesn't. An engineer uses the runbook during an incident, discovers a step is wrong, and either fixes the runbook (rare) or doesn't (common).
The discipline that works: Closing an incident includes a runbook review step. The incident close checklist:
Post-incident checklist:
[ ] Incident report filed
[ ] Runbook reviewed: were the steps accurate?
[ ] If not: runbook updated with correct steps
[ ] Alert threshold reviewed: was the alert tuned correctly?
This takes 5 minutes and is the only maintenance practice that actually keeps runbooks current, because it's attached to the moment when the gap between the runbook and reality is most visible.
# PostgreSQL — Connection Pool Exhausted
**Severity:** P2
**On-call contact:** #platform-engineering
**Escalation:** @platform-lead (30 min SLA)
---
## Immediate action
1. Scale down non-essential traffic: disable batch jobs and background workers
that hit this database:
`kubectl scale deployment/background-worker --replicas=0 -n production`
2. Check current connection count:
`psql $DATABASE_URL -c "SELECT count(*) FROM pg_stat_activity;"`
3. If count > 450 (pool limit is 500): kill idle connections:
`psql $DATABASE_URL -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND state_change < NOW() - INTERVAL '10 minutes';"`
4. Monitor connection count: should drop below 300 within 2 minutes.
---
## Symptoms
- Alert fires when pg_stat_activity > 480 for 2 consecutive minutes
- Typical symptoms: requests queue, then timeout; 503 errors from all services
using this database
- May fire alongside: "Service response time > 2s" and "Error rate elevated" alerts
---
## Diagnosis
**Check 1: Which services are holding connections?**
`psql $DATABASE_URL -c "SELECT application_name, count(*), state FROM pg_stat_activity GROUP BY application_name, state ORDER BY count DESC;"`
Healthy: top 3 services total < 400 connections
Unhealthy: one service > 200 connections → investigate that service (go to Check 2)
**Check 2: Is a runaway query holding connections?**
`psql $DATABASE_URL -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes' ORDER BY duration DESC;"`
If long-running queries: terminate with `pg_terminate_backend(pid)` for PIDs > 10 min old.
**Check 3: Is a deployment causing the spike?**
Check recent deployments in Argo CD: did a new version deploy in the last 30 minutes?
If yes: roll back the deployment.
---
## Resolution
**If idle connections accumulated (most common):**
Steps above (kill idle connections + scale down batch workers) should resolve.
Scale workers back up 15 minutes after connections stabilize.
**If a runaway query:** Terminate long-running queries (Check 2). Find the query
source; escalate to the owning team if unknown.
**If a deployment caused a connection leak:** Roll back. File bug with owning team.
---
## Escalation
If connections don't drop below 350 within 10 minutes of initial action:
- Escalate to @database-team
- Provide: current connection count by application, any long-running queries,
recent deployment log
---
## Post-resolution
- [ ] Restore batch worker replicas to normal
- [ ] Verify connection count stable for 15 minutes
- [ ] Identify root cause for incident report
- [ ] Update this runbook if steps were inaccurate
**Last updated:** 2026-10-01 by @preet
**Related:** [PostgreSQL connection pool design doc | ADR-021 pool sizing]
A runbook that gets read during incidents is short, action-first, linked directly from the alert, and accurate because it's maintained as part of the incident close process. The content investment is 30-60 minutes per runbook; the return is measured in minutes saved during every subsequent incident. The maintenance investment is 5 minutes per incident that uses the runbook; the return is a runbook that remains accurate over months and years of system evolution. The alternative — runbooks that nobody reads, so every incident is reinvented from scratch — is measurable in longer mean time to resolution and avoidable escalations.
More WebSnips articles that pair well with this topic.
How to build a knowledge base for a dev team — a practical guide for engineering teams who want a shared knowledge system that engineers actually use, that stays current as the team grows, and that reduces the time engineers spend re-answering the same questions.
How to document a microservices architecture — a practical guide for engineering teams navigating service sprawl, where the challenge is not documenting individual services but making the relationships, contracts, and operational behavior of a distributed system legible.
How to keep a changelog developers trust — a practical guide for engineering teams who want a CHANGELOG.md that consumers of their API or library actually read and rely on, rather than a dump of commit messages that obscures more than it reveals.
How to save and organize design docs — a practical guide for engineers and engineering teams who want their design documents to remain findable, useful, and connected to the decisions they documented, rather than accumulating in an untended archive.
How to take notes during code review — a practical guide for engineers who want to get more from code review than the immediate feedback loop: building a personal reference of patterns, anti-patterns, and architectural decisions accumulated across months of reviews.
How to track tech-debt decisions — a practical guide for engineering teams who want to manage their technical debt as intentional trade-offs rather than accumulated accidents, with a tracking system that makes debt visible, prioritizable, and repayable.