Developer Productivity
Write effective runbooks that guide engineers through incidents and operations.
Your system breaks at 2am.
On-call engineer gets paged.
Wakes up.
Confused.
Logs in.
"What do I do?"
They scramble.
Without documentation:
With a runbook:
A runbook is the ops documentation that saves you at 2am.
This guide covers writing runbooks that work.
A runbook is a procedure document.
It tells someone exactly what to do in a specific situation.
Not:
Yes:
When used:
When stress/panic hits, people forget things.
They can't think clearly.
Runbook removes need to think.
Just follow steps.
With runbook: 15 minutes
Without runbook: 2+ hours
Savings: 105 minutes per incident
Under pressure, engineers make mistakes.
Runbook provides guardrails.
"Did you check X?"
Yes → next step.
Mistakes prevented.
Without runbook: Only expert can handle incident.
With runbook: Anyone can follow it.
Team isn't bottlenecked on one expert.
On-call engineer: "I know what to do."
They follow steps.
They recover the system.
Confidence + competence.
Use this template for every critical procedure:
# [Procedure Title]
[What event triggers this? What symptoms?]
[What access/permissions needed?] [What tools should be available?]
[Critical / High / Medium / Low]
[How long does this typically take?]
[30-second summary of what you'll do]
Command: [Exact command] Expected result: [What should you see if working?] If not: Go to Troubleshooting
Command: [Exact command] Expected result: [What should you see?]
Command: [Exact command] Expected result: [What you should see]
Command: [Exact command] Expected result: [What you should see]
After fix, verify:
Solution: [What to do]
Solution: [What to do]
If fix doesn't work: Step 1: [Rollback action] Step 2: [Verify rollback]
If this doesn't work:
[Date] by [Name]
[Next quarterly review]
---
# API Service CPU Exhaustion Recovery
Critical (customer-facing feature down)
15–30 minutes
ssh prod-api-1
top
# Look for: CPU usage > 90%
Expected: One process using lots of CPU
ps aux | grep -E "node|python|java"
# Note PID of high-CPU process
Expected: See which process is the culprit
tail -f /var/log/api.log | grep "SLOW QUERY"
Expected: See slow queries if issue is database
Is it:
# Get query ID
QUERY_ID=$(ps aux | grep "mysql" | grep -o "query.\*" | head -1)
# Kill query
mysql -u admin -p$MYSQL_PASS -e "KILL $QUERY_ID;"
# Restart service
kubectl rollout restart deployment/api
# Verify
kubectl get pods -l app=api
# Wait for new pods to be Running
# Activate rate limiter (already deployed)
kubectl set env deployment/api RATE_LIMIT_ENABLED=true
# Restart with new config
kubectl rollout restart deployment/api
After fix, verify:
top shows CPU < 80%curl https://api.prod/health returns 200Solution: Don't restart again. Page senior engineer immediately.
Solution: Try backup server: ssh prod-api-2
Solution: SSH to server directly and use docker commands
If fix makes things worse:
kubectl rollout undo deployment/apicurl https://api.prod/healthIf this doesn't work:
2025-02-10 by Alice Chen
2025-05-10 (quarterly)
---
WRONG: "Query the database and find slow queries"
RIGHT: "Run: mysql -u admin -p$MYSQL_PASS -e 'SHOW FULL PROCESSLIST;'"
On-call engineer might not know exact MySQL syntax.
Be specific. Provide exact commands.
WRONG: "Fix the database issue"
RIGHT: "Prerequisites: SSH access to prod-db-1, MySQL CLI installed, credentials in ~/.my.cnf"
Runbook assumes reader can't access something.
Verify prerequisites first.
WRONG: "Restart the service"
RIGHT: "Is it a database query or service memory leak?
- Database: Kill the query
- Memory: Restart service
- Network: Check firewall"
Different symptoms need different fixes.
Provide decision tree.
WRONG: [Procedure ends without checking if it worked]
RIGHT: [After each step, verify result. "Expected to see: X"]
How does on-call engineer know if they fixed it?
Include verification steps.
Runbook written 6 months ago.
System changed.
Commands don't work.
Fix: Set quarterly review date. Update before then.
What could break?
Next time incident happens:
Don't use runbook.
Go through procedure.
Solve the problem.
Document every step.
Use what you just did.
Exact commands.
Exact output.
Decision points.
Have someone else follow it.
Do they succeed?
Any confusing parts?
Update.
Next incident, have on-call follow runbook.
Does it work?
What's missing?
Update immediately.
Pros: Searchable, linked, organized
Cons: Need to remember to check wiki during incident
Pros: In same place as code
Cons: Not easily found if you're in production server
Pros: Accessible when no network
Cons: Gets outdated
/runbook [name]Before runbooks: 2+ hours
After runbooks: 15–30 min
Savings: 90+ minutes per incident
Track: % of incidents where runbook was used
Track: % of runbooks that successfully resolve issue
A runbook is the procedure that saves you at 2am.
Template elements:
How to start:
In 6 months, every critical procedure will have a runbook.
On-call incidents will be faster and less stressful.
For team wiki, see Team Wiki Setup Guide. For onboarding, check Onboarding Documentation System.
More WebSnips articles that pair well with this topic.
Implement Architecture Decision Records to document why you made important technical decisions. ADR template, examples, and workflow for software teams.
Build async documentation practices for remote teams. How to write documentation that answers questions before they're asked and reduces meeting load.
Build a personal documentation system for developers. Capture solutions, architecture decisions, and technical context so you never solve the same problem
Build an engineering documentation culture developers will actually use. Covers incentives, templates, tooling, and practices for living documentation.
Write meeting notes that drive action. Template, best practices, and workflow for capturing decisions, action items, and context from every meeting.
Build an onboarding documentation system that gets new hires productive faster. Templates, structure, and maintenance workflows for effective onboarding docs.