A postmortem (also called a post-incident review or PIR) is a structured analysis conducted after a significant incident, failure, or outage — documenting what happened in chronological order, identifying the root causes, analyzing contributing factors, and defining concrete action items to prevent recurrence. The blameless postmortem approach, popularized by Google's SRE practices, focuses on system and process failures rather than individual mistakes, creating an environment where engineers can share information honestly without fear of punishment.
A postmortem turns a failure into an organizational learning event.
Where the Postmortem Concept Comes From
The medical term "postmortem" (Latin: "after death") originally referred to an examination of a body after death to determine cause. In organizational and engineering contexts, the term was borrowed to describe the examination of a "dead" project, incident, or failure to understand what went wrong.
The concept of systematic post-failure analysis has existed in safety-critical industries for decades:
- Aviation accident investigation (the NTSB investigates plane crashes, not to punish pilots, but to find systemic failures)
- Nuclear power plant incident review (after Three Mile Island 1979 and Chernobyl 1986)
- Medical M&M (Morbidity and Mortality) conferences — structured peer review of patient outcomes
The tech industry adapted these practices explicitly. Google's SRE book (Beyer, Jones, Petoff, Murphy, 2016) articulated the "blameless postmortem" as a core SRE practice, emphasizing that:
- Complex systems fail in complex ways — individual human error is rarely the root cause
- Punishing individuals for errors in complex systems drives problems underground
- The goal is to understand failure modes well enough to prevent recurrence, not to assign blame
The blameless postmortem is now standard in mature engineering organizations, though the degree of actual blamelessness varies significantly in practice.
The Structure of a Postmortem
Standard postmortems include:
Incident summary:
A 2-3 sentence overview: what happened, when it happened, how long it lasted, and what was affected. Written for an audience not present during the incident.
Impact:
Quantified effect — how many users affected, duration, financial impact, SLA violation. Concrete numbers, not vague descriptions.
Timeline:
Chronological sequence of events — when the incident started (often before it was detected), when it was detected, key moments in investigation, when actions were taken, when service was restored. Specific timestamps.
Root cause analysis:
The causal chain that produced the incident. Not "the engineer deployed bad code" (that's a proximate cause) but the conditions that made the bad deployment possible — lack of testing in staging, insufficient review process, missing monitoring. The "5 Whys" technique (ask "why" five times to get from symptom to root cause) is commonly used.
Contributing factors:
Other factors that made the incident worse or harder to recover from — even if not the root cause. Alert noise that delayed detection. Documentation gaps that slowed investigation. On-call handoff that created knowledge gaps.
Detection:
How was the incident discovered? User reports? Monitoring alerts? How long between incident start and detection? Gaps in detection are often action items.
Resolution:
What steps resolved the incident? What was tried first that didn't work?
Action items:
Concrete, assigned, time-bound tasks to prevent recurrence. "Improve monitoring" is not an action item. "Add p99 latency alert on the payment API with a threshold of 800ms, owner: Sarah, by 2024-11-15" is.
A Worked Example
Incident: Production database outage, 2024-10-08, 14:23-16:47 UTC (2 hours 24 minutes)
Summary:
A routine index migration on the users table acquired a table lock that blocked all production reads and writes, causing a 2.5-hour outage affecting all paying customers. The migration had been reviewed and approved but was not tested against a production-sized dataset.
Impact:
- ~8,200 customers unable to access the product during the outage
- $47,000 in estimated revenue impact (from churned trials + SLA credits)
- 3 enterprise customers filed escalations; 1 threatened churn
Timeline:
- 14:20: Engineer runs
ALTER TABLE users ADD INDEX in production console
- 14:23: Database CPU spikes to 100%; API error rate reaches 95%
- 14:31: On-call engineer wakes to PagerDuty alert (8-minute detection gap)
- 14:35: Migration identified as cause; attempts to kill migration begin
- 14:52: First kill attempt fails (migration holds lock)
- 15:20: DBA escalated
- 16:44: Migration completes naturally; service begins recovery
- 16:47: Service restored
Root cause:
The ADD INDEX statement on a 40M-row table acquired an exclusive table lock for the migration duration. On a production-sized table, this took 2.5 hours. The migration was tested in staging against a dataset of 10,000 rows, where it completed in <5 seconds.
5 Whys:
Why did the outage occur? → Migration held a table lock.
Why was this allowed? → Migration was not tested at production scale.
Why was testing insufficient? → Staging uses a subset of production data (10K vs 40M rows).
Why does staging use a smaller dataset? → No process requires production-scale staging testing.
Why does no such process exist? → We've never had a migration failure at this scale before (first occurrence).
Contributing factors:
- No runbook for "migration running long" scenario
- On-call alert threshold allowed 8 minutes before paging (too high for database errors)
- Migration could not be safely killed once started
Action items:
| Action | Owner | Due |
|---|
| Document migration checklist requiring production-scale staging validation for tables >1M rows | Sarah | 2024-11-01 |
| Reduce database error rate alert threshold from 5% to 1% | Mike | 2024-10-15 |
Research and document pt-online-schema-change as lock-free migration alternative | Sarah | 2024-11-01 |
| Create "migration running long" runbook | DevOps | 2024-10-22 |
| Scale staging dataset to 10% of production size (from 10K rows) | Alex | 2024-12-01 |
This postmortem produces 5 concrete action items that address the root cause and contributing factors — preventing the same class of failure in the future.
Blameless vs. Blame-Focused Postmortems
| Aspect | Blameless postmortem | Blame-focused postmortem |
|---|
| Focus | System and process failures | Individual mistakes |
| Outcome | Engineers share honestly; complete timeline emerges | Engineers protect themselves; timeline is incomplete |
| Action items | Fix the process | Discipline the individual |
| Learning | Organizational: what can be improved | None: "don't make mistakes" is not actionable |
| Long-term effect | Reduces incident frequency | Drives errors underground; increases incident frequency |
The blameless approach doesn't mean individuals are never accountable — it means postmortems aren't the venue for accountability. A blameless postmortem assumes that engineers are not malicious and that mistakes in complex systems are usually products of the system, not individual failure. This assumption produces more complete information and better outcomes.
Common Misconceptions About Postmortems
"Postmortems are only for major outages."
Postmortems are most valuable for significant incidents, but "significant" is a lower threshold than teams assume. A 30-minute degradation affecting 10% of users, a near-miss that required quick action to avert an outage, a deployment that required an immediate rollback — all of these produce learnings worth capturing. The cost of a postmortem is low; the cost of repeated incidents from the same root cause is high.
"The goal of a postmortem is to prevent the specific incident from recurring."
The goal is to identify the class of failure and the systemic conditions that produced it — and address those. Addressing only the proximate cause of a specific incident ("we now have a safeguard against this specific migration pattern") misses the systemic vulnerability ("our staging environment doesn't test at production scale, so any production-scale risk is invisible in staging").
"Blameless postmortems mean there are no consequences for poor performance."
Blameless postmortems separate the learning-and-improvement function from the performance-management function. The postmortem is not the venue for performance management; it's the venue for system improvement. These are separate conversations with different goals. Conflating them undermines both.
Related Concepts
Runbooks: The operational procedures that postmortems often surface as missing — "we didn't have a runbook for this scenario" is a common finding.
Institutional knowledge: Postmortems capture organizational learning from failure — one of the most valuable categories of institutional knowledge.
Architecture decision records (ADRs): ADRs capture design decisions; postmortems capture what happened when those decisions met reality under failure conditions.
Mean time to recovery (MTTR): The metric postmortems directly improve — the faster and more complete the recovery playbook, the lower the MTTR.
Frequently Asked Questions
How soon after an incident should you do a postmortem?
Within 48-72 hours is the general guideline — soon enough that memories are fresh, but not so soon that the immediate recovery chaos is still active. Google's SRE practices recommend scheduling the postmortem review within 5 business days. For significant incidents, starting the document (timeline, impact) within 24 hours is valuable even if the full review happens later.
Who should attend a postmortem?
The engineers who were directly involved in the incident (who know what happened), plus their managers (who don't speak during the review but understand the organizational impact), plus anyone whose systems contributed or were affected. Postmortem reviews shouldn't be a large committee — 4-8 people is typical. The document should be shared more broadly after.
How do you ensure action items from postmortems actually get completed?
Postmortem action items have a well-known tendency to be filed and forgotten. High-completion approaches: assign each item to a specific person (not a team); give each item a specific due date; track action items in the same project management tool as other work; include a postmortem review section that checks completion of prior postmortem items.
Key Takeaways
- A postmortem is a structured analysis after an incident — documenting what happened, why, and what actions prevent recurrence.
- Blameless postmortems (popularized by Google SRE) focus on system and process failures rather than individual mistakes — producing more complete information.
- Standard sections: summary, impact, timeline, root cause, contributing factors, detection, resolution, action items.
- Action items must be concrete: specific owner, specific action, specific due date — not general improvement goals.
- Root cause vs. proximate cause: effective postmortems find the systemic conditions that enabled the failure, not just the immediate trigger.
- Not just for major outages: any incident worth preventing recurrence of is worth a postmortem.
Conclusion
Postmortems are the mechanism by which engineering organizations convert failures into learning. Done well — honestly, promptly, focused on systems rather than blame, and with concrete action items that actually get completed — they reduce incident frequency over time, improve team confidence in handling future incidents, and build the organizational knowledge of how systems actually fail under real conditions. Done poorly — focused on blame, filed and forgotten, or conducted only for the most visible failures — they provide the appearance of learning without the substance. The investment in a strong postmortem culture pays back in fewer repeat incidents and in teams that can discuss failure honestly.
Try WebSnips free — organize research into past incidents, relevant vendor outage reports, and best-practice sources into collections that inform your postmortem process and action items, making the organizational learning from failures searchable and accessible.