How to Run a Blameless Postmortem

Why Blame Doesn't Work

When a system fails, the tempting first question is: who made the mistake?

It's the wrong question. Not because people don't make mistakes — they do. But because individual error is almost never the root cause of a systemic failure. The individual who made the error was working within a system: with imperfect tooling, incomplete information, processes that created the conditions for the error, and constraints that made the failure mode difficult to see in advance.

This understanding is at the core of blameless postmortems. The Google SRE book (Beyer et al., 2016) describes the principle: "The goal of postmortems is to learn from the failure and implement systemic improvements, not to assign blame." If the response to an incident is "Alice made a configuration error," the next incident caused by a configuration error happens for the same underlying reasons: the same unclear configuration documentation, the same deployment process that didn't catch the error, the same lack of staging environment that would have revealed it before production.

John Allspaw and Paul Hammond's 2009 presentation at Velocity Conference, "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr," introduced the concept of blameless postmortems to a wide engineering audience. The core insight: engineers who fear punishment for mistakes will hide information, avoid risk, and optimize for not being blamed rather than for preventing failures. A culture where blame is the response to incidents is a culture where incidents accumulate over time rather than decrease.

The alternative — identifying what systemic conditions made the failure possible and changing those conditions — produces fewer incidents over time. That's the goal.

---

What a Blameless Postmortem Produces

A blameless postmortem produces three outputs:

1. A shared understanding of what happened. A timeline of the incident, from first symptom to resolution, that everyone on the team can read and understand. Not a judgment — a narrative.

2. Identified contributing factors (not root causes). The conditions that made the failure possible. These are almost always plural and systemic: insufficient monitoring, an ambiguous configuration option, a gap in the deployment process, a process that worked in 99% of cases but not this one.

3. Closed action items. Specific improvements to systems, processes, or documentation that address the contributing factors. Assigned to named engineers with deadlines.

The postmortem fails if it produces only the first output (a timeline with no action items) or if action items exist but are not assigned and closed.

---

Before the Meeting: Timeline and Impact Collection

The postmortem meeting is not the place to reconstruct the timeline. It's the place to agree on what the timeline means. The timeline reconstruction happens before the meeting.

Assign a "scribe" immediately after the incident: This is the engineer who will collect the timeline, not necessarily the one who resolved the incident. The scribe gathers:

• Monitoring and alerting data (when did each alert fire, when did each metric change)
• Deployment logs and change records (what changed in the hours and days before the incident)
• On-call log (who was paged when, what actions they took and when)
• Slack/communication logs from during the incident
• Any relevant logs from the systems involved

Build the timeline before the meeting: The timeline is a chronological log of what happened:

INCIDENT TIMELINE — [System Name] — [Date]

14:22 UTC  Deployment v2.4.1 completes to production (8/8 pods healthy)
14:35 UTC  First reports of slow responses in #platform-alerts
14:38 UTC  PagerDuty alert fires: "API response time > 2s for 5 min" (pages @on-call)
14:41 UTC  @on-call acknowledges; begins investigation
14:45 UTC  @on-call checks database metrics — connection pool at 480/500
14:47 UTC  @on-call kills idle connections; connection pool drops to 280
14:50 UTC  Response times still elevated; @on-call escalates to @database-team
15:05 UTC  @database-team identifies new query introduced in v2.4.1 not using index
15:12 UTC  Emergency deployment v2.4.2 with query fix
15:16 UTC  Response times return to normal; incident declared resolved
15:30 UTC  Monitoring confirms stable; all-clear posted to #incidents

Collect impact data before the meeting:

• Duration: how long were users affected?
• Scope: what was the impact? (Percentage of requests failing, features unavailable, data at risk)
• Financial or SLA impact if quantifiable

---

The Meeting: Structure and Facilitation

Attendees: The engineers directly involved in the incident (those who were paged, those who escalated, those who resolved it). The team lead or engineering manager. Anyone whose system was a contributing factor. The scribe. Keep it to 6-8 people maximum; larger meetings diffuse accountability.

Facilitator: A named engineer (ideally not the one most directly involved in the incident) who guides the meeting through the structure, keeps the discussion on contributing factors rather than blame, and ensures everyone's observations are heard. This role is important — without a facilitator, postmortems tend to either become blame sessions or become long technical discussions that never produce action items.

Duration: 60-90 minutes for most incidents. A very significant incident (long duration, significant impact) may warrant two meetings: one for timeline validation, one for root cause analysis and action items.

The meeting structure:

1. Timeline walkthrough (15-20 minutes) Walk through the prepared timeline with the full group. The goal: everyone leaves with the same understanding of what happened and when. People who were not directly involved during the incident learn the full sequence; people who were directly involved may fill in gaps or correct the timeline.

Key facilitation prompt: "Did I capture that correctly? Is there anything that happened that's not on the timeline?"

2. Contributing factors (20-30 minutes) The most important part of the meeting. The facilitator asks: "What conditions made this failure possible?" Not "who made the mistake" — "what about our systems, processes, or monitoring made this failure more likely?"

Common contributing factor categories:

• Monitoring gaps: "We didn't have an alert on connection pool utilization above 80%"
• Process gaps: "The deployment checklist didn't require load testing the new query against production data volume"
• Documentation gaps: "The database index requirement wasn't documented; engineers writing queries don't know about it"
• Tool gaps: "The query analyzer doesn't flag missing indexes in the development environment"
• Design gaps: "The query worked with the data volume in staging but not production — there's no staging environment that matches production data scale"

5 Whys technique (from Taiichi Ohno's Toyota Production System): Starting from the immediate cause, ask "why?" five times to reach a systemic contributing factor:

1. Why was the API slow? — Because the database was overwhelmed
2. Why was the database overwhelmed? — Because a query in v2.4.1 wasn't using an index
3. Why wasn't the query using an index? — Because the engineer didn't know the index requirement
4. Why didn't the engineer know? — Because it wasn't documented anywhere the developer would see it
5. Why wasn't it documented? — Because we don't have a database index policy that new query authors are required to consult

The systemic cause is: no database index policy + no process requiring developers to consult it before shipping new queries. Fixing these prevents the next engineer from making the same mistake. Blaming the engineer who shipped v2.4.1 prevents nothing.

3. Action items (15-20 minutes) For each contributing factor identified, what specific action would make the system more resilient? Each action item needs:

• What will be done (specific, not "improve monitoring")
• Who is responsible (named person, not "the team")
• When it will be completed (specific date, not "soon")

Example action items from the worked example above:

• Add connection pool utilization alert at 80% threshold → @preet → by 2026-10-15
• Write and publish database index policy → @nadia → by 2026-10-22
• Add query plan review to the deployment checklist → @james → by 2026-10-15
• Create staging environment with production-scale data for performance testing → @platform-team → by 2026-11-30

4. What went well (5 minutes) Before closing: what did the team do well during the incident? Escalated appropriately? Used monitoring tools effectively? Communicated clearly? Identifying what worked reinforces those behaviors and prevents the postmortem from becoming a purely negative experience.

---

The Postmortem Document

After the meeting, the scribe writes up the full postmortem document. It should be completed and shared within 24-48 hours while the incident is fresh.

# Postmortem: [System] — [Incident Name] — [Date]

**Severity:** P2  
**Duration:** 54 minutes (14:22-15:16 UTC)  
**Impact:** [Percentage] of API requests failed or timed out; 
  [N] users affected; $X estimated revenue impact

---

Summary

[2-3 sentence summary: what failed, for how long, what resolved it]

---

Timeline

[Full chronological timeline from before first symptom to all-clear]

---

Contributing Factors

1. [Factor 1]: [Description of the condition that made this failure possible]
2. ...

---

What Went Well

• [Observation 1]
• [Observation 2]

---

Action Items

Action	Owner	Due Date	Status
Add connection pool alert at 80%	@preet	2026-10-15	Open
Publish database index policy	@nadia	2026-10-22	Open
...

---

Timeline of Action Item Completion

[Updated as action items are closed]

---

Following Up on Action Items

The postmortem document is filed and immediately forgotten — this is the second most common way postmortems fail (after blame). Action items that aren't tracked and closed don't prevent future incidents.

Track action items in your task management system. Create tickets for each action item immediately after the postmortem. Link the ticket to the postmortem document. If your team uses sprint planning, include postmortem action items as real work in the sprint backlog.

Review open action items in weekly team meetings until they're closed. This keeps accountability visible without being punitive: "the action item from the October incident is still open; what's blocking it?"

At the next incident: reference prior postmortem action items. If the same contributing factor appears in two incidents, the action item from the first postmortem wasn't completed or wasn't effective — investigate why.

---

Common Postmortem Failures

Blame by implication: Avoiding the words "blame" and "fault" while still framing the discussion around one person's actions. "We need to make sure engineers test their queries before shipping" is blame-adjacent if the subtext is "because Alex didn't." Redirect to systemic causes.

Vague action items: "Improve observability" is not an action item. "Add alert on connection pool utilization above 80% with PagerDuty integration to the platform team" is. Vague action items have no clear owner and no clear definition of done.

No follow-up: Postmortem documents filed in a wiki and never referenced again. Action items that never get scheduled as actual work. A culture that treats the postmortem document as the deliverable rather than the closed action items.

Only post-morteming bad incidents: Teams that only run postmortems on P1 incidents miss the opportunity to learn from near-misses and smaller failures that have the same contributing factors. A P3 incident that reveals a monitoring gap is worth a brief postmortem; it prevents the P1.

---

Key Takeaways

1. Blame is counterproductive: engineers who fear punishment hide information and avoid risk; a blameless culture produces more incident information and more systemic improvements.
2. Contributing factors are almost always systemic and plural: the question is not "who made the mistake?" but "what conditions made this failure possible?" — and the answer is usually: monitoring gaps, process gaps, documentation gaps, design gaps.
3. The 5 Whys technique surfaces systemic causes: asking "why?" five times starting from the immediate failure reaches the systemic condition that made the failure possible — the level where improvements prevent recurrence.
4. Action items need owners, deadlines, and tracking: postmortem action items that aren't tracked in your task management system and closed within a reasonable timeframe don't prevent future incidents.
5. The postmortem document is not the deliverable — the closed action items are: a postmortem filed in a wiki with open action items that never close is theater; closed action items that change the system are the actual output.

---

Conclusion

A blameless postmortem is a structured learning process that converts an incident into systemic improvements. The timeline establishes shared understanding; the contributing factor analysis identifies what made the failure possible; the action items address those conditions before the next failure. The discipline that makes postmortems valuable over time is follow-through — tracking action items as real work, closing them before the next incident, and checking whether prior action items address the current incident's contributing factors. A team that runs consistent blameless postmortems accumulates a record of systemic improvements; over years, the incident frequency and severity reflect those improvements.

To go deeper, check out Best Web Clipper Extensions.