Developer Knowledge
How to keep a debugging journal — a practical guide for developers who want to stop re-discovering the same bugs, reduce debugging time by building a searchable record of past investigations, and turn frustrating debugging sessions into durable knowledge.
Debugging is expensive. A complex bug can consume four to eight hours — sometimes more — of a senior engineer's time. The investigation generates knowledge: hypotheses tested, environmental conditions eliminated, root causes identified, fixes validated. That knowledge is genuinely valuable.
It almost never gets recorded.
The usual pattern: you find the bug, you fix it, you close the ticket, you move on. The knowledge lives in your memory. Six months later, you — or a colleague — hit the same bug or a similar one. The investigation starts again from zero, because there's no record of the last one.
This is not a failure of motivation. It's a structural problem: there's no moment in the workflow when "record what I learned debugging this" feels like the right next step. The natural next step after fixing a bug is closing the ticket or starting the next task.
A debugging journal inserts that recording step. It turns debugging sessions from single-use investigations into reusable knowledge. Over time, a debugging journal becomes one of the most valuable pieces of documentation a developer maintains — not because of any single entry, but because of the searchable, accumulated pattern of "what I've debugged and what I found."
A debugging journal is not a log of what you did hour by hour. It's not a replacement for git commit messages, incident reports, or ticket comments. It doesn't need to be beautiful or comprehensive.
It's a structured record of the most important things you learned during a debugging session, organized so you can retrieve them later when you encounter a similar problem.
The reader of a debugging journal entry is primarily you, six or twelve months from now, facing something that looks similar to what you debugged today. The entry should give that reader enough context to recognize whether the current situation matches the past investigation and enough information to apply what was learned.
A debugging journal entry has six sections:
Symptom: What did the system do wrong? Specific — not "it was slow" but "requests to the /search endpoint were timing out after 30 seconds under load, starting at 14:23 UTC on 2026-10-12. No errors in logs; just timeouts."
Environment: System version, deployment environment, relevant configuration, date/time, load or traffic pattern at the time. Bugs are often environment-specific; without this context, an entry that matches the symptom may not match the situation.
Hypotheses tested: What did you think might be causing it, and what did you do to test each hypothesis? List them with their outcomes — especially the ones that turned out to be wrong. A hypothesis that was ruled out is valuable information: it saves future investigators the same wrong turn.
Root cause: What was actually causing the problem? Be specific: not "a memory leak" but "the worker process's event loop was being blocked by a synchronous database query in a library that's documented as async-only; the blocking was under a feature flag that only activated under the specific load pattern."
Fix: What resolved it? Include the commit reference, the configuration change, or the workaround.
Prevention: What would prevent this class of bug from happening again? A missing test, a monitoring gap, a documentation gap, a configuration default that should be changed. This is the most valuable section for the team and for the codebase — but it's also the section most often left blank.
DEBUGGING JOURNAL ENTRY
Date: 2026-10-12
System: Search service (v2.4.1)
Environment: Production; AWS us-east-1; PostgreSQL 14.3 on RDS db.r6g.large
SYMPTOM:
/search endpoint timeouts (30s) starting ~14:23 UTC. Load was ~2.5x normal
(promotional campaign started 14:00). No 5xx errors in logs; requests just
hung and were dropped by the load balancer. Health checks passing.
HYPOTHESES TESTED:
1. Database connection pool exhaustion → Checked RDS metrics: connection count
at 48/100, no pool errors. RULED OUT.
2. Memory pressure on search workers → Worker memory at 65%; no OOM events.
RULED OUT.
3. Elasticsearch query volume → Search queries completing in 50-80ms during
incident. RULED OUT.
4. Event loop blocking in Node.js worker → Added event loop lag monitoring;
lag spiked to 8-12 seconds during high load. CONFIRMED.
ROOT CAUSE:
redis-client v2.3 used for caching search results calls `client.get()`
synchronously when the cache is cold (cache miss). Under normal load, cache
hits >95%; at 2.5x load, the cache warm-up period on the promotional items
produced 40% cold misses. Each cold miss blocked the event loop for ~80ms;
at 40% miss rate × 2.5x load, cumulative blocking exceeded the request
timeout window.
FIX:
Upgraded redis-client to v3.1 which is fully async; cache misses no longer
block the event loop. Deployed at 16:45 UTC; timeouts resolved within 3 minutes.
Commit: 7f3a9b2
PREVENTION:
1. Add event loop lag alert (threshold: 500ms) to monitoring — this would
have flagged the issue immediately instead of requiring manual investigation.
2. Add a synthetic test that runs the search endpoint under cache-cold conditions
(flush cache, run load test) — existing load tests run with warm cache.
3. PR checklist item: verify any new Redis/cache client library is async-safe.
The rule of thumb: write an entry for any bug that took more than 30 minutes to debug.
Definitely write an entry:
Skip the entry for:
The 30-minute threshold is approximate. The real test: would a future engineer benefit from knowing what hypotheses you tested and why they were wrong? If yes, the entry is worth writing.
The best time to write a debugging journal entry is immediately after resolving the bug — while the investigation is fresh, before the details fade. An entry written from memory three days later will be missing the failed hypotheses (the most valuable parts) and the specific environmental details that made the bug hard to find.
The practical workflow:
The scratch notes don't need to be clean — they're working notes, not the entry. The entry is the structured synthesis you write after the investigation concludes.
A debugging journal organized as a flat chronological log becomes hard to search as it grows. Organize for retrieval:
By system or service: A section for each service or subsystem you maintain. All search service bugs are in one place; all authentication bugs in another.
By symptom pattern: Alternatively, organize by symptom category: timeouts, memory issues, data corruption, race conditions, configuration errors, third-party integration failures. This makes pattern recognition faster — seeing all memory-related bugs together reveals patterns.
Both: Organize by service, tag entries with symptom categories. A search in "search service" + filter for "timeout" finds every timeout investigation in the search service.
Search is primary: Whatever the organizational structure, full-text search is the main retrieval mechanism. Write entries using the specific error messages, library names, and configuration keys that you'd search for when you encounter the same symptom again. "ERROR: ETIMEOUT in redis-client" as part of the symptom description is searchable; "the redis cache timed out" is not.
A personal debugging journal becomes more valuable when the most generalizable entries are promoted to a team-accessible knowledge base.
The principle: entries that reveal a bug class that could affect other team members (not just one service, not just one engineer's setup) belong in a shared knowledge base. Entries that are specific to one service or one engineer's configuration can stay personal.
What belongs in the team knowledge base:
The team debugging knowledge base doesn't need elaborate tooling. A dedicated section of the team wiki, a Confluence space, or a GitHub repository with markdown files all work. The key properties: searchable, linkable from tickets, and writable by any team member without friction.
Entry from a backend engineer:
Date: 2026-09-03
System: Notification service (Node.js 20, Express 4.18)
Environment: Staging; Docker on k8s; 2Gi memory limit
SYMPTOM:
Notification service pods restarting every 6-8 hours with OOMKilled.
Heap memory growing steadily at ~15MB/hour from initial 180MB after restart.
No memory growth under synthetic load tests.
HYPOTHESES TESTED:
1. Request handler not releasing references → Added heap snapshots at 1h intervals;
WebSocket connection objects accumulating. CONFIRMED direction.
2. WebSocket connection objects growing → Found 8,000+ WebSocket objects in
6-hour heap snapshot; should be ~200 (our MAU on staging). CONFIRMED.
3. Connection close event not firing → Logged 'close' event; fires correctly
on client disconnect. MISLEADING - the connection was being closed but
the object wasn't being cleaned up.
4. EventEmitter listener leak → Added listener count logging; emitter growing
by 1 listener per connection without removal. ROOT CAUSE.
ROOT CAUSE:
`this.wss.on('connection', ...)` handler adds an EventEmitter listener on each
connection. The cleanup code in the 'close' handler called `removeListener()`
with a different function reference than the one added — so listeners were
added but never removed. The heap held the reference; GC couldn't collect.
FIX:
Store the listener function reference in the connection object at creation; use
that reference for both add and remove. Verified in staging: heap growth
eliminated over 24h.
Commit: b8e7123
PREVENTION:
1. Add `this.wss.setMaxListeners(250)` and alert on the listener count
approaching the limit — this would have been caught much earlier.
2. Add a 12-hour staging soak test to the CI pipeline — the bug is invisible
in 1-hour load tests.
3. Document EventEmitter listener cleanup pattern in our Node.js conventions guide.
A debugging journal transforms each expensive debugging session from a single-use investigation into a reusable asset. The format — symptom with specific details, environment, hypotheses tested (including the wrong ones), root cause, fix, and prevention — produces entries that are retrievable, usable, and genuinely informative to a future reader facing a similar problem. The discipline is writing the entry immediately after resolution and organizing it for search rather than chronology. Over time, the journal becomes the fastest way to answer "have I seen this before?" — and often, "yes" is the answer.
More WebSnips articles that pair well with this topic.
How to build a knowledge base for a dev team — a practical guide for engineering teams who want a shared knowledge system that engineers actually use, that stays current as the team grows, and that reduces the time engineers spend re-answering the same questions.
How to document a microservices architecture — a practical guide for engineering teams navigating service sprawl, where the challenge is not documenting individual services but making the relationships, contracts, and operational behavior of a distributed system legible.
How to keep a changelog developers trust — a practical guide for engineering teams who want a CHANGELOG.md that consumers of their API or library actually read and rely on, rather than a dump of commit messages that obscures more than it reveals.
How to save and organize design docs — a practical guide for engineers and engineering teams who want their design documents to remain findable, useful, and connected to the decisions they documented, rather than accumulating in an untended archive.
How to take notes during code review — a practical guide for engineers who want to get more from code review than the immediate feedback loop: building a personal reference of patterns, anti-patterns, and architectural decisions accumulated across months of reviews.
How to track tech-debt decisions — a practical guide for engineering teams who want to manage their technical debt as intentional trade-offs rather than accumulated accidents, with a tracking system that makes debt visible, prioritizable, and repayable.