Debugging production issues with evidence

From alert to a reviewable root-cause hypothesis

Cost Management tooling demo · 2026-09-05

An alert tells us that something failed — not why

An API timeout, a stalled worker, or queue lag can be caused by many layers:

  • application code or a retry loop
  • an individual pod or a bad deployment
  • a shared dependency such as PostgreSQL
  • an expensive query shape or tenant-specific data volume
The goal is not to collect more logs. It is to follow one correlation anchor — an error, timestamp, pod, task, or tenant — until the evidence rules hypotheses in or out.

The investigation moves from symptom to database evidence

center

Each layer answers a different question. None, by itself, is the root cause.

Start with GlitchTip: establish the incident's shape

What it gives us

  • exact exception and affected service
  • event timestamp and host/pod tag
  • first/last seen and recurrence count
  • a candidate provider, task, or request path

Example: a Gunicorn WORKER TIMEOUT is an anchor for a narrow log and RDS search window — it is not proof that PostgreSQL timed out.

# Retain the event detail locally
scripts/glitchtip-snapshot \
  --events <issue-id> \
  reports/glitchtip-events.json

# Then investigate with the reusable workflow
$investigate-alert <issue-url>

Question answered: Is this new, recurring, isolated, or broad?

Kubernetes tells us what is happening right now

Check before forming a database theory

  • pod readiness, restarts, OOMs, rollout state
  • worker count and HPA state
  • current log tail for the affected pod
  • whether work advances or repeatedly stops at the same point
kubectl --context <explicit-prod-context> \
  -n hccm-prod get pods

kubectl --context <explicit-prod-context> \
  -n hccm-prod logs <pod> --since=15m

Evidence: process health and the application’s last known action.

Kibana reconstructs activity across pods and restarts

Pod logs are immediate but narrow. Kibana lets us search a bounded time range
across replicas for the alert's tenant, provider, task, or message.

scripts/kibana-search --environment prod \
  --query '"WORKER TIMEOUT"' \
  --from 2026-09-04T12:37:00Z \
  --to   2026-09-04T12:41:00Z --size 20 \
| jq -r '.responses[].hits.hits[]._source
  | "\(.["@timestamp"])\t\(.["@log_stream"])\t\(.["@message"])"' \
| rg --color=always 'WORKER TIMEOUT'
The critical move is correlation: the pod and error timestamp identify what to search, while Kibana reveals the surrounding request or task sequence.

RDS CloudWatch logs test the PostgreSQL hypothesis

scripts/cost-prod-rds-postgresql-logs \
  --around 2026-09-04T12:39:05Z \
  --window-seconds 180 \
  --all-events \
  --output reports/rds-alert-window
We find Meaning
Slow statement for the same schema Useful correlation; inspect query and timing
still waiting, a lock wait, or deadlock Direct database contention evidence
Only unrelated tenant activity Database cause is not established; continue elsewhere

--all-events matters: it includes errors, lock waits, and deadlocks, not only
the slow-query records selected by the default mode.

A query plan turns "slow" into a testable hypothesis

EXPLAIN (COSTS, VERBOSE, BUFFERS, FORMAT JSON)
SELECT ...;

scripts/gabi-explain-json query.sql \
  --output reports/query-plan.json

This captures the planner estimate through the read-only gabi-cli connection.

Question answered: Where does PostgreSQL expect the work, and what data shape or index could change it?

PEV2 makes plan review a shared activity

scripts/pev2-report \
  --title 'Cost report aggregation — comparison' \
  --plan baseline=reports/baseline.json \
  --plan optimized=reports/optimized.json \
  --output reports/comparison.html
  • inspect scan, join, sort, aggregate, and parallel nodes visually
  • compare estimated versus actual rows and timing when ANALYZE is appropriate
  • share a local HTML artifact during review rather than a pasted terminal table
`EXPLAIN ANALYZE` executes the query. The helper requires an explicit `--allow-analyze` acknowledgement after the target and query scope are checked.

The workflow is now encoded, not rediscovered under pressure

Need Repeatable entry point
Alert context and history investigate-alert skill + GlitchTip snapshot
Cross-pod application logs scripts/kibana-search
PostgreSQL evidence scripts/cost-prod-rds-postgresql-logs --all-events
Query-plan capture scripts/gabi-explain-json
Plan visualization scripts/pev2-report
Operational guidance Cost Management runbook and focused guides

The tools preserve raw evidence locally, while the runbook captures how to use
it safely and consistently.

Demo: follow a real alert without guessing

  1. Open the GlitchTip issue; identify timestamp, recurrence, and pod.
  2. Check pod health and the current log tail in Kubernetes.
  3. Search the same window and correlation values in Kibana.
  4. Query RDS CloudWatch for slow statements, errors, waits, and deadlocks.
  5. If a query needs explanation, capture JSON through Gabi and compare the plans.
open reports/comparison.html

Click the visualization to open the interactive PEV2 comparison.