Your team's AI collaborator

Alert: checkout-service p99 latency > 10s
checkout-service p99 latency spiked from 380ms to 12.4s at 02:38 UTC.

Pulled APM traces since 02:15 UTC. Every slow trace is stuck on the same downstream call, POST /api/inventory/reserve to inventory-service.

Checked GCP Cloud Run logs for inventory-service. Instance count dropped from 8 to 1 at 02:35 UTC. Autoscaler shows a config change pushed at 02:34 UTC that set maxInstances to 1. That came from PR #3301 merged earlier tonight. It was supposed to update env vars but accidentally included a Terraform change to the scaling config.

Just saw the page. Nadia, you need a hand?

CodeRabbit already found it. Accidental scaling change in #3301. @CodeRabbit open a revert for just the scaling config, don't touch the env var update.

Done. PR #3315 opened: fix: revert inventory-service maxInstances to 12

Isolated just the scaling revert. The env var update from #3301 is untouched. Once merged, Cloud Run will scale back up within ~90 seconds.

Merging. Nadia, watch the dashboard.

Latency dropping. Back under 500ms. We're good.

Done. ENG-4120 created in Linear: "SEV-2 Postmortem: inventory-service scaling regression" with timeline, root cause, and remediation. Linked to PR #3301 and PR #3315.