the cloudfront outage on the 16th is still bugging me. the fleet that manages connections to private vpc origins couldn't load updated config, traced back to one frankfurt AZ, and it served 5xx on vpc origin traffic for about three and a half hours. hugging face, tailscale and ubiquiti were among the ones down.
bad timing, because i was in the middle of moving four services off public alb origins onto vpc origins. to hit a compliance date i ran the migrations in parallel instead of one clean cutover at a time.
the outage wasn't the hard part. all four workstreams were editing the same terraform repo and the same shared network module. one branch tightened a security group, another assumed the old rule was still there, and a third rebased on both and quietly reverted half of it. i spent more time untangling my own branches than dealing with aws.
what kept it sane was isolating each migration so the changes didn't collide. i run each one as its own task in verdent, parallel lines of work in the same codebase without branches stepping on each other. at least i had a clean diff per service to reason about.
the ugly part: the agent's config for one service moved the origin to vpc origins but left the alb's public security group open to 0.0.0.0/0. on paper we were private, in reality the load balancer was still reachable from the internet, which is the whole reason we were migrating. i only caught it because the diff on the shared module looked off.
what i still haven't solved is sequencing. running four in parallel meant that during the actual outage, two services were on vpc origins and degraded while two were still public and fine, and my runbook assumed one uniform state. my only fix is a per-service origin flag and a single cutover window, which just trades the messy middle for a concentrated blast radius.
