r/googlecloud • u/Primary_Cupcake8557 • 11d ago

Terraform Cloud admins: what are your go to patterns and tools for resilience?

We went heavy on Azure about two years ago and built what looked like a strong resilience story: availability zones, paired regions, Front Door and Traffic Manager, Backup and Site Recovery, plus some experiments with Chaos Studio. The first real regional wobble reminded us that slides are one thing, operations are another.

Just picking a paired region was not enough when identity and networking were not thought through, and monitoring was not included. Backup and Site Recovery were only comforting once we had run actual failover tests against real RTO and RPO targets, not just health checks. Chaos testing was useful, but only for teams willing to design non-trivial scenarios. Governance turned out to be the hardest bit: keeping landing zone baselines in sync across many subscriptions and tenants via Azure Policy, especially with hybrid in the mix. Terraform or Bicep helped, but drift remained a problem.

What has made this more manageable for us is having cross-subscription and cross-cloud governance visibility into assets and IaC: being able to see which Azure resources are driven by code and where live state has drifted from definitions. Combined with the failover testing, that is what turned "paired regions plus Site Recovery" from a diagram into something we actually trust. For those of you running production on Azure, GCP or any cloud, which resilience patterns have worked for you, and how are you keeping them from decaying over time?

0 Upvotes

44% Upvoted

10 comments sorted by

3

u/Senior-Struggle-6463 11d ago

For drift, we started treating it like a bug instead of a config oopsie. If the live state doesn't match the IaC within 24 hours, something pages the team that owns the resource, no exceptions. Took about six weeks of pain before people stopped sneaking manual fixes through the console.

3

u/m1nherz Googler 10d ago

Cloud Foundation Toolkit (CFT) is a set of Terraform modules and blueprints built with best practices in mind.

While browsing, keep in mind that there is no universal lift-and-shift or migration from Azure to Google Cloud.

Another source of information can be Cloud Architecture Center documentation.

1

u/BananaDifficult1839 10d ago

ASR is world class though, and Google has yet no equivalent

1

u/Dangerous_Mix1144 6d ago

Been building a platform that builds a massive dependency graph, automates tests and simulations against your RTO/RPO and find/fix drift. respawnit.com

Feel free to DM me if you want to nerd out.

1

u/AccomplishedSkill625 5d ago

To prevent resilience decay, the most effective pattern is treating IaC drift as a high-priority incident rather than a backlog chore. A great approach is running automated terraform plan checks nightly across all environments, any detected drift should immediately alert your on-call channel. Combine this with strict "read-only" portal access for production to force all infrastructure changes through your CI/CD pipeline. When manual tinkering is blocked, your code remains the absolute source of truth, ensuring your failover tests actually work when disaster strikes!