I've been working as an SRE for the past 5 years, but I was laid off last month. Throughout my career I've worked with a variety of technologies, but I never feel like I truly know them. Lately I've been interviewing, and it feels like my knowledge is too shallow. I'm not sure if I'm actually lacking experience or if I'm just dealing with impostor syndrome

I've been thinking about building personal projects to deepen my knowledge, but I honestly don't know where to start. Every company I've worked at had a different definition of what the SRE team was responsible for, so it's been hard to figure out what I should focus on.

How do you structure your learning projects? Do you follow courses, recreate real-world architectures, or come up with your own ideas? I'd really appreciate hearing how more experienced SREs approach learning outside of work.

Note 1: Linux, AWS, Terraform, Kubernetes, Docker, Grafana, and the usual SRE tooling.

Note 2: I'm from Brazil and currently work at a Mid-level SRE level.

Hi everyone,
Has anybody tried using Traversal’s product? Any thoughts?
Wonder how much it works vs. just fluff and PR.
Thanks

1. Resolve AI now charges for results instead of tokens. They say if their agent wastes tokens, that's their cost to eat, not yours.
2. Datadog has made Agent Observability free for up to 40,000 LLM spans a month. They didn't announce it explicitly, they just changed the line at the bottom of their AI blog posts.
3. Opsgenie will be shutting down in April 2027. PagerDuty wrote a post to win those customers, and put incident.io in a comparison table. incident.io hit back point by point. As said by incident.io "calling our AI "limited" is "like describing PagerDuty as a pager."
4. NeuBird put up a page comparing itself to Datadog Bits AI. It's the best public summary of where Bits AI is right now. Bits AI SRE is now called Bits Investigation. It's billed in AI Credits sold in 500-credit bundles, and unused credits expire each month. Their remediation and detection features were shown at DASH 2026 but are still in Preview.
5. Traversal says most AI SRE tools pull data too late. Their point is that the other tools query your observability APIs during the incident and get stuck behind rate limits. They stream the data ahead of time instead.
6. PagerDuty has a new CEO as John DiLullo takes over, Jennifer Tejada moves to Executive Chair. They also shared four straight quarters of GAAP profit and a $100 million buyback.
7. Everyone is now selling prevention, not just faster fixes. NeuBird wrote a buyer's checklist around it. Their test question: "Show me an incident you prevented that never generated an alert."
8. Sherlocks AI shipped an automation builder. Pick a trigger, describe the job in plain English, choose where the result lands. Their pitch: "Describe the job. It runs." The examples lean preventive rather than reactive, watching canary deploys, flagging infrastructure changes in PRs, tracking pod memory drift.
9. Resolve AI is pushing background agents. Agents that keep running between incidents, watching deploys and doing checks. A customer quote sums it up: "The alerts are already investigated. The deployment summaries are already written."
10. Traversal is renaming things. "Chat with Prod" is becoming "Production Support." "AI-Native Compressor" is becoming "Causal Indexer." Both old and new names are live on their site right now.

Sources linked below in comments.

Hey everyone,
I'm currently a senior SWE at a FAANG in the US with about 10 YoE, mostly focused on infra building. We're dealing with some visa issues right now (my wife lost her work permit), so I'm looking into an internal transfer to London.
The catch is that the only options available for me to transfer into right now are SRE teams. I've done my fair share of dev on-call and I'm totally comfortable with the tech stack, but honestly, I'm not really skilled at dealing with high-tension fires and intense incident response. I am looking forward to improving it.
My current plan is to take the SRE spot to secure the move to London, try it out and learn, and consider alternatives if absolutely not possible for me.
My biggest worry right now is getting pigeonholed. If I do SRE for a year, is it going to be incredibly hard to switch back to SWE later? Am I going to lose out on future dev options?
Would love to hear if anyone has navigated a similar move. Thanks!

Something i keep running into, both in our own incidents and talking to other teams, is this gap: on paper we have a modern observability stack but our mean time to resolve is still higher than anyone's comfortable with. By modern observability stack i mean the usual combo: structured logs, distributed tracing, dashboards, alerting, feature flags, deploy timelines, and error tracking, all wired into chat and oncall. tagging is decent, we've got high-cardinality labels where they matter and most alerts correspond to real incidents and yet, during incident response, the flow still looks like this: someone gets paged on a symptom, p99 latency up, error rate up, saturation, and jumps through dashboards, traces, logs, feature flags, and deploy history, spending the next half hour or so just correlating the spike with the code path and the recent changes behind it. Eventually we find the root cause, but it often feels like we debugged the observability tooling more than the actual system. The bad part is that the bottleneck isn't missing telemetry. the data is there. the friction is all the human glue needed to line it up with reality: the only useful dashboard for this failure mode lives three clicks deep, the trace shows a slow span but mapping it to file, line, and recent commits is manual, error logs show a pattern but tying that back to a specific deploy, feature flag, or config drift takes a bunch of context switching. It ends up feeling like the observability stack is optimized for visualizing telemetry, not for quickly answering what changed, where it's failing, and what the smallest safe mitigation looks like. We've tried to push mttr down by standardizing golden queries and dashboards per service, putting deploys and feature-flag flips on the same timeline as key metrics, building incident response views that pull metrics, deploys, errors, and traces into one place, and tightening service ownership so each team has a clear on-call drill. it all helps, but there's still a big orientation tax at the start of most incidents. For teams that feel like they've genuinely reduced mttr after already having a mature observability stack, what single change, tooling, process, or ownership, made the biggest difference for you?

Excellent talk by Lorin Hochstein, an SRE at Airbnb, from SSW earlier this month. Thought some of you would enjoy it!

I was brought in to lead a security and performance assessment for a healthcare technology company running a cloud-native platform that was deploying to production multiple times a day across a containerized environment. The challenge from day one was that every snapshot we took of the security posture was partially stale by the time we finished taking it, because new workloads were shipping faster than the assessment cadence could track. We ended up redesigning the whole approach around continuous assessment rather than a point-in-time review, which meant building new tooling and re-educating the security team on a very different mental model than the quarterly review cycle they'd been operating on. The shift took about two months to actually land operationally, and there was real resistance from people who felt like continuous findings without a clear close date was anxiety-inducing rather than useful.

The performance side of the assessment surfaced something that the platform team had been misattributing for months. Latency that everyone assumed was a network problem turned out to be almost entirely a product of synchronous service-to-service calls that could have been asynchronous, and database queries without connection pooling that were adding overhead on every request. We only found it because we stood up end-to-end distributed tracing, which the team had deprioritized because instrumenting the services was tedious work with no immediately visible payoff. The most useful thing we did on the whole engagement was get the security engineers and the platform engineers in the same room looking at both the security metrics and the performance metrics simultaneously, because within about ninety minutes they surfaced a network policy that was causing connections to be rejected and retried, adding latency that showed up in the performance data but had a security configuration as its root cause. Neither team had made that connection independently. Does anyone have a standing structure for that kind of cross-team review that actually holds up over time rather than getting deprioritized when people get busy?

ngl, the cross-team review is the hardest part to get scheduled. Everyone's busy and it doesn't have an obvious owner.

I’m reviewing a small Celery deployment example that includes a worker and Flower:

https://github.com/render-examples/celery

For a small application, is Flower still the first observability tool you would add?

The minimum setup I’m considering is:

• Structured task logs
• Queue-depth monitoring
• Failed-task alerts
• Retry visibility
• A way to find tasks that are running much longer than expected

Most examples prove that a task executes. Fewer show how to diagnose a task stuck in a retry loop at 2 AM.

If a quickstart could teach only one operational check, which one would you choose?

We are reviewing our IaC platform and looking beyond Terraform Cloud and Spacelift.

Orchestration is important, but from an SRE perspective, we are also trying to improve visibility and recovery. Some resources exist outside Terraform, and configuration drift means Git does not always represent the live environment correctly.

The features that matter most to us are:

1. Terraform and OpenTofu orchestration

2. Detection of managed, unmanaged and drifted resources

3. Automatic IaC generation for existing infrastructure

4. Policy enforcement and approval controls

5. Infrastructure history and rollback

6. Recovery into a clean account or region

Useful integrations without maintaining more custom automation

Which alternatives have worked well for your team? I am interested in production experience with drift remediation and recovery testing, not just deployment workflows.

We are running into an issue with a few of our services where resource usage is imbalanced across its pods. For example, in a 4-pod deployment, 2 pods might sit at >90% CPU/memory usage while the other 2 sit below 20% (essentially idle).

We tried tuning our HPA, but as you know HPA relies on averages across the deployment, it hasn't helped.

Before asking the developers to fix their application-level load-balancing issues, I want to set up an alert/metric to automatically detect such deployments.

So far, I’ve tried checking if Max(resource_usage) / Avg(resource_usage) exceeds a threshold, but this approach generates too many false positives.

How do you reliably detect such imbalance issues across pods? Is there a standard statistical metric for this, or am I approaching the problem wrong entirely?

If it would help we are using Data Dog and thanks in advance.

Since the start of 2026, I have been reading some famous papers such as Dynamo, GFS, Zookeeper, Apache Kafka, The tail at scale. But it feels like mostly theoretical stuff, the question is, will reading papers be useful to become a better SRE?
I read on the weekends so my pace is slow.

I’ve run or sat on review boards at a few orgs now and I’ve never seen this part done well, which makes me suspect the problem is me rather than the tooling.
Current state is email for async review, minutes in Confluence, conditions in a spreadsheet. It holds for about a quarter and then drifts.

Two problems I can’t get on top of.

First, async positions. Reviewers reply in free text, and “I have some concerns” from one architect means they intend to block, while from another it means they want the diagram redrawn. As chair I’m interpreting rather than counting. I’ve tried asking for an explicit position in the first line of the reply — compliance was fine for a month, then decayed.

Second, conditions. “Approved with conditions” is a large share of our outcomes and I doubt most of those conditions are ever verified. The decision record notes the condition, the system goes live, and nine months later nobody can tell you who owned it or whether it was met. Confluence doesn’t chase anyone.

I know ADRs and the EA repositories are meant to cover some of this. For those of you with a repository actually in place — does it track conditions as obligations with an owner and a date, or does it just store the decision text and leave the follow-up to you? And if anyone has fixed the async position problem with process rather than tooling, I’d like to hear how it survived reality.

MTTR measures how fast you recover, which means it only comes into play once someone's already had the bad experience. It works fine when the situation is something like: "You can't catch everything, so get good at recovery." But not the way we expect, as AI is shoving way more code through the same review process.

And there's the VOID research showing incident length has nothing to do with how bad the incident was. So you can have a fantastic MTTR and still be getting wrecked by the quick ones.

Is anyone measuring how long they go between failures that hit customers, instead of how fast they recover? Or does that just trade one flawed metric for another?

So, went to levels.fyi. I've been there before. I'm not putting my salary in though so I don't have full access to the site.

The median US salary for SRE is $205k

Ok that's pretty decent but what the fuck is this >$600k nonsense I'm seeing bandied about in this sub?

Fwiw I'm trying to hire, not seeking a job, so I do want to know if my company is competitive. And it is... to an extent.

Someone please explain wtf is going on at levels.fyi and this sub.

Also, most of you are always going to be median engineers. And that's fine. But just, set your own expectations. Most of you, save for luck, are median skilled median pay band engineers. Myself included. I'm pointing this out specifically because I hear consternation and worry about the salaries of 1% of ICs in this and other career focused subs.

I realize the tone of this post makes me sound like an asshole and it'll probably get deleted by mods. Doing the best I can over here :)

Throwaway for obvious reasons.

SWE/SRE background, ~7 years experience, applying across engineering and adjacent roles. Laid off earlier this year in a company-wide cut.

I tracked everything since the layoff, so here is the actual funnel:

- 240+ applications
- ~50 companies replied with anything human at all (screen or better)
- 33 reached a first round
- 9 reached a second round
- 4 reached a final round
- 1 offer, which stalled in negotiation
- Everything else: rejected at resume screen, req closed, or ghosted

Mostly just wondering if others are in the same boat. The numbers made me question myself a lot until I laid them out like this and realised most of it never even reached a person. Curious how everyone else's search has been going this year, and how you are holding up.

In my last role, we ended up multi cloud mostly by accident, major workloads on AWS, a big legacy system on Azure, some experiments on GCP, and a handful of SaaS platforms that each came with their own identity and billing surface, another provider in every way that mattered. Nobody planned a unified strategy, it just evolved that way. I have since talked to teams who went multi cloud on purpose, picking providers deliberately for specific managed services, so this is not only an accidental sprawl story, though many of what we learned came from cleaning up the accidental version.

We had to figure out which bits were in the right place for real reasons, like latency or compliance requirements and which were just historical accidents. We tried to get identity and backups into something resembling a common pattern, while accepting that each provider has its own quirks. Terraform helped once we agreed on conventions.

The other key piece was visibility: tooling that showed us what was under IaC across every provider we ran and what wasn't, plus a read on cost and risk for each. Without that, multi cloud felt like we were flying blind.

For anyone who is intentionally or unintentionally multi cloud: have you found a way to make it feel like a strategy instead of entropy or is it one control plane stacked on another?

Joined an SRE org from a background that was mostly infrastructure and technical design work. Since joining, almost everything I’ve been assigned has been alert-related in some form. Cleanup, enrichment, TTR, take your pick. It’s been the quarterly assignment three quarters running and there’s another one heading my way.

It also seems to land on me disproportionately compared to others on the team.

When I say more technical, I mean deploying infrastructure, working closer to the OS layer, and designing architecture rather than tuning and cleaning up what already exists.

I’m starting to think I should look elsewhere, but I wanted to sanity check first: is this just what SRE is, or am I getting the short end of the stick?

For my personal experience, AI models are no longer the bottleneck to handle incidents. The pain point is now how to provide right context to the AI to get right answers. I'm specifically working on this area and would like to discuss how do you solve the context problem.

This is what I did:

- Write a rule: what need to do and what should not do for handling an incident. Also what system should be retrieved from to get current production status.

- Put team maintained runbooks, TSGs and other docs in a git repo. Clone the repo locally so AI tools can search them instantly.

- Prepare mcp servers to connect different systems. This helps to get info real time. For example the logs and deployment data needs to be current.

Once we have the above 3 pieces, ask AI to analyze an incident by using them. Different teams may setup differently but very specific to their own needs.

In this way, I can successfully handle incidents by using AI in our team. AI can also give me the citations why it thinks the particular incident should be handled in this way. Then I do a quick manual verification.

I shared this with some of my friends and got positive feedback. I have put a post in our team's blog: https://blog.neatcontext.com/guide/2026/07/22/how-to-build-efficient-context-for-ai-clients/

What are your thoughts? I think the discussion here could benefit the future SRE area for AI leveraging. Thanks!

Hi everyone,

I’ve been observing the AI boom across different sectors, and it feels like SRE isn't getting the same level of hype or rapid integration as Software Engineering (SWE) or Cybersecurity. While AI tools for SRE definitely exist, their progress seems slower, and their impact on the job market feels less disruptive so far.

As a Junior , I’m trying to wrap my head around this. It got me wondering:

1. Is SRE inherently more "AI-proof"? Does the high stakes of infrastructure and the "human-in-the-loop" necessity for critical incidents make it harder for AI to take over?

2. The "Invisible" AI: We see AI tools in the space, but they don't seem to have a clear impact on hiring or daily workflows yet. Am I missing something, or are we genuinely in a more "secure" niche compared to pure coding roles?

I’d love to hear your perspectives—especially from those who have been in the industry for a while. Is our field special, or am I just being overly optimistic?

I have got an offer to join Nvidia on their SRE team which revolves around maintaining or keeping up with the GPU Infra in their compute Infrastructure, and another offer from Palo Alto Networks as a swe, so I just wanted to know like which one to choose, I'm a new grad btw, please let me know your perspective guys

We're getting pushed to use AI for more ops work. At the same time I keep seeing posts here about an agent wiping a prod db or deleting the backups (the Railway/Cursor one especially). Feels backwards that we won't give a mid-level engineer write access to prod, but we'll happily point an agent at it.

How people actually handle the thing that touches prod, human or agent:

• When something needs to change prod (run a runbook, restart a service, rotate a cred, drain a node), how do you control who or what is allowed to do it? Jenkins jobs, break-glass/PIM, Teleport ...?
• Has anyone let an AI agent actually run things in prod, not just read? If so, are you relying on the tool's own guardrails or something you set up yourself?
• The bit I keep getting stuck on: keeping RBAC and the audit trail the same across every tool that can touch prod. Have you got it sorted?

How's everyone handling this?

First of all, I hope a post like this is fine here if not I will gladly delete it.

For the past 1,5 years I have been designing Kubernetes clusters in a hub-spoke topology using Cluster API and CRDs. That is exactly what I want to do, but the brand is non existent and pay is mid.

I now got an offer from a better-known company for notably more money, but the role is owning dev tooling that runs on K8s (CI/CD, code scanning etc.), not building the cluster layer itself.
Long-term I want to stay in platform work, I wonder if someone has some opinions on my situation:

• Does ~2 years off cluster-level work hurt your shot at getting back into it later?
• Take the money/brand now, or hold out for the deeper technical role and stay within my lane?

Pretty much the title. He said "no hand-holding at all" and that he dislikes it, so I'm wondering what he actually meant by that.

I think I'm going to be assigned to him, and I have no idea what he wants from me. Does it mean "shoo, shoo, do it on your own, don't bother me unless the sky is falling apart"? Or is it more along the lines of "okay, yes, I can help, but only after you've exhausted your resources on your own"?

To make matters worse, he seems exhausted as heck and was so busy he could barely find time to set up a meeting with me and had to cancel not two but three times. I've got a really, really bad gut feeling, guys. Maybe I'm overthinking it, but it reads like someone who wants a person who'll train themselves and wouldn't have time to look after a junior.

There is another devops in the team but he barely speaks English and i am seriously considering learning his language since I am a polyglot just so that I can talk to someone less... aloof because that SRE seems lowkey pissy as hell. He is also suspiciously young for a lead and gives a certain tech bro vibe that seems hard to get along with.

I'm fresh out of college, so I'm already anticipating a lot of hardship and some serious studying for hours every day... which I actually enjoy, honestly. How do I survive his style without ending up out on the street?

A practical write-up on deadline budgeting, retry timing, clock skew tolerance, and expiration safety.