One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7.
The solution is to hire operations teams globally, and then setup follow-the-sun operations whereby the people being paged are always naturally awake at that hour, and allows them to work normal eight hour shifts. But this requires companies to, gasp, have specialized developers and specialized operators collaborate before allowing new feature work into production, to ensure that the operations teams understand what the services are supposed to do and keep it all online. It requires (oh, the horror!) actually maintaining production standards, runbooks, and other documentation.
So naturally, many orgs would prefer to burn out their engineers instead.
I would respectfully say that you are wrong. I speak from experience. At Netflix we tried to hire for around the clock coverage. But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time.
Yes, you would get calls at 2am, sometimes multiple days in a row. But you were only on call once every six to eight weeks, and we scheduled out well in advance so you could plan your life accordingly.
As a bonus, for the five weeks you weren't on call, you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am.
> It requires (oh, the horror!) actually maintaining production standards, runbooks, and other documentation.
I disagree with this too. Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary, or at least surface the right information automatically if you can't automate it.
This is the same approach as night shifts for nurses.
There’s a lot of evidence to suggest that the effects on this infrequent but consistent disturbance to their circadian rhythms causes all kinds of physiological damage. One example [1]. We have to do better. I think the original suggestion of finding specialised night workers or those in other timezones is more humane.
Here is a reference that is a bit more attributable to the on call experience. There is a tangible human cost to after hours responses during an on call rotation. I personally do not recommend on call roles to any technology professional who can avoid them due to these health consequences of an on call requirement.
> Sleep plays a vital role in brain function and systemic physiology across many body systems. Problems with sleep are widely prevalent and include deficits in quantity and quality of sleep; sleep problems that impact the continuity of sleep are collectively referred to as sleep disruptions. Numerous factors contribute to sleep disruption, ranging from lifestyle and environmental factors to sleep disorders and other medical conditions. Sleep disruptions have substantial adverse short- and long-term health consequences. A literature search was conducted to provide a nonsystematic review of these health consequences (this review was designed to be nonsystematic to better focus on the topics of interest due to the myriad parameters affected by sleep). Sleep disruption is associated with increased activity of the sympathetic nervous system and hypothalamic–pituitary–adrenal axis, metabolic effects, changes in circadian rhythms, and proinflammatory responses. In otherwise healthy adults, short-term consequences of sleep disruption include increased stress responsivity, somatic pain, reduced quality of life, emotional distress and mood disorders, and cognitive, memory, and performance deficits. For adolescents, psychosocial health, school performance, and risk-taking behaviors are impacted by sleep disruption. Behavioral problems and cognitive functioning are associated with sleep disruption in children. Long-term consequences of sleep disruption in otherwise healthy individuals include hypertension, dyslipidemia, cardiovascular disease, weight-related issues, metabolic syndrome, type 2 diabetes mellitus, and colorectal cancer. All-cause mortality is also increased in men with sleep disturbances. For those with underlying medical conditions, sleep disruption may diminish the health-related quality of life of children and adolescents and may worsen the severity of common gastrointestinal disorders. As a result of the potential consequences of sleep disruption, health care professionals should be cognizant of how managing underlying medical conditions may help to optimize sleep continuity and consider prescribing interventions that minimize sleep disruption.
> But what ended up working much better was taking that same team and having each person on call for a week at a time, all based in Pacific Time.
Our support team does the same, and they seem to be quite happy with it. They also get the following Friday off (in addition to compensation).
They do their best to shield us developers from after-hour calls, usually one can get things moving enough that it can be handled properly in the morning.
Even as a dedicated operations team for a product, we did this too. On call person worked tickets and took calls for one week at a time, the rest of the team worked on ways to make on-call suck less. For an eight person team it worked well for about three years until bigger stuff happened in the org and we all parted ways.
> you were highly incentivized (and had the time) to build tools or submit patches to fix the problems that woke you at 2am.
Ah, so you worked on a team where the SRE needs were prioritized over the feature requests? Because in most companies where I've worked, Product + Customer Service + Sales + Marketing + Executives don't really have time or patience for the engineers to get their diamond polishing cloths out. They want to see feature development. They're willing to be forced to prioritize exactly which feature they'll get soonest, and they understand that engineering needs time to keep the systems running, but in most businesses I've worked, the business comes first.
> Documentation and runbooks are useless in an outage. Instead of runbooks, write code to do the thing. Instead of documentation, comment the code and build automation to make the documentation unnecessary
We do that too. If you could write code to Solve All The Problems then you'd never need to page a human in the first place ;)
I'll give you a simple example of where you can't write code to solve this sort of thing. Let's say that you have an autoscaler that will scale your server group up to X servers. You define an alert to page you if the autoscaler hits the maximum. The page goes off. Do you really want to write code to arbitrarily increase the autoscaler maximum whenever it hits the maximum? Why do you have the maximum in the first place? The entire reason why the autoscaler maximum exists is to prevent cost overruns from autoscaling run amok. You want a human being, not code, to look at the autoscaler and make the decision. Do you have steady-slow growth up to the maximum? Maybe it should be raised, if it represents natural growth. Maybe it shouldn't, if you just raised it last week and it shouldn't be anywhere near this busy. Do you have hockey-stick growth? Maybe the maximum is working as expected, looks like a resource leak hit production. Or maybe you have a massive traffic hit and you actually do want to increase the maximum. Maybe you'd prefer to take the outage from the traffic hit, let the 429s cool everyone off. But good luck trying to write code to handle that automatically, and correctly for you!
> or at least surface the right information automatically if you can't automate it.
Ah, well, that's exactly what the dedicated operations staff are doing, because when you have three follow-the-sun teams, you need standards, not three sets of people who each somehow telepathically share the same tribal knowledge?
Don't get me wrong, I'm not anti-automation or something. If your operations folks are click-clicking in consoles all day long, the same click-clicking every day, probably something's wrong. But the SRE model asks for operations automation to stick within operations teams, not development teams.
> Ah, so you worked on a team where the SRE needs were prioritized over the feature requests?
Yes, it was an SRE team. All we do is write tools to make operations better, but more importantly we write tools to make it easier for the dev teams to operate their own systems better. But yes, we had products teams that would push back on our requests because they had product to deliver, and that was fine. We'd either figure out how to do the work for them, or figure out a workaround.
> We do that too. If you could write code to Solve All The Problems then you'd never need to page a human in the first place ;)
Well yes, that's the idea. You can't get to 5 9s of reliability unless it's all automated. :)
> I'll give you a simple example of where you can't write code to solve this sort of thing.
I could easily write code to solve the thing. Step one, double the limit to alleviate immediate customer pain. Step two, page someone to wake up and look at the graphs and figure out what the better medium term solution is to get us through until the morning, including links to said relevant graphs.
You're not gonna have a cost overrun doubling the limit for one night. And if there is a big problem, the person will get paged again a few hours later and have more information to make a better decision.
> But the SRE model asks for operations automation to stick within operations teams, not development teams.
Yes, but I'm not sure I see why that's bad. I don't see any purpose for a dedicated operations team, especially a follow the sun team. If you're Google and you already have offices all around the world, sure, it will be better. But it makes no sense to hire an around the world team just for operations if the rest of your company is in one time zone.
> Yes, it was an SRE team. All we do is write tools to make operations better
Go back to my original comment. If you're an SRE team, then basically, you're the operations team for the developers. I'm talking about where developers are responsible for their own operations and there is no team that gets paged instead of them - "most orgs ask a single team to handle on-call around the clock for their service."
> Step one, double the limit to alleviate immediate customer pain. Step two, page someone to wake up
See, what I read from this is: a) violate my system efficiency KPIs while b) paging someone in the middle of the night anyway. So, lose-lose.
> But it makes no sense to hire an around the world team just for operations if the rest of your company is in one time zone.
Why does it make any more sense to hire developers remotely who are in your time zone ± three hours? Because that's what most companies are doing these days. If you're already hiring people remotely then you can hire Operations/SRE staff a little further afield and see that as a benefit (follow the sun) rather than a problem.
> the rest of your company is in one time zone.
For what it's worth, we also hire salespeople around the globe :) Fact of the matter is, it would be so, so nice for Slack to turn off the ability to @channel in the #random channel so that people who are asleep don't get pinged ...
We were an SRE team building tools for the development teams who got paged in the middle of the night. The devs writing the services were operating their own services and were getting paged. We would sometimes also get paged for a serious incident so we could coordinate if multiple development teams were involved.
Each team managed their own rotation schedules, we just made sure they had one.
> See, what I read from this is: a) violate my system efficiency KPI
If you're being graded on your system efficiency and not customer satisfaction, well then sure, your way might make sense (but I'd still say it doesn't). But your business will suffer if you optimize for efficiency over customer satisfaction.
> Why does it make any more sense to hire developers remotely who are in your time zone ± three hours?
Because it's a lot easier to run a team where everyone on the team can meet at the same time. If you have an around the world team, there is no time of day where you can have a meeting and everyone gets to attend during their workday. Realistically you can maybe get away with a nine hour time difference. Any more than that and you have people excluded.
Especially if the bulk of your devs are in one or two time zones, your operators will be even more disconnected from them since they will never be able to interact with the devs, and the devs will have no empathy for the operators who they also never interact with.
> For what it's worth, we also hire salespeople around the globe
Sure, but they aren't writing code that your operators have to run. :)
I think we both agree that it's better for devs to get paged for their services instead of operators, and if that's the case, its far better for all the devs to work together and know each other and be in the same or nearly same time zone.
> But your business will suffer if you optimize for efficiency over customer satisfaction.
But who are the customers? Business, engineering, or finance? :)
> it's a lot easier to run a team where everyone on the team can meet at the same time.
Of course it's easier. It's also easier not to maintain documentation or standards, just be a five person startup and have everyone be in the same room. Enterprise communication is hard! Even when you're in the same time zone. The question isn't "how do I get my life to be a utopia?" but "which challenges should I choose?". If you run an organization, you need to put your employees first, even ahead of your customers. Employees and customers both come and go but 80% of the time the effect of an valued employee leaving is far worse than a customer leaving, and you have far more control over whether employees leave than whether customers do. So you can either put your employees first (build a calm workplace) or you can put your customers first (prioritize feature development velocity in organizational design).
> I think we both agree that it's better for devs to get paged for their services instead of operators
No! Dev should never be paged! If I "buy" Jenkins off-the-shelf, and it breaks down in production, guess what, I don't get to page the Jenkins developers! Why should internally developed services be any different? If Ops needs to page someone from Dev instead of waiting for a response at normal business cadence, then this is an Ops failure, not a Dev failure!
> But who are the customers? Business, engineering, or finance? :)
The business's customers. The ones who pay your company so they can pay you, and your reason for having a job at all.
> Why should internally developed services be any different?
Because they're your core competency and you have control over it. If you could page the Jenkins developers you probably wouldn't hesitate to do it, because you'll get better results. Why not get the best results you can from an internal service?
> If Ops needs to page someone from Dev instead of waiting for a response at normal business cadence, then this is an Ops failure, not a Dev failure!
I couldn't disagree more. That is absolutely a dev failure -- they wrote a service that couldn't operate under the conditions given. It's either a bug or an architecture issue, but no matter what, it's a dev issue and the dev should be responsible for building a service that can actually run in production.
You and I have very different ideas of a successful engineering organization. I would never want to work for your org as an operator or a dev. As an operator the last thing I want is devs to throw whatever they write over the wall and then say "not my problem anymore!", and have to rely on getting retrained every time the code changes. And as a dev I wouldn't want to be in an organization that accepts sloppy developers who aren't responsible for building solid code that can run under adverse conditions and who don't get to experience the issues in production for themselves.
Facebook makes their devs get paged, Netflix does, Amazon pages their devs, Dropbox pages devs, Stripe pages devs, and Google pages their devs too until they have demonstrated multiple quarters of success, and only then does an operator take over. And if the service has too many failures, support falls back on the devs until they can make the service stable again.
Making devs responsible for creating code that actually works well in production is a good thing.
> As an operator the last thing I want is devs to throw whatever they write over the wall and then say "not my problem anymore!", and have to rely on getting retrained every time the code changes. And as a dev I wouldn't want to be in an organization that accepts sloppy developers who aren't responsible for building solid code that can run under adverse conditions and who don't get to experience the issues in production for themselves.
How can you classify anybody who writes on-prem software as being a "sloppy developer"? Jira, Jenkins, GitLab, pretty much any database you can imagine (MySQL, PostgreSQL, Redis, Elasticsearch, Kafka...), Grafana, any Linux distribution, they're all written by "sloppy developers"?
Where did I say that Dev gets to "throw code over the wall"? How would you feel if I unilaterally decided for you, as a developer, which tools you get to use? If I came up with some policy that the whole organization can only run Windows machines and I "threw that policy over the wall" at you?
You're arguing against a strawman that's completely inconsistent with how harmonious follow-the-sun Ops actually works.
I would not call follow the sun ops as harmonious. If anything I'd call it adversarial. Ops is always trying to blame dev for outages and dev is always trying to blame ops. Each accuses the other of not sharing all the necessary information.
Look at all that on-prem software you just mentioned. The developers of every one of those complain that they need better bug reports, and the people who operate them complain they need better documentation. Things would be much better if those devs worked directly for every company that uses them, and in fact in a lot of cases one of the contributors is an operator at a company. Why do you think companies like to hire open source devs? To get better access to someone who knows the codebase!
It's far better if the operator is the developer. Sometimes we live with that not being the case because the software is made by others. But when given the choice, I will always opt for the dev running the software themselves.
> Step one, double the limit to alleviate immediate customer pain.
I've been oncall for systems where that would not work.
Doubling the memory means you need twice as many machines. Depending on the service, that could require significantly increased network bandwidth. Now the network is saturated and every node needs to queue more data. Now latency and throughput are even worse, and even more requests are being dropped, so you automatically double the limit again...
While that all may be true (but are indications of a poorly architected system), my code would still work. It would double the limit and then page someone. If they logged in and saw all those failures, then they could address those issues.
The whole point is that having an around the world follow the sun team would not alleviate those issues or make anything better.
This. Absolutely this. Working on large distributed system can be both exhilarating and exhausting. The two often go hand in hand. However, working on such systems without diligence tips the scales toward exhausting. If your testing and your documentation and your communication (both internal and with consumers) suck, you're in for a world of pain.
"But writing documentation is a waste of time because the code evolves so fast."
Yeah, I hear that, but there's also a lot of time lost to people harried during their on-call and still exhausted for a week afterward, to training new people because the old ones burned out or just left for greener pastures, to maintaining old failed experiments because customers (perhaps at your insistence) still rely on them and backing them out would be almost as much work than adding them was, and so on.
That's not really moving fast. That's just flailing. You can actually go further faster if you maintain a bit of discipline. Yes, there will still be some "wasted" time, but it'll be a bounded, controlled waste like the ablative tiles on a re-entry vehicle - not the uncontrolled explosion of complexity and effort that seems common in many of the younger orgs building/maintaining such systems nowadays.
The solution to get paged at off hours a lot is rarely to hire additional teams to cover those times for you, at least not long term. For things you can control, you should fix the root causes of those issues. For things you can't control you should spend effort on making them within your control (eg architecture improvement). This takes time, so follow-the-sun rotation might be a stop gap solution, but you need to make sure it doesn't cover over the real problems without them getting any better.
From experience, it's really hard to fix the root causes of issues when you were woken up three times the night before and had two more of the same incident occur during the workday. In my case I struggled along for a couple years but the best thing to do was just leave and let it be someone else's problem.
If they cared about that they would either pay me so much money I'd be insane to walk away or they would hire people in other time zones to cover the load. Instead they chose to pay for their customer satisfaction with my burnout. The thing about that strategy is... eventually the thing holding their customer satisfaction together gets burnt out. So I leave. And even then they're still getting the better half of the bargain.
Sorry, I accidentally said you did the wrong thing for leaving. That wasn't my intention. Of course, leaving was the right choice for you.
What I meant was the company you were working for does not get the best quality or customer satisfaction by overworking you to the point where you have to leave. It would have been better for their software quality to handle things differently.
I don’t think this is a stable long term solution. The “on call” teams end up frustrated with the engineers who ship bugs and this results in added process that delays deploys, arbitrary demands for test coverage, capricious error budgets, etc. It’s much better to have the engineers who wrote the code be responsible for running it, and if their operational burden becomes too high, to staff up the dev team to empower them to go after root causes. Plus the engineers who wrote the code always have better context than reliability people who tend to be systems experts but lack the business logic intuition to spot errors at a glance.
I don't think the parent was implying you're never on call for your code, just only on call during working hours.
One of the challenges for larger companies in trying to make teams on-call 24/7 is that your most senior engineers often have enough money that they don't have to take on-call. Some variation of the following conversation happens in Big Tech more than most people seem to anticipate:
"hey, so I have 7 mil in the bank, a house, and kids; so I'm not taking on-call anymore"
"I understand on-call is a burden, but the practice is a big part of how we maintain operational excellence"
"Alright, I quit"
"Woah woah woah, uh, ok, what about we work on transitioning you out of on call over the next 6 months?"
"Nah, I'm done"
"This is going to be really disruptive to the team!"
"Yeah man it sucks, I really feel for you"
My understanding is a few famous outages at large cloud providers are a direct result of management not anticipating these conversations and assuming 24/7 on-call from a single geographically centered team of high powered engineers was sustainable.
> The “on call” teams end up frustrated with the engineers who ship bugs and this results in added process that delays deploys, arbitrary demands for test coverage, capricious error budgets, etc.
This is poor operations culture. Software is no different from industrial manufacturing. You QA before you ship product to customers and you QA your raw materials before you start to process them. Operations is responsible for catching show-stopper bugs before they hit production. This means that operations is responsible for pushing to staging, not developers; operations stakeholders need to be looped into feature planning to ensure that feature work will easily integrate into the operations culture (somebody's got to tell the developers they can't adopt MySQL if it's a PostgreSQL shop, etc.). Fundamentally, Ops needs to be able to say No to Dev. The SRE take on it is to "hand the pager back to Dev", but the actual method of saying No is different from Ops culture to Ops culture.
> reliability people who tend to be systems experts but lack the business logic intuition to spot errors at a glance
If Dev didn't build the monitoring, the observability, put proper logging in place, etc., then honestly, Dev isn't going to spot the errors at a glance. Customer Service will when customers complain. @jedberg seems to think that Developers should write code to auto-solve their operations issues. If Developers can write code to auto-solve their operations issues, and Developers obviously anyway need to add telemetry etc., then why, pray tell, should it be so unreasonable to expect Developers to be able to succinctly add the kind of telemetry and documentation that explains the business logic, according to an Operations standard, such that Operations can thus keep the system running?
Correct. Throwing software over the wall to "other people" and letting them deal with the problems of running the software is guaranteed to lead to low quality, inefficient processes, or usually both.
I'd argue that timezone is just part of the problem. If you're responsible for a high oncall load, you are subjected to a steady, unpredictable stream of interrupts requiring you to act to minimize downtime or degradation. Obviously it's worse if you get these at night, but it's still bad during the day.
I think the anti-pattern is having one team responsible for another's burden. You want teams to both be responsible for fixing their own systems when they break, AND be empowered to build/fix their broken systems to minimize oncall incidents.
At the end of the day, there's a human cost to responding to pages, and there's a human cost to collaboration.
Both of those can drive burn out. Personally, I find all that collaboration work very hard and stressful, so I work better in a situation where I get pages for the services I control; but that would change if pages were frequent and mostly related to dependencies outside of my control. It also helps to have been working in organizations that prioritize a working service over features. Getting frequent overnight issues that can't be resolved without third party effort that's not going to happen anytime soon is a major problem that I see reports of in threads like this.
I can also get behind a team that can manage the base operations issues like ram/storage/cpu faults on nodes and networking. The runbooks for handling those issues are usually pretty short and don't need much collaboration.
One anti-pattern I've found is that most orgs ask a single team to handle on-call around the clock for their service. This rarely scales well, from a human standpoint. If you're getting paged at 2:00 in the morning on a regular basis you will start to resent it. There's not much you can do about that so long as only one team is responsible for uptime 24/7.
The solution is to hire operations teams globally, and then setup follow-the-sun operations whereby the people being paged are always naturally awake at that hour, and allows them to work normal eight hour shifts. But this requires companies to, gasp, have specialized developers and specialized operators collaborate before allowing new feature work into production, to ensure that the operations teams understand what the services are supposed to do and keep it all online. It requires (oh, the horror!) actually maintaining production standards, runbooks, and other documentation.
So naturally, many orgs would prefer to burn out their engineers instead.