Back to videos

How to Safely Fix Issues Without Escalation

The only real solution is incident automation.
1 min
play_arrow
Summary

Unpopular opinion on fixing issues:

Investing in DevOps tools to detect incidents and hiring qualified engineers to fix them is not the right way to do it.

Why do I see that as a problem?

I believe that simply detecting issues and hiring qualified people is necessary but not sufficient.

Let’s think about a four-nines SLO – that’s 4.4 minutes a month.

Every  issue will break your SLO, as it might take:
- 10-15 minutes on average to get an issue to somebody, and then
- up to an hour to fix it.

So there's no chance you can do anything except dissatisfy people whenever you have an issue.

The only real solution to that is automation.

That’s why I started Shoreline to build incident automation to help people:
- automatically fix issues in production
- grow the number of people who can safely fix things without escalation

For issues that require human judgment, the first person who looks at the issue should be able to resolve it rather than routing it through 6 other people.

And having a system take care of the mundane work is way better for me than dealing with something the 5th or 25th time.

Because, like everyone, I like sleeping at night.

Transcript

View more Shoreline videos

Looking for more? View our most recent videos
1 min
Shoreline Customer Spotlight: TigerGraph
Automating mundane tasks and debugging were just a few of the DevOps requirements TigerGraph VP of Product and Innovation, Dr. Jay Yu, needed to scale in the cloud with his small team. Shoreline delivered.
3 min
How to Reduce Alarm Noise
In any company, 50-80% of the alarms are noisy. Employees get trained to snooze these alarms – which isn’t always the right thing to do. Wouldn't it be better if you could easily see which are your top issues each week, and which alarms might be set incorrectly?
2 min
Debugging a Fleet as Easily as an Individual Box
Underneath the covers, the underpinning technology is a lot like a parallel SQL database.