Server Drift

When we’re building a single server it’s easy to click the right buttons and ensure it’s configured just so. But then we start using the server. Requests come in for initially innocuous things. “Can we open up a port for debugging?” “The security is getting in the way. Can we turn it off temporarily?” Then patches get applied, systems get upgraded, and manual adjustments are made. Is the server still in compliance with industry best practices?

This gets much more difficult when we have a fleet of VMs. Spinning up a single machine is cake. But it gets hard when we have a dozen or a hundred. And did we build this one the same as the one we built a month ago?

We need a way to validate that the server’s configuration matches what we expect – even as we evolve our standards or upgrade systems. It can be incredibly time consuming to remote into each server, capture the version and configuration details for each system and package installed on the server, then move onto the next to do the same. Once we’re done, the data is already stale.

Automating this with tools like Ansible only goes so far.  The operator needs to know to look for an issue.  Most of the time, they’ll run the Ansible command only to have it come back clean.  “Should I really run this every day when I’ve gotten no value from it in months? There are so many more urgent tasks,” says the Ops team.  And suddenly it’s been months since we checked.  And then there’s the dreaded ticket from a user complaining of inconsistent system functionality, data corruption, or service outages.  Oops.