Analyze and act upon remediation incidents with Shoreline Events

Shoreline makes it easy to automate fleet-wide remediation by creating an Alarm associated with your infrastructure Resources -- the Alarm fires when an incident occurs, which executes an associated remediation Action. Monitoring everything Shoreline does during this remediation process is accessible via the graphical web UI. Still, we knew it was critical to provide access to the same data via the CLI.

We're excited to announce the release of Shoreline Events, which allows you to use the Op CLI to evaluate the status of every Alarm, Action, Bot, and Resource within your infrastructure. Every part of the remediation loop is tracked and associated with an Event, giving you the ability to execute many Op commands to filter and act upon the results.

Event Types

Shoreline generates Events anytime a core object is created or modified.

• Alarm Events - Created whenever an Alarm triggers, resolves, or ends.
• Action Events - Created whenever an Action executes, completes, or fails.
• Bot Events - Created whenever a Bot executes, completes, or fails.
• Resource Events - Created whenever a Resource registers or de-registers.

For example, use the <code-block>events | type="alarm"<code-block> command to get all Alarm Events:

<code-embed>op> events | type="alarm"

RESOURCE_NAME              | RESOURCE_TYPE | ALARM_NAME                           | STATUS    | STEP_TYPE   | TIMESTAMP
i-0192cd8c4f55219c2        | HOST          | JSON_Metric_auto_name                | triggered |             |
                          |               |                                      |           | ALARM_FIRE  | 2021-08-31T14:47:20-07:00
                          |               |                                      |           | ALARM_CLEAR | 1969-12-31T16:00:00-08:00
test2-cust.shoreline-l2jf7 | POD           | simple_pod_alarm                     | resolved  |             |
                          |               |                                      |           | ALARM_FIRE  | 2021-08-30T11:05:44-07:00
                          |               |                                      |           | ALARM_CLEAR | 2021-08-30T11:06:02-07:00
test2-cust.shoreline-l2jf7 | POD           | mixed_metric_linux_cmd_alarm         | resolved  |             |
                          |               |                                      |           | ALARM_FIRE  | 2021-08-30T11:05:44-07:00
                          |               |                                      |           | ALARM_CLEAR | 2021-08-30T11:06:02-07:00
test2-cust.shoreline-l2jf7 | POD           | linux_cmd_duration_res_env_var_alarm | resolved  |             |
                          |               |                                      |           | ALARM_FIRE  | 2021-08-30T11:05:44-07:00
                          |               |                                      |           | ALARM_CLEAR | 2021-08-30T11:06:02-07:00
i-00f4a29cc5a8716d1        | HOST          | always_fire_alarm                    | resolved  |             |
                          |               |                                      |           | ALARM_FIRE  | 2021-08-30T11:05:43-07:00
                          |               |                                      |           | ALARM_CLEAR | 2021-08-30T11:05:52-07:00<code-embed>

Filter Events

You can filter Event records using many of the same Op commands that power the rest of the platform. Check out some simple examples below.

• Count the number of Alarm Events

<code-embed>op> events | type="alarm" | count

RESOURCE_TYPE | RESOURCE_NAME                        | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
             |                                      | ALARMS     | 54    | 54      | 0      | 54
             |                                      | ALARMS     | 16    | 16      | 0      | 16
POD           | test5-cust.shoreline-nvbdm           |            |       |         |        |
             |                                      | ALARMS     | 1     | 1       | 0      | 1
CONTAINER     | test5-cust.shoreline-729bc.shoreline |            |       |         |        |
             |                                      | ALARMS     | 16    | 16      | 0      | 16
POD           | test5-cust.shoreline-729bc           |            |       |         |        |
             |                                      | ALARMS     | 1     | 1       | 0      | 1
HOST          | i-0d991c45ccdb96db4                  |            |       |         |        |
             |                                      | ALARMS     | 1     | 1       | 0      | 1
CONTAINER     | test5-cust.shoreline-wtjfd.shoreline |            |       |         |        |
             |                                      | ALARMS     | 16    | 16      | 0      | 16
POD           | test5-cust.shoreline-wtjfd           |            |       |         |        |
             |                                      | ALARMS     | 3     | 3       | 0      | 3
CONTAINER     | test5-cust.shoreline-nvbdm.shoreline |            |       |         |        |<code-embed>

• Get Events by status

For example, this command gets the first five Alarm Events that fired and then resolved:

<code-embed>op> events | type="alarm" | status="resolved" | limit=5

RESOURCE_NAME              | RESOURCE_TYPE | ALARM_NAME                     | STATUS   | STEP_TYPE   | TIMESTAMP
test4-cust.shoreline-9qm7j | POD           | simple_pod_alarm               | resolved |             |
                          |               |                                |          | ALARM_FIRE  | 2021-09-01T15:44:07-07:00
                          |               |                                |          | ALARM_CLEAR | 2021-09-01T15:44:36-07:00
test4-cust.shoreline-9qm7j | POD           | mixed_metric_linux_cmd_alarm   | resolved |             |
                          |               |                                |          | ALARM_FIRE  | 2021-09-01T15:43:07-07:00
                          |               |                                |          | ALARM_CLEAR | 2021-09-01T15:44:36-07:00
test4-cust.shoreline-9qm7j | POD           | linux_cmd_duration_res_env_var | resolved |             |
                          |               |                                |          | ALARM_FIRE  | 2021-09-01T15:43:07-07:00
                          |               |                                |          | ALARM_CLEAR | 2021-09-01T15:44:36-07:00
test4-cust.shoreline-zrwhf | POD           | mixed_metric_linux_cmd_alarm   | resolved |             |
                          |               |                                |          | ALARM_FIRE  | 2021-09-01T15:43:00-07:00
                          |               |                                |          | ALARM_CLEAR | 2021-09-01T15:43:18-07:00
test4-cust.shoreline-zrwhf | POD           | linux_cmd_duration_res_env_var | resolved |             |
                          |               |                                |          | ALARM_FIRE  | 2021-09-01T15:43:00-07:00
                          |               |                                |          | ALARM_CLEAR | 2021-09-01T15:43:18-07:00<code-embed>

• Get Events associated with specific Resources

<code-embed>op> pods | name=~"shoreline-mxxhw" | events | type="bot" | count

RESOURCE_NAME              | EVENT_TYPE | TRIGGERED | FINISHED | ACTIVE | TOTAL_COUNT
                          | BOTS       | 18        | 18       | 0      | 18
                          | BOTS       | 18        | 18       | 0      | 18
test4-cust.shoreline-mxxhw |            |           |          |        |<code-embed>

• Get Events by tags

For example, the following finds Alarm Events for Resources within either the <code-block>us-west-2a<code-block> or <code-block>us-west-2b<code-block> availability zones.

<code-embed>op> events | type="alarm" | az=["us-west-2a", "us-west-2b"] | count

RESOURCE_TYPE | RESOURCE_NAME       | AZ         | EVENT_TYPE | FIRED | CLEARED | ACTIVE | TOTAL_COUNT
             |                     |            | ALARMS     | 99    | 93      | 6      | 99
             |                     |            | ALARMS     | 4     | 3       | 1      | 4
HOST          | i-0777445c24c2b8344 | us-west-2b |            |       |         |        |
             |                     |            | ALARMS     | 5     | 4       | 1      | 5
HOST          | i-0627f066048c1b3d9 | us-west-2a |            |       |         |        |
             |                     |            | ALARMS     | 6     | 5       | 1      | 6
HOST          | i-00f4a29cc5a8716d1 | us-west-2b |            |       |         |        |
             |                     |            | ALARMS     | 4     | 3       | 1      | 4
HOST          | i-0192cd8c4f55219c2 | us-west-2a |            |       |         |        |<code-embed>

Check out the Filter Events documentation for more details.

Filter Resources by Events

Alter the Resources returned by a Resource query based on associated Events. This capability is particularly potent when piped to additional Op commands.

To illustrate, consider the following Op statement:

<code-embed>op> hosts | filter((events | type="alarm" | name="my_cpu_alarm" | status="triggered" | count) >= 3) | `top -b -n 1 | head -n 15`<code-embed>

This statement goes through the following steps:

• Get all host Resources
• Filter down hosts to those that <code-block>triggered<code-block> the <code-block>my_cpu_alarm<code-block> Alarm at least three times
• Execute a custom Linux command to output the current processes

Although we've just scratched the surface of what it can do, the new Events command allows you to use the CLI to determine the status of everything that Shoreline is remediating on your behalf.

Head over to the Events documentation for more details and examples. If you're ready to jump in and see what Shoreline can do, please click here to request a demo.