Networking Issues

Typically, Shoreline does not trigger an automated repair for this type of incident. Instead, Shoreline provides a series of diagnostics that help on-call teams more quickly pin-point the specific network issue and nodes affected by the issue. These diagnostics eliminate hours of wasted time that operators would otherwise spend trying to manually uncover the issue. Here are the diagnostics run by Shoreline:

1. Curl an HTTP endpoint in parallel across the fleet and return a status code. Way to check if services your system is depending on, can each instance of your application connect and authenticate to the service.
2. DNS lookup. Checking to see if each instance of our application resolves domains to IP addresses in the same way. Sometimes a portion of the fleet might have stale entries and therefore might have failed requests.
3. Ping. Used to check connectivity at the network layer. Can the nodes in one region or availability zone connect to nodes in another region. Sometimes you don’t have connectivity, sometimes there is high latency and other times there is high packet loss.
4. Measure the number of outbound requests to a specific port. Can help you detect if you are connecting to APIs or ports that are unexpected. Sometimes there are too many processes running, generating an unexpected number of connections.
5. Measure the number of inbound requests to a specific port. Can help you detect if you are receiving an unexpected number of connections or API calls from external sources.