Back to videos

Why We Leverage Wavelets for Data Compression

Wavelets are the best way to deal with errors in the underlying data stream
2 min
play_arrow
Summary

For Shoreline, wavelets are the best way to deal with errors in the underlying data stream. It’s because they allow storing data at a very high resolution.

Here’s how it works:

1. Take a series of values and construct a binary tree where:- the top of the tree is the average- the left-hand side is the difference between the top average and the average of the first half- the same on the right-hand side
2. Keep recursing.
3. Remove all the zeros and minimal values that don't add to the total value.That’s how we get ~30x compression, compared to storing the timestamp and the value at 64 bits, while still getting a very good accuracy for the 4 decimal digits for all metrics.

Wavelets give an energy signature and focus your attention on the parts of the underlying signal that are changing a lot (either up or down).That’s why we prefer it over other compression techniques where you typically consider large spikes as noise to be filtered away.This is why wavelet compression is one of the core underpinning technologies of Shoreline.

Transcript

View more Shoreline videos

Looking for more? View our most recent videos
4 min
Shoreline on Shoreline: Unauthorized Root Access Detector
Hear from Shoreline Op Pack Engineer, Kaustubh Prabhakar, on how valuable it is to use Shoreline Unauthorized Root Access Detector.
2 min
Slack vs. Waste
Waste is when resources are deeply over-provisioned, underutilized, or not utilized at all. Slack appears like the same thing, but you create it with purpose. It's important to understand the difference to drive costs down.
3 min
Decoding Taylor Swift’s Ticketmaster Debacle
What can we learn from the Ticketmaster (Taylor Swift) Debacle? Ticketmaster experienced an unprecedented demand that resulted in their site crashing for many hours. If they had designed a reliable service with an escalator-like system instead of an elevator, this could have been avoided.