I read this paper a few months ago and summarized it for my team. Sharing the notes here in case it's helpful for others:
- when storing time series keys, you can save a lot of space by encoding them as a first timestamp followed by timestamp offsets (deltas)
- when storing time series values, you can save a lot of space by realizing sequential data points tend not to be volatile... e.g. a "writes per minute" series is more likely to be 100, 99, 101, etc. than it is to be 100, 999999, 1, etc. This means you can encode the current value by using XOR tricks with prior values to save space
- the authors suggest using in-memory caching for recent data, but doing eventual persistence to HBase or similar distributed filesystems; thanks to this, you can get real-time operational metrics that are fast and space efficient, yet still have a system that scales out horizontally for historical storage
Those encoding tricks go back to research from the 1960 and even earlier. For example, if you expect the statistical distribution of the delta encoding to produce small values, it can be followed by Rice-Golomb coding [1], which further reduces the number of bits needed for each value.
Good tips, on that note you really don't have to use HBase or some always-on managed filesystem. You can actually use S3! I know this sounds weird as you'd rack up API costs way faster than storage cost, but we implemented a simple batching approach for timeseries data that could handle 100M+ messages for $10/day (over 100GB+ of data, includes all costs, processing, storage, backup). Did a 2 min screen cast here: https://www.youtube.com/watch?v=x_WqBuEA7s8 . This hopefully can be a lifesaver on costs for some people, and keeps your system overall simpler.
This is neat! S3 is always my first choice when it comes to scalable cheap storage. The price and scalability beats almost everything in this category. Reliability is unprecedented as well. I am wondering what was you file format for storing the TS data.
> when storing time series keys, you can save a lot of space by encoding them as a first timestamp followed by timestamp offsets (deltas)
It's actually a delta of deltas, which means for regular time series that a huge portion of rows will contain a 0 (more than 96% according to the paper) because all events arrive at a fixed interval; if there's a slight delay in either direction, the stored value will be small in most cases.
- when storing time series keys, you can save a lot of space by encoding them as a first timestamp followed by timestamp offsets (deltas)
- when storing time series values, you can save a lot of space by realizing sequential data points tend not to be volatile... e.g. a "writes per minute" series is more likely to be 100, 99, 101, etc. than it is to be 100, 999999, 1, etc. This means you can encode the current value by using XOR tricks with prior values to save space
- the authors suggest using in-memory caching for recent data, but doing eventual persistence to HBase or similar distributed filesystems; thanks to this, you can get real-time operational metrics that are fast and space efficient, yet still have a system that scales out horizontally for historical storage
The Morning Paper also did a good analysis here: https://blog.acolyer.org/2016/05/03/gorilla-a-fast-scalable-...