#opentsdb

By Chris Larsen, Architect

OpenTSDB is one of the first dedicated open source time series databases built on top of Apache HBase and the Hadoop Distributed File System. Today, we are proud to share that version 2.4.0 is now available and has many new features developed in-house and with contributions from the open source community. This release would not have been possible without support from our monitoring team, the Hadoop and HBase developers, as well as contributors from other companies like Salesforce, Alibaba, JD.com, Arista and more. Thank you to everyone who contributed to this release!

A few of the exciting new features include:

Rollup and Pre-Aggregation Storage

As time series data grows, storing the original measurements becomes expensive. Particularly in the case of monitoring workflows, users rarely care about last years’ high fidelity data. It’s more efficient to store lower resolution “rollups” for longer periods, discarding the original high-resolution data. OpenTSDB now supports storing and querying such data so that the raw data can expire from HBase or Bigtable, and the rollups can stick around longer. Querying for long time ranges will read from the lower resolution data, fetching fewer data points and speeding up queries.

Likewise, when a user wants to query tens of thousands of time series grouped by, for example, data centers, the TSD will have to fetch and process a significant amount of data, making queries painfully slow. To improve query speed, pre-aggregated data can be stored and queried to fetch much less data at query time, while still retaining the raw data. We have an Apache Storm pipeline that computes these rollups and pre-aggregates, and we intend to open source that code in 2019. For more details, please visit http://opentsdb.net/docs/build/html/user_guide/rollups.html.

Histograms and Sketches

When monitoring or performing data analysis, users often like to explore percentiles of their measurements, such as the 99.9th percentile of website request latency to detect issues and determine what consumers are experiencing. Popular metrics collection libraries will happily report percentiles for the data they collect. Yet while querying for the original percentile data for a single time series is useful, trying to query and combine the data from multiple series is mathematically incorrect, leading to errant observations and problems. For example, if you want the 99.9th percentile of latency in a particular region, you can’t just sum or recompute the 99.9th of the 99.9th percentile.

To solve this issue, we needed a complex data structure that can be combined to calculate an accurate percentile. One such structure that has existed for a long time is the bucketed histogram, where measurements are sliced into value ranges and each range maintains a count of measurements that fall into that bucket. These buckets can be sized based on the required accuracy and the counts from multiple sources (sharing the same bucket ranges) combined to compute an accurate percentile.

Bucketed histograms can be expensive to store for highly accurate data, as many buckets and counts are required. Additionally, many measurements don’t have to be perfectly accurate but they should be precise. Thus another class of algorithms could be used to approximate the data via sampling and provide highly precise data with a fixed interval. Data scientists at Yahoo (now part of Oath) implemented a great Java library called Data Sketches that implements the Stochastic Streaming Algorithms to reduce the amount of data stored for high-throughput services. Sketches have been a huge help for the OLAP storage system Druid (also sponsored by Oath) and Bullet, Oath’s open source real-time data query engine.

The latest TSDB version supports bucketed histograms, Data Sketches, and T-Digests.

Some additional features include:

HBase Date Tiered Compaction support to improve storage efficiency.

A new authentication plugin interface to support enterprise use cases.

An interface to support fetching data directly from Bigtable or HBase rows using a search index such as ElasticSearch. This improves queries for small subsets of high cardinality data and we’re working on open sourcing our code for the ES schema.

Greater UID cache controls and an optional LRU implementation to reduce the amount of JVM heap allocated to UID to string mappings.

Configurable query size and time limits to avoid OOMing a JVM with large queries.

Try the releases on GitHub and let us know of any issues you run into by posting on GitHub issues or the OpenTSDB Forum. Your feedback is appreciated!

OpenTSDB 3.0

Additionally, we’ve started on 3.0, which is a rewrite that will support a slew of new features including:

Querying and analyzing data from the plethora of new time series stores.

A fully configurable query graph that allows for complex queries OpenTSDB 1x and 2x couldn’t support.

Streaming results to improve the user experience and avoid overwhelming a single query node.

Advanced analytics including support for time series forecasting with Yahoo’s EGADs library.

The more monitoring you have, the more trouble you find. If you don’t have monitoring, trouble finds you.

— John Arundel (@bitfield)

January 29, 2015

This tweet just arrived after we have deployed our new monitoring system and I couldn't agree more. In a few words it explains why we and everyone trying to push their limits in monitoring keep doing it.

About 3 years ago when we started working on a number of production servers, one of the first missing things was a way to see which machine does what and being able to do so without actually sshing into the machine and running some command-line tools. After some rationalization (which i'll not be going into) and research (non of which i've conducted), we have deployed zabbix. At first, a server storing data on mysql and daemons on individual machines, pumping metrics to the server. Then we have added several advanced tools into the mix, moved storage to postgresql, developed lots of customizations, made and unmade lots of screens. Currently it still is in our TVs, our browser tabs, still sending mails when something that requires attention happens.

We have tried several other things in those past years but their benefits never outweigh the burden of another adventure of an infrastructure size deployment effort. Until we decided that we need much more metrics per machine/service (see the tweet) and we should be able to correlate wildly different metrics much more easier. None of which are zabbix's strong suit. That's when we started looking for alternatives, more specifically time-series databases. I tried to evaluate some of them but i won't be going into the discussion. We chose opentsdb because we already have HBase installed and grafana provides a fantastic UI.

But a datastore and a means to graph over the data stored is only half a monitoring solution. Next, we need a way to get the data from the machines to the store. With Zabbix, as a complete monitoring solution, this is done with zabbix-agent. This process collects system metric values from various places and communicates those to the zabbix-server. While it may be possible to tap into this communication and tee the data elsewhere, it probably requires a lot of zabbix protocol knowledge which i don't have. So i started looking for other options.

First one was tcollector, side-project of opentsdb, a series of python scripts reading values from kernel/service and sending them to a TSD. With some installation effort, it works great. Its easily extendable with extra scripts, not much overhead. But it falls short of a hidden requirement: While it's a metric value collection daemon, the only place you can send those values is opentsdb. I didn't want another collector daemon running on the machines exclusively for one datastore. The core requirement is to collect values, sending those values somewhere should be pluggable.

Which gets me to my second option: collectd. It's an independent daemon with tons of plugins which are for either collecting values or sending them to somewhere. Its written in C, performant, widely supported, etc. etc. But... I really don't like installing ttf-dejavu-extra or fontconfig just to be able to collect some kernel-level metrics from a headless server. What i mean is, it took some time for me to find a good way to install it with some TSDB plugin and without all of the RRD stuff. That's definitely not collectd's fault by the way. It's probably just me being unnecessarily uneasy. Anyways, what i couldn't find was a good way to install it with a good working TSDB plugin. Long story short, C is not my strong suit so i couldn't get much further.

Last option was a complete coincidence. I started googling from scratch and found some other monitoring tool recommending diamond as a collector daemon (i don't remember who). On the other hand ceph monitoring-management toolchain calamari uses diamond too, so we'd have a way to collect ceph metrics in a known way. After some reading through the docs and the code, it was clear that it is the collector of choice. Diamond offers the best of two previous worlds i ventured: It's composed of easily extendable/modifiable python scripts while being completely datastore agnostic. There are lots of collectors as well as handlers (datastore bindings) with sensible dependencies and a clean way to package & install. I deployed it on a select few servers to for starters.

Diamond is so datastore agnostic that metric names are not opentsdb-y. OpenTSDB naming schema recommends keeping metric names abstract, and setting anything machine/part specific as tags. So when diamond says metric=servers.server-03.cpu.cpu2.user, opentsdb should hear metric=cpu.user host=server-03,core=cpu2. TSDB handler for diamond handles the hostname-as-tag conversion, but the rest is defined in collectors so it does not go into the detail. Another problem was handler not reading TSD server replies off telnet connection, causing messages to pile up at socket queues to the point where TSD server crashes. Adding a few more requirements to the mix, it was a better idea to write a brand new TSDB handler. After a few hours of work, our diamonds now speak OpenTSDB 2 HTTP API, can use more than one TSD server and can be configured to parse tags from metrics. And they log opentsdb error messages so that we can see when we missed a metric or something. It's running without problems for about 3 weeks now and we already started to find more troubles, which was the point. Only thing i'm missing is the ability to do simple mathematical operations directly on opentsdb. One last piece in this new monitoring puzzle is alarms. I'm thinking riemann but haven't had enough time to actually start working on it.

To wrap it all up, zabbix is a great way to start monitoring. It's an opinionated tool which gets results fast when you have nothing. Because, to be honest, you don't know what to monitor unless you start monitoring. Kernels and services throw zillions of numbers at you and unless you have years of expertise it's not easy to find which numbers matter most. Zabbix templates tell you which numbers to collect, how you should graph them and even what are the normal values so that it can immediately start sending alarms when some problem happens. But not everything can be expressed in templates, and even if it was possible, zabbix-server and/or the relational datastore it sits on top has limits. That's when you need to leave the comfort zone of complete solutions and step into the flexibility of collecting, storing, graphing and alerting with specialized systems. You have to find which values you need eyes on, which low-level metric effects which part of the running service. Fear not, it's fun when you have such great tools.

How-to: What is the difference between OpenTSDB and Graphite? #programming #dev #solution

What is the difference between OpenTSDB and Graphite?

As far as I can tell, here are the main differences:

OpenTSDB does not deteriorate data over time, unlike Graphite where the size of the database is pre-determined.

OpenTSDB can store metrics per second, as opposed to Graphite which has minute intervals (I’m not sure of this, Graphite docs show retention policies which stores metrics every…

Machine learning scientists don't use time series?

After a deep discussion with GraphLab architects I was gobsmacked to realize ML scientists don't use time. They hold the notion that introducing real-time feedback will destabilize the system. I call utter bullshit. Of course the system will destabilize if you don't apply control theory to it.

Thus

ML = f(x) where x is "big data" and batch (not a time series) 

vs

RT = f(x) + f(t) where x is small data and t is time series (will adapt)

Another cop out is we can't scale ML on a cluster because we'll have multiple models (one in each cluster). Again hogwash, sharded models can be tested in real-time for stability and pushed out in near real-time to the shards (github meets devops). How do ML data scientists think the Israelis do the Iron Dome? 

The RAF sent the Dam Busters to aim and fire Barnes Wallis bouncing bomb because no amount of "big data" can correct the algorithm without real-time feedback.

return put responses when present

support png format for TSD::Format.query

require put tags with a default of host: