Hosted Graphite Blog

Obvious fact alert: New Year's Resolutions don’t work for anyone (this lifehack clickbait has five reasons why). They especially don’t work for companies because if the resolution was really that important you probably would have taken it on already. The January-specific orbital tilt of the earth won’t make you especially free to take new stuff on, and so you won’t.

Rather than pretending you’ll take do extra work (and beating yourself up because you didn’t achieve unachievable goals) it’s never the wrong time to take stock and see what you can eliminate. What’ll let you be more productive at work or even let you be home an extra hour each week?

You could be riding horses instead of fighting Grafana!

In the past six months we in Hosted Graphite have taken this to heart and started farming out small parts of our development stack. We’ve stopped trying to maintain our own continuous integration set up (that's on CircleCI now). Instead of staying on a clunky chat workflow with IRC or an open source chat client, we’ve moved to Slack. These were hard decisions (we like open source!) but these changes allow us to stop building a production line and focus on building a product.

For most other tech companies, Graphite, Grafana, and StatsD are some of these awkward but vital tools. If you’re a small company and trying to justify the costs of Hosted Graphite, calculate how much developer time you spend maintaining your monitoring. Maybe it’s just a few hours a week (congratulations if it is), but even that adds up in a month. For Hosted Graphite the extra tooling that we’ve switched to costs us, for a company of 12, around €100 a month. However the pure value in hours saved and also just de-stressing and speeding up our workflow is worth vastly more than that €100 to us. Hell, the time savings can even pay for themselves directly in more time spent on sales if we want.

“Stop building a production line and focus on building a product”

As we get firmly into a new year we’re trying to figure out what parts of our stack need to be streamlined. It’s becoming less obvious for us as 2015 was a big year in streamlining our processes. We're now working on speeding up our workflows around our user-facing documentation and our blog. A little bit of work on automation or moving to a different solution and the few minutes deployment for those becomes a few seconds. It’s not work done to save those few minutes, it’s because if a deploy is instant we’ll fix more errors in documentation and work more naturally. This allows us to be a faster moving company and our customers will enjoy better documentation and communication from us.

We’ve just launched some new webhooks available to Hosted Graphite. These services allow you to create new events in your Grafana Dashboard when something interesting (or worrying...) happens.

Hosted Graphite webhook integrations are available under the “Data sources” tab.

Continuous Integration services

Create events when your builds run: flag successful or failed builds.

CircleCI 

Codeship 

Website Status 

Need to know when an API you rely on goes down?

StatusPage.io

Website Health Monitoring

Did your site just go down?

Pingdom

PaaS

Flag deployments to your applications

Heroku

AWS Cloudwatch

Our AWS Cloudwatch metrics integration has been given a feature facelift - now you can add multiple AWS Cloudwatch accounts to your Hosted Graphite account and pull metrics from all of them!

Once your accounts are added you can also describe what data you want to pull - you have the option to retrieve only data marked with custom tags.

Google Auth and AzureAD

We’ve also added login via Google Auth and AzureAD - this allows your team to use their google authentication credentials to log in with all the benefits that that offers - two-factor auth and management within the Google or Azure portals.

We’re pleased to announce that we have upgraded to the latest version of Grafana (2.1). 

Building upon the UI changes, new features and structural changes introduced in Grafana 2.0, Grafana 2.1 brings with it a number of small changes which will help you to get even more out of your dashboards.

When we ask our customers what they use to track the uptime (or more importantly downtime) of their sites, Pingdom is typically the answer. As part of an integrated monitoring philosophy, it’s essential that your internal monitoring process can also relate to external checks - combining multiple sources of information gives you a clear picture of your users’ experience.

We’ve added Pingdom as a new data source to Hosted Graphite - simply copy the webhook URL and direct your Pingdom alerts to it and you can view uptime events as annotations on your graphs.

If you like metrics (and we do!) StatsD is a fantastic way to get a wide variety of statistical views of your data, as well as providing the very useful ability to sample your data instead of firing giant streams out over the intertubes. It’s also got wide language support and there are dozens of client libraries to help you get started quickly.

One complaint we hear when running our Hosted StatsD service is that StatsD is a little bit too noisy. If you’re sending a wide range of metrics, the full set of StatsD data views (your percentile data, minimums, maximums etc) ends up producing eight times more metrics than if you just sent plain metric data. Couple that with the fact that Hosted Graphite already captures all of these views for every metric sent and you’re basically just eating up metrics that you probably don’t need. To help with this, we’ve just added filtering for data sent to Hosted StatsD - you can keep the views you want and we won’t store the ones you don’t need.  There’s a simple check-box setup:

We’re pleased to announce our Annotations feature. 

Annotations allow you to record intermittent happenings in your infrastructure - for example deploying a new build to your servers, marking when that long-running reporting task completes, or flagging that nasty outage you just fixed (I hope you don’t need to use that one).

One of the best ways to ensure your whole team knows what your technology is doing is to ensure that everyone has quick and easy access to your dashboards. One of the best ways to do that is to get your dashboards displayed on a giant screen right in the middle of your work area.  Sometimes this can be a bit of a pain - you need to devote some machine to power the dashboard and pipe it to the TV as well as the hassle of setting it up. Enter Screenly!

One element of friction that Hosted Graphite has had for a while is that in order play with a dashboard, you actually need to send data.

While it might seem pretty obvious that you need to have data in a system before you can graph it, we thought that perhaps this was actually a pretty big ask for some users. We’re essentially asking you to give up some time to hook up a service or run a script, wait a minute and then dig into the dashboard options to see if you actually like it enough to use it on anything serious. 

Test Data Streams

Well, we’re sorry  and we’ve added something that will make life a lot easier. We’re now including a test data stream with all accounts, so that as soon as you sign up you can begin testing graphite’s various functions and our advanced data views on live graphs and see whether these options work for you. 

Interactive Grafana Guided Tour

In addition to that, we now have a fully interactive guided tour of Grafana which will walk you through the steps of creating a graph, adding and editing metrics, and playing with some of the advanced features. 

Nassim Taleb is famous for a few things - as the author of the 2007 blockbuster “The Black Swan” he examined the impact of rare high-impact events and how they tend to be explained away in retrospect with pithy or simplistic narratives. He’s also renowned for his reluctance to give interviews, so it’s a pleasure to hear him expound on his theories at length in this recording with James Altucher.

Melting old ladies

Of possible interest to monitoring is his simple explanations of the effect of averages on data, namely in relation to humans. Take the example of an old woman who likes the temperature to be a comfortable 70 degrees Fahrenheit. If you take an average of 70 degrees does it sound reasonable? Let’s examine a possible set of data - if it’s 0 degrees half the time, and 140 degrees the other half we’ve set a successful average of 70 degrees but our little old lady has unfortunately perished by either freezing or being burned to a crisp. Whoops!

The bell curve - or a gaussian curve representing normally distributed data. If your data fits into a normal distribution you can do useful things like anomaly detection* as the normal distribution gives you an idea of how frequently you should see certain values. If we imagine that this somehow represents temperature variation in a different version of our old lady example, we would expect the 0 degrees or 70 degrees events to fall on either side in the “very rare” event category and our old lady would probably be pretty happy within a standard deviation of the mean; the fat bit in the middle of the bell curve which would roughly cover a range of ~50 to 100 degrees. (She might get a little uncomfortable, but is unlikely to melt like the Nazi in Raiders of the Lost Ark.).

Bell curves, averages and systems

So, for a lot of systems applications this turns out to be a poor model to follow - first of all, the data may not be normally distributed but follow something closer to a long tailed distribution. The average of this data set may not represent anything particularly useful. If we take something like the time it takes to process a request on your web server, your average may fall within your SLA criteria but there may be large numbers of outliers pulling data one way or another. 

Percentiles

In situations like this, percentile data is useful. Let’s take our web request timing example again: Viewing data for at the 50th percentile gives you a view of the median user experience - If the 50th percentile (median) of a response time is 750ms that means that 50% of my transactions are either as fast or faster than 750ms - sounds alright depending on what your site is doing. The 90th percentile view of the same data may be around 1250ms, which means that 90% of your requests fall on or under that speed with 10% of all requests completing slower. You may have a 98th+ percentile request that is 1750ms or slower, and this might represent a lesser-used use case such as a reporting function which takes a lot of time.

Using a percentile view of your data, you can see what the typical experience is for your users. A degradation in the 50th percentile from 750ms to 1000ms means that 50% of your requests just had a 25% slower experience and you probably need to start looking into it.

One of the really useful things about Graphite (and maybe the main one if you were going to pick one standout that has led to it’s wide adoption), is that you can just fire a new metric at the collector and Graphite will happily accept it and you get useful graphs.  Add some code to your app, or configure a plugin for collectd or diamond, restart your app and quickly your new metrics appear like magic!

There are two possible issues with this:

With a lack of basic controls, this can also be a problem - if someone commits a chunk of unreviewed code that fires off a rapidly changing or random element in a metric name, you’re going to end up with a whole lot of junk in your system. Add a username to your metric name in a system with a few million unique users? Whoops!

Metric name collisions - if you have more than one server sending any given metric name at the same time, it’s like the movie Highlander (with less Freddie Mercury / lightning effects): THERE CAN BE ONLY ONE!

As Jason Dixon summed it up in a recent post on the Graphite-dev mailing list:

It's assumed that you avoid namespace collisions in each backend cluster. Otherwise, whichever backend returns the query first, "wins”.

Let’s say someone accidentally uses the same metric name in a few different places - picture trying to get useful information from two completely separate sets of data that have been interpolated side-by-side rather than collected together and processed?

Well, that sucks. When building out the backend for Hosted Graphite, we spent a lot of time trying to figure out the best and worst parts of Graphite so we can focus on the good and eliminate or mitigate the bad.  In the usual love-hate relationship that people have with Graphite the fast metric creation is great, and collisions are just sort of an annoying behaviour.

In our setup, we have control over where we collect from - and have removed any issue with metric collisions. We’re greedy! If you send the same metric name from multiple servers, we collect all the data. By default we display you the average, but we also collect all the data points and give you a true sum, minimum, and maximum as well as a few other more exotic views like a random sampling of data to be used for percentile data.

We've just added some additional services to our AWS integration to automatically pull data for:

AWS Billing

AWS Cloudfront

AWS Redshift

DynamoDB

Route53

SQS

These can be found on the integrations tab of the app at https://www.hostedgraphite.com.

By Trevor Parsons, PhD, Chief Scientist & Co-founder, Logentries

As one of the founders of Logentries, over the past few years I have had the chance to meet thousands of our users – at tech talks, meet-ups, trade shows or where ever takes your fancy . As a product focused company we believe it’s important to spend a lot of time out in the community listening to what people have to say, which tools they are using, what technologies they are running on and, most importantly, what problems they are facing when trying to understand their systems.

One consistent murmur (or maybe more of a chant...) we continue to hear is “Graphite, graphite, graphite…”

For anyone who doesn’t know, Graphite is a widely used, near real-time metrics-gathering system. It is written in Python, and allows for metrics to be captured from multiple systems, providing the ability to take a high level aggregated view of your data or to drive down to a single source of data.

Although typically used to capture information like CPU or memory use, Graphite is completely agnostic about the nature of the data being stored in it and we consistently talk to people who also use it for application performance metrics, and in some cases business related data.

In fact, a very common use case for graphite is to consume metrics extracted from your log data. For example, Logstash can parse metrics out of log files and send them to Graphite where they can be stored and visualized.

As big believers in allowing people to concentrate on what they do best – which is usually developing and managing their systems, rather than building their own logging framework or monitoring infrastructure – I was excited when I first came across Hosted Graphite.

I like to think that Logentries and Hosted Graphite are Kindred Spirits. Like Logentries, Hosted graphite is all about taking the pain away from you having to roll your own monitoring solution, by providing a hosted version of Graphite. While this can be tempting when dealing with small amounts of metrics or logs, as your volumes grow, rolling your own solution becomes time consuming and costly. My advice is generally ‘get off the complexity elevator before its too late’… do you really want to be one of those guys/gals saying, “We’d love to use a SaaS service for this, but we have simply invested too much time and resources in our own solutions…”

To make this an even easier decision, Logentries and Hosted Graphite now allow you to easily grab interesting metrics from your logs and to visualize them in your hosted graphite account through our new integration.

How does this work, you ask?

It’s dead simple. Logentries allows you to extract key metrics from your logs, for example, response time, http status codes, website form submissions, transaction values, or whichever metrics that exist in your logs that you deem important.

Once you identify these in your logs, Logentries can automatically extract them and roll them up into metrics dashboards, providing you with: ‘Average response time over the past day’; ‘A break down of http status codes’; ‘Total number of sign-ups per day’; and ‘The total number of daily transactions’ as part of your Logentries dashboard.

Example Logentries Dashboard

The Logentries and Hosted Graphite integration allows you to take any metrics that Logentries extracts from your logs and then visualize these in your Hosted Graphite account.

Export Your Log Metrics to Hosted Graphite

The advantage here is that you are likely sending a bunch of other valuable data into Hosted Graphite that may never make it to your logs, so it gives you a single point of correlation across all your data sources.

Logentries Metrics in Hosted Graphite

We've been collecting feedback from our customers on the bits of Hosted Graphite that need improvement.  Overwhelmingly the answer was like the old grade school report card that our interface "was not working to potential".  

Ok, we were listening! We've been hard at work developing a completely new interface for Hosted Graphite. Ww feel like we've put together a far more intuitive layout that leaves us plenty of room for the new ideas we're launching.

What'd we change?

New tasty-looking UI! Easier on the eyes, fewer calories and packed with vitamins.

Improved Team Sharing - Invited users flow straight into the team account directly, no more unintuitive 'impersonation' process.

Metric Expiry - The ability to control when you want to eliminate stale metrics.

New Integrations - Stay tuned!

We're finally releasing Grafana dashboards to all plans. Thanks to all our beta testers for their great feedback!

Hosted Graphite has been available as a Heroku add-on for some time, but we've just added a new way to get system information out of your Dynos instantly - automated system dashboards. 

Using Heroku's log-runtime-metrics we retrieve and process Heroku's syslog information for your Dynos including:

Load average - 1 / 5 / 15 minute averages

Memory and Swap - Resident memory, disk cache, swap, total memory, and cumulative totals for pages written to and read from disk.

HTTP metrics - The number of requests broken down by HTTP method and status codes, data transferred, and connect and service times.

From this data, we create automatic Grafana dashboards giving you a great high-level overview of what's happening in your Heroku apps.

It looks a little something like this:

The real power in this approach is when you combine it with the application metrics specific to your app that Hosted Graphite brings. This way, you get a detailed view of your app, and also the affect that your code has on the underlying Dynos that it runs on. Benchmark that new function that you're pretty sure is good to drop into production and then make 100% sure by seeing what it does to the memory and load average on the box.

The end result is both native and custom application metrics in one place as well as all the advanced data types you expect from Hosted Graphite - min/max/sum/averages, and arbitrary percentile data. 

How do I get it?

New Hosted Graphite Heroku users won't need to do anything; we'll start pulling metric data from your syslog data once you add the Hosted Graphite app.

For existing users you can add us as a syslog drain using the Heroku drains command, and we will start pulling this data and graph it for you.

To enable the drain, use the following command:

heroku drains:add https://webhooks.hostedgraphite.com/heroku/logs/?apikey=<your api key here>

For more details on how this works, see Heroku's documentation.