Graphdat

Six months ago Graphdat released Smart Alarms.  We gave you the ability to filter out the noise and minimize the false positives.  By selecting the time interval and the aggregate (sum, min, max, avg), our alarms became more powerful, but they were limited to only sending an email.  While many services will let you action data off of an email (nice job Kevin), it can be limiting.

So today, we announce the release of Graphdat Actions.

Graphdat Actions allow you add multiple actions to an alarm.  Now when an alarm triggers you can do anything:

send an email

post to your chatroom (campfire, flowdock, hipchat)

trigger an issue with PagerDuty,

call a URL to add a new server to your infrastructure

call a URL to restart an application,

call a URL to turn on your coffee machine

We have hand picked the initial list of actions based on your feedback, if there are more you would like to see, let us know. 

Being notified when something happens is fantastic.  Opening and Resolving incidents in PagerDuty is great, but what if we could resolve the issue automatically?  When an alarm triggers, let action it.

Instead of dreading that 2am email, how about we resolve it.

In this example, we will create an alarm on the memory usage on a server, and if it goes beyond a certain threshold, we'll restart the service.

Open the Settings dialog and select the Actions tab.  We do not have any actions installed yet, so lets create one.  Click on the New Action button.

We see a list of the available actions.  You can click on the action to read more about it, or click Add to configure an action of that type for your account.  We want to add the URL action in this case, as we want Graphdat to call us when something happens, so click on that one.

The URL action takes in a couple of parameters, so lets go through them.

Name: what should we call this?  This will show up in the alarms dialog, so we want something descriptive.

Method: what HTTP Method should Graphdat use when calling our server?  This example uses a POST, so Graphdat will send some helpful data in the body of the POST so we can adjust what action our server should take

Protocol: HTTP or HTTPS?  If you have a certificate, use HTTPS.

URL: how do we contact your server?  Make sure to include the port and path in the URL.  HTTP will default to 80 and HTTPS will default to 443, if they are different, make sure you set them.

Username: does the URL require authentication?  In this example our URL will check for the username and password.

Password: same as above

POST data: Are there any key values pairs that you want to add to the request?  Enter these here key=value, one per line.

Enter in all of the values and click Save.

Now that we have an Action, we need to attach it to something.  Lets create an Alarm that look at the Memory Usage on a server.  When the memory usage goes above a certain threshold, restart the application.

Click on the Alarms tab on the left and then Add alarm.

We want to alarm the Memory Usage, so we select Memory Usage from the drop down, enter in our values and pick the new Restart My Service action from the drop down and hit Save.

With my new Alarm and Action in place, we need something on the server to listen to that POST.  Below is a simple webserver in node.js that listens for the POST and runs a command.  In this case, it's restarting a service called 'graphdat' using supervisord  

https://gist.github.com/codemoran/7921146

With the server up and running:

$ node restart-graphdat-on-the-local-machine.js

We can see the service is up and running.

Now how do we test that this works?  We can either spike the memory on the server, or we can edit the alarm to a very low value so we know that it will trigger.  That sounds much better.  Lets set the alarm to a 1 second period and to a threshold of 1%.

With the alarm saved, we can see that it triggers a couple of seconds later.

If we have a look in our console, we can now see that the service was restarted.

Put the Alarm back to it original settings.

The alarm is resolved

And the service ignores the resolution because we do not want to restart the service when everything is up and running.

With Graphdat Plugins + Graphdat Alarms + Graphdat Actions, you can be notified of when things are happening on your server infrastructure and in your applications, and now, you can do something about it.

One of the most common comments we get is, "really like your real time graphs, can I use Graphdat to monitor X", where X is some popular middleware stack or other back end technology.  Not that long ago we published an API so that you could send any data you desired to Graphdat for real time graphing.  This is great but requires a bit of coding to get the results you would like.  

Wouldn't it be great if there was a shared community of code packages that could be added to Graphdat for graphing the most popular (or even some not so popular) sources of real time stats?

Introducing Graphdat Plugins!

Graphdat Plugins is an open source repository of code bundles that are built on top of the Graphdat API to provide graphing of whatever you can think of, in real time.  As of this introduction the following plugins are available:

NGINX

Apache

MEMCACHED

Redis

HA-Proxy

DiskUse-Summary

Multi-Core

...and more are being continuously added.  Anyone can write their own plugins either to add to the public repository or just for their own use. Each plugin provides a new set of graphs which can be added to existing dashboards or can even include new dashboards of their own.

Plugins are easy to add to your account.  Let's add one now.

Step 1 - Select the plugin

Choose Settings from the side menu then Plugins.  The click Get Plugins.

You will now see a list of plugins currently in the public repository.  For this walk through, go ahead and select the diskuse_summary plugin.

When a plugin installs, it adds the following to your account:

Additional metrics.  You can view these from the Metrics tab in Settings.  You can also add these metrics as graphs to existing dashboards using the Dashboard Editor from the main side menu.

Additional dashboards (optional).  If a plugin adds many new metrics the author of the plugin may choose to provide one or more additional dashboards that present these new metric graphs. These new dashboards will be available from the dashboard drop down list in the upper left of your dashboard.

After clicking on install you would have seen a message stating which of the above have just been installed.  Click OK to continue.

Installing a plugin into your account is a one-time process.  After this you must choose one or more places to put this plugin in your environment and set the configuration parameters are needed by the plugin.  The plugin is added to your environment using a graphdat-relay.

Step 2 -  Add a relay

Ok, what's a relay?  In order for plugins to operate in your environment you need to install a graphdat-relay.  The relay works like a harness for the plugin, keeps the plugin running, configures the plugin, and sends output from the plugin back to Graphdat for graphing.

Installing is a simple process that is described in the Graphdat relay documentation.

Once the relay is installed it should automatically appear in the Relays section under your Settings tab in the menu.

Step 3 - Add the plugin to the relay

Go to the Relays tab in your Settings dialog and click on the new relay you have just setup above.

On the next panel you will be told that there are no plugins currently running on this relay so let's pick the Plugin we just installed from the dropdown list.

Plugins can have arbitrary configurations, it just depends on the plugin author's requirements.  If some configuration parameters are needed for this plugin a panel now appears where you can enter this information. For diskuse_summary, a list of device root paths is required.  For this walk through let's just add a single root device by entering '/'.

That's it!  Your plugin is now configured.  You can return to this configuration panel at any time to change these settings as needed.

The last panel you will now see is the relay output console.  This gives you a remote view of what the relay is doing.  Wait a few moments now and you should see the new configuration you just saved being passed to the relay, then the relay starting up the diskuse_summary plugin.

At this point your plugin is now running and sending data back to Graphdat.  Now let's get some of this data onto your dashboard.

Step 4 - Add metric(s) to dashboard

In the case that a plugin comes with it's own dashboard you would have simply been able able to select the new dashboard from the dashboard drop down in the upper left.  However, in the case of diskuse_summary it does not add a new dashboard, it simply gives you a new metric which you can place on your own dashboard  (note: even with plugins that provide addition dashboards you can still put the new metrics on your own dashboards as desired).  

To add the metric to your dashboard first open the Dashboard editor from the side menu (notice the new 'Disk use' metric in the toolbox) then increase the number of columns on the dash to make room for the new graph.

You should see some new cells open up on the right side of the dashboard.  Next we just drag and drop the new Disk use metric into one of the new cells you just created.

The new metric should drop into place (note: for now we will leave the cell below the new graph empty but in practice you would probably want to re-arrange or resize some of the other graphs to balance out the look of your dashboard but we will leave this exercise to you).

Now just click Save, the dashboard will refresh, and presto!  Your new data is now being graphed on your dashboard.

That's all there is to it!  Because this is the first time setting up your relay and the initial install of your plugin there were a few more steps but now you can easily add more plugins to your relay, change your dashboard layout, or configure your plugin easily with minimal steps.

Background

Graphdat has recently opened up its API to provide great looking dashboards for anything you wish to graph.  With this new API you can now graph any measurable metric in realtime, however, there is a small amount of work involved in getting the metrics you are measuring up to Graphdat for display.  To help solve this problem we introduce the Graphdat Plugin.

A plugin is a tiny application that can be written in any language or script which simply reads the metric you are after and reports its value.  The plugin is launched using a tool called graphdat-relay which manages configuration, polling, and communication with Graphdat.

Plugins not only make it very easy to add arbitrary metric monitoring to Graphdat but because plugins are kept in a central, public, repository, they can be shared by all.  So if there is some metric in your system that needs to be tracked, chances are there is a plugin already written for it.

Preparation

Before starting we will make a few assumptions regarding the following:

A Graphdat account - it is assumed you've already created your Graphdat account.

NodeJS - Although plugins can be written in any language you desire (you specify a command line to run the plugin which can call any program or script engine), for the purposes of this step-by-step we will use the NodeJS language so it is assumed you have NodeJS installed.

A GitHub account - Because all Graphdat plugins use github as a repository you should make sure you have a github account (there is no charge to signup and create public repositories).

SCM tool - Github usually works in conjunction with a source code management tool installed on your system.  For the purposes of this step-by-step we will assume this tool is git but the commands are so simple that you should not have any problem translating these steps for whichever tool you are using.

Step 1 - Create the metrics

The first step in creating any plugin is deciding what metrics you want to track.  In Graphdat the idea is that when a type of metric is created, it is given a universal name and is typically associated with a single plugin.

For the purposes of this step-by-step let's create a plugin that will track CPU usage of individual CPU cores.  The default CPU metric that the graphdat agent collects is only the system wide average CPU so with this plugin we can now dive into individual CPU core usage information. 

Creating the metric is simple.  Log in to Graphdat then open your Settings dialog..

..then go to the Metrics tab..

..and click on Create Metric.  On this screen you are prompted to enter some information.  Now for a quick word on metic identifiers.  Metric identifiers must be globally unique so for the purpose of this step-by-step you will need to come up with a unique metric identifier.  Try just appending your user name to CPU_CORE (ex: CPU_CORE_BOB99).  If the name exists you will be told and you can change it.

Go ahead and enter the following:

Name: Per core CPU utilization

Short name: CPU Core

Identifier: CPU_CORE_... (the name you created above)

Description: Per core CPU utilization

Unit: Percentage

Default Aggregate: Average

Click Save and your new metric should show up in the list.

Note that a plugin can provide several metrics if desired.

Step 2 - Create the plugin

Now the easy part, creating the actual plugin.  As noted above all plugins are simply github repositories so to start off we need to create a new repository.  Log in to github and click on New Repository

On the next screen you are asked for some information.  Enter the following:

Repository name: graphdat-plugin-cpu-core

Public: Yes

Initialize this repository with a README: Add .gitignore: Node  Add a license: Apache v2 License

Note that we chose Node for this step-by-step but you plugins can be any language you like.  We also chose the Apache v2 open source license as this is the default license Graphdat uses for all its open source.

Now click Create repository, all good.

At this point we have a new repository and github has been kind enough to initialize it with some files.  Let's clone the repository locally.  The easiest way to do this is to first copy the github URL to the clipboard by clicking the clipboard icon on right side of your new repository page.

(Note that we have chosen SSH for this step-by-step.)

Now we just use git from the command line to bring it down:

$ git clone [email protected]:<your github username>/graphdat-plugin-cpu-core.git

Go into the folder..

$ cd graphdat-plugin-cpu-core

..and create a new file called index.js.  In this file paste the following code:

https://gist.github.com/pdiemert/6979264

In the code above we used CPU_CORE as the metric name.  You should replace this with the metric name you created from above.

Note: Although not applicable in this step-by-step, you may choose to pass configuration information to your plugin for testing purposes.  To do this, simply create a file called param.json in this directory that contains a JSON object with the parameters/value pairs you require.  During normal operation the relay will overwrite this file with the settings that are chosen in the Settings / Relays / Plugins screen.  See the section on paramSchema in the plugin documentation on how to define configuration parameters. 

Now give your plugin a try from the command line..

$ node index.js

Now, assuming all is working up to this point, you should now be seeing some output of your CPU core percentages, reported once a second.  That is all the coding you will need to do for this plugin!  Press CTRL+C to stop the plugin.

Next, we need to create a description of this plugin.  Create another file in the same directory called plugin.json.  In this file paste the following:

{ "description" : "Provides per core CPU utilization", "command" : "node index.js $(pollInterval)", "metrics" : ["CPU_CORE"] }

As before, you should replace CPU_CORE with the name of your metric.

That's it!  You have created a minimal plugin.  Before we can deploy it make sure to check it into github:

git add . git commit -m "added source files" git push

Step 3 - Testing the deployment

Before submitting your plugin to Graphdat for inclusion in the list of available plugins you will want to first test installing the plugin into your own account.  At the command line:

$ curl https://api.graphdat.com/v1/plugins/private/cpu_core/<your github username>/graphdat-plugin-cpu-core -X PUT -u <your email>:<your API key>

You will now see the plugin listed when browsing all available plugins from Settings / Plugins / Get Plugins.

Click Install to add this plugin to your account.  Next you will need to have installed the Graphdat Relay on the machine you wish to monitor.  The relay will host the plugin and communicate the measurements back to Graphdat.  Go to the machine you wish to monitor and install the relay (note: the Graphdat Relay is written in NodeJS so it is assume you have NodeJS already installed).

$ npm install graphdat-relay -g

Let's now create a directory where we want to run the relay.  This is the directory where we will create the configuration file and also where the relay will put any plugins you select.

$ mkdir .graphdat-relay $ cd .graphdat-relay

There is one small piece of configuration required to get the relay running.  We will create the initial configuration with the following:

$ graphdat-relay -e <your email> -t <your api token>

This will create a new file in this directory called config.json.  The email and API token values can be found in your Settings / Account tab.  You can browse the complete relay reference for more background on how it works.  Now that the relay is configured, from the same directory as the config.json, launch:

$ graphdat-relay

You should now be able to see the relay in your Settings / Relays tab.  Click on the relay to see the plugins it is running.  Next click to add the cpu-core plugin:

Once you select the plugin you will be taken to the plugin configuration screen.  We don't need to change any settings so just click Save.  As soon as you save the plugin configuration you will next be shown the relay console.  If we wait a few moments we will see the relay install the plugin:

The plugin is now running!  For this step-by-step we have chosen not to add a new dashboard (see plugin documentation on how to do this).  Instead, we need to add the metric graph to our dashboard using the dashboard editor.  Open the dashboard editor now:

Notice that in the toolbox we can see our new metric:

Just drag + drop this metric onto the dashboard and click Save.

Done.  On your dashboard you should now be seeing the per-core CPU performance being graphed!

If you were building your own plugin you'd probably be making further changes to the plugin before release.  You can simply repeat the steps above (from the point of pushing your changes to github) and you will notice an 'Apply update' link next to the plugin from the Settings / Relays / (Relay) tab.

Step 4 - Submit to Graphdat

Graphdat has doubled its customer base in the past six months and now 1,000 customers use our platform to improve the performance of their servers and applications.  You can see the demo here if you are new to the family.

We couldn't be more thrilled with this achievement, and we wanted to take a moment to thank you.

A lot happens on the way to a thousand.  For a startup, every customer matters.  Each interaction, each signup, each referral, with every agent you install, you help shape the company.  

We have received fantastic feedback on the product about it's ease of use, the real time animations and the amazing price.  Graphdat is maturing as a product and in case you missed it, we have recently released a ton of new features:

Graphing as a service (Gaas)

The largest feature since the original launch of Graphdat.  Graphing as a Service puts you into the drivers seat and allows you to `will` your graphs into existence.  We created the Graphdat Agent to pull the statistics from your servers that we believed were important.  We gave you an amazing dashboard straight out of the box, but we know you want more.  Redis, memcache, the number of support tickets you receive or even the number of coffee's being made from your coffee machine, GaaS has you covered.

Building your own dashboards

With GaaS you have the ability to design your own Graphs, but what about your dashboards?  With `build your own dashboards` you can alter the Graphdat experience, allowing you to create different views targeted to what you care about.  Everyone can have their own dashboard to visualize what is important to them. 

Partner API

You love Graphdat so much, you want to offer it to all of your customers. With Graphdat Partners we easily allow you to add users to your account through a simple REST API, embed our dashboards into yours, and bring the Graphdat experience to each one of your customers.  Instead of having to roll your own monitoring system, with static, stagnant graphs, you can let Graphdat do the heavy lifting.

Hourly pricing

Why pay for what you don't use?  We had a look at the pricing strategies out there and thought to ourselves, why doesn't anyone get it?  The number of servers you have may increase and decrease based on your load, you add/remove servers in the cloud all of the time, why do you have to pay for the entire month when you only ran it for one day?  Graphdat does the right thing without you needing to even ask.

Alarming your servers and applications

Another alarm went off again, probably a false alarm, it gets ignored again, but maybe this time it was real?  No one knows because they just ignore every alert now.  Alerts are at the heart of what we do at Graphdat, we need to be alerted when something goes wrong, and false alarms don't cut it.  So we built, `Smart alarms` to get rid of the noise and to tell the truth.

What makes us different?

All of these features are available to all of our users.  There is not an upgraded pro plan, or three tiered pricing plans, at Graphdat we give you all of our features.  Pay for what you use, receive great customer service, nothing more, nothing less. 

Thank you again for helping us get where we are today, and thank you for your continued support.  For retweeting our links, for liking us on Facebook, for referring us to your friends, for posting about us on Stackoverflow and Quora, and for contributing to our open source SDKs.

We love you guys!

Sincerely,

The Graphdat Team

For those not using Graphdat yet, try Graphdat Today

The big release announcement is here, Graphing as a Service is released.

 Your data is probably stored in different locations across multiple services, or maybe potentially you are not tracking the data because you do not want to go through the hassle of setting up your own custom reporting service, it's a lot of work, trust us! Let Graphdat help you get everything into one place. 

  No longer are you limited to just what metrics our agent provides for you.  No longer are you limited to only the SDKs that we have available, now you are truly free to graph as you wish.

We have opened up our platform, provided a REST API to add your own metrics and allow you to create your own graphs, with whatever data you is important to you.

Do you want to graph MongoDB?  Maybe Redis or Memcache?  Or maybe you just want to track your disk usage on your server?  With our new Graphing as a Service (GaaS), you can do just that.

This is the first release of many, so let me get a couple of things out there first.

Pricing.  This is a beta release of GaaS, so we are offering it for free to all of our customers while we are in beta.  When we move out of beta, we will offer two separate prices based on what you need.  Same as before, if you are running a Graphdat agent, we will continue to charge the same low hourly pricing (nothing has changed) on a per agent basis.  If you add in your own Graphs (or do not use an agent and only use the custom graphs), we will charge per metric for the custom metrics.  Pricing will be somewhere around $0.25 / 10 Million data points (so about what the current agent is costs right now).  We will still offer 1 free server or the equivalent in data points, so nothing changes there, same sweet deal.

Drill down support.  The first round of graphs do not support the ability to drill into them (like you can for CPU processes or see the code level in Application Responses).  Support for drill downs is coming, but this is just the first version so you have something to get started with.

With those out of the way, lets start creating our first custom graph.  Disk Space is something that we care deeply about at Graphdat, with multiple terabytes of data and growing.  Lets start by graphing the amount of free space on our test server.

Firstly, lets create the metric that we want to graph.  In the Settings dialog, choose the Metrics tab and create a new Metric.

There is a lot of information here, lets go over what it all means.

name: what should the metric be called.  This metric name will show up on your dashboard and in your alert emails.

short name: sometimes we have less space to work with, so we need an abbreviation, the dashboard builder is one example.  disk space is pretty short, so we can use the same text for both.

identifier: this is the primary key for the metric.  It is global across all of Graphdat.  This is how we will identify your metric when you send us data (it will also be used when the plugin system comes into play to make it easier for you to write and share plugins).  Create your own personal namespace and make it easy for yourself.  In this case, all of my metrics will start with SIGURD as they will come from our test environment, code name sigurd.  So lets create this metric in that namespace.

description: we display this along side the metric in the list screen so you know a little bit about your metric.

unit: what type of data are we tracking and how do we want to roll up the numbers so they make sense when the values get larger?  

Count of Bytes tracks things like network traffic.  When you send larger values, we roll it up into K and MB.  ex. 8192

Percentage tracks the percentage of something.  How busy is the CPU or how full is the disk? ex. 0.12

Duration tracks time.  How long did that HTTP Request take in milliseconds?  When you send larger values, we roll it up into seconds and minutes. ex. 1000 

Number is a count.  How many requests did you receive, how many cups of coffee did you drink? ex. 1

default aggregate: if you normally only view things in real time, you may not have even noticed that you can change the aggregates (in real time we are showing you the raw numbers).  If you zoom out with your mouse, or select a larger time span from above, a new drop down appears next to your graphs and let you manipulate the data.  Do you want to see the average of the values in that interval, or maybe you want to see just the min or the max value at in time period, just select it from the dropdown.  In this case, selecting the `default aggregate` lets us pre-select the drop down for you to make you life just a little easier. 

 Now that we have our new metric, we need to create a graph to be able to see it.  Because we are creating a disk usage graph, we'll need to create a dashboard with all of our disk metrics (for a refresher in how to build dashboards, you can check out this post)  

Go into the dashboard builder and drag on your metrics.  Your newly created metric now appears on the left, so lets drag that over and create our disk stats dashboard, give it a name and hit Save.

With our dashboard ready, we can send a couple of test metrics to make sure things are working correctly.  Documentation can be found on the developer site

With a little bit of cURL love we can test it out nice and easy.

curl https://api.graphdat.com/v1/measurements \ -u [email protected]:api.896a98bf4f \ -X POST \ -H "Content-Type: application/json" \ -d ' { "source": "sigurd-db", "metric": "SIGURD_DISK_SPACE", "measure": 0.10, "timestamp": 1378262618 }'

Lets go over what those values actually mean:

email: what email address do you use to sign in with

api-token: this is YOUR personal API Token.  If you are running this as an automated process, create a user for it and use that token instead.  That way you can set that account as a non-admin account, user accounts are free to create, so create as many as you need.  You can find your API token in the Settings -> Account tab.

source: where did the data come from?  In the legend at the top of your dashboard there is a list of servers (sources), what name would you like this metric to be associated with.

metric: what identifier did you give for the metric? What is the metrics primary key.

measure: what value are we sending?  Given this is a percentage, lets try with 0.10 (0.10 == 10%).

timestamp: timestamps are a little tricky, they are Unix time.  If you leave the timestamp out, we do the right thing and use the current date, but when you send a batch of metrics, you need to send a timestamp so we know where to place the metrics.

So lets run the curl command a couple of times and check the graph for the results (you can remove the timestamp field for this test, otherwise you will need to keep updating the timestamps for each request).  Success, we have data!

Now that we know this works, lets write a little node app to run so we can leave it running in the background and report our stats.

https://gist.github.com/6433040

And it worked, now we have our first graph.  Lots of free space on the server, just what we want to see.

  Next, we need to make sure we are alerted when things go wrong.  If the disk fills up, we need to know.  To create an alarm, open Settings -> Alarms and create a new alarm (for a refresher in how to create an alarm, you can check out this post).

With our new dashboard, our new custom graph, our new service and our new alarm, we have successfully extended Graphdat.  

GaaS for the win!

Log into your Graphdat account to see the magic.

For support, drop us an email at [email protected] or on twitter @graphdat

The Graphdat team has been hard at work on building an amazing new feature set, the first is which is the ability to build your own dashboard.

Your Graphdat Dashboard starts with everything we thought could be important to you.  Out of the box we give you our best guess on what  you need.

CPU, Memory, Disk and Network graphs show what is happening on your server.  If you double click the CPU or Memory graphs, you can zoom in and see what is consuming your CPU or Memory (scroll the mouse wheel out a little bit or expand your time range a little if you do not see any results).  We poll this data less frequently than the actual numbers to keep the CPU consumption of the Graphdat agent nice and low.

Application Request Count and Application Response Times show what is happening with your applications.  When you add an SDK (webserver or your chosen programming language) you can see all of the requests that are being made and how long they are taking.  Double click on the graphs to zoom in, and you can see break downs of your code if you have added a little bit of instrumentation.

But, we know this is just our best guess.  The power needs to sit with you, so you can build what you need.  So Build your own Dashboards was created to fill that gap and give the power back to you.

Lets get started.

You will notice there are two new visual elements on your dashboard.  

The Dashboard picker at the top, that allows you to quickly change your dashboards

The Dashboard Edit tool which allows you to create or modify your existing dashboard

Lets modify the existing Dashboard, so click on the Dashboard Edit Tool.

In the editor, you can see the available Graphs that you can drag and drop over to your dashboard, you can click the X to remove any of the Graphs and you can preview your changes before saving them.

There are many customization options that alter the look and feel of your dashboard:

Header: contains the dashboard selector, the pause button and the time range selector.  You probably want to keep these on unless you are using it as a wallboard, auto means we choose based on the size of your browser window.

Legend: contains the list of all of your servers.  It is very handy to turn a server on or off by clicking on it in the legend, and it lets you know where your data is coming from.

Slider: The graph running on the bottom that lets you change your time range, quickly scroll back to another point in time and gives a nice overview of what is happening with the selected metric.

Pad: How much padding would you like between your Graphs?

Rows/Columns: How many rows and columns would you like in your dashboard?

Back Color: What color would you like the background to be?

Light Theme: Graphdat support both a light and a dark theme, which one do you prefer?

Ok, now that we have gone over the basics, lets add some Graphs.

The Graphs have drag handles, so you can expand the graphs to the configuration that you want.

Graphs that have the same type of units can be added together.  So if both graphs show percentages, or both graphs show bytes, or both graphs show a number, we can combine them into a single graph.  By hovering over the top or the bottom of the graph, you can see split. 

In the example dashboard we do not have an SDK hooked up, so lets get rid of the application graphs, expand the CPU and MEMORY graphs and add our disk stats and hit SAVE.

Now I have the dashboard that I wanted.  But I want to have one dashboard with just CPU and Memory because sometimes that is all I want to look at.

So from the Dashboard selector, select create a new dashboard.

Configure the layout with just the CPU and Memory graphs, and give the dashboard a good name.  Click Save and you now have two dashboards that you can select from.

If you want to make the cpu-and-memory graph the default dashboard (so when you come back, this is the one that you see).  You can either check the Make Default checkbox in the dashboard editor, or choose it in the Dashboard tab in Settings.

  Graphdat dashboards, just the way you like it.  

Log into your Graphdat account to see the magic.

For support, drop us an email at [email protected] or on twitter @graphdat

We've been getting some feedback lately that you would like our pricing to be more dynamic.  Specifically, there are two particular scenarios that we hear about time and time again:

Spot use:  You only enable the agent when you are tracking down a problem.  While we encourage you to continually run the agent so you get historical statistics (unlimited retention), sometimes spot usages makes the most sense.  Because you are not using it for the whole month, why should you have to pay for it?

Scaling up/down:  Your server numbers increase and decrease with demand, and you have Graphdat agents running on all of the servers (of course you do).  But when you scale up to 15 servers and then back down to 5, you are billed for all 15 servers and that doesn't seem right.

With our current pricing these scenarios could be more expensive than you would like, because you are paying to monitor a server that is not reporting any data.  You are paying for the entire month, but only using a fraction of it.

We have heard your cries and in light of this new demand, we sought to improve our billing with as little impact as possible to those that are not affected, but solving the problem for you as well.

    Graphdat announces hourly billing.

  $0.014 per monitored server hour per month

  No minimum charges

    What does this mean for you?

If you are running a single server continuously for an average month (730 hrs), this will still cost the same as it does today, about $10.  However, if you stop/start Graphdat agents, you will only be charged for what you use.  All good.

But wait, what about the 1 free server deal that you are offering, how does that work?

With the new hourly billing model it is somewhat problematic to offer a single server for free because we now allow you to stop / start machines at will.  Which machine is the free one?  To solve this problem and keep providing our awesome 'free forever' offer, we are also announcing:

    The first 750 hours of every month are free

    Where did that number come from?

We took a typical month (730hrs) and added in some extra free hours to handle the fluctuation in the length of calendar months for a total 750hrs.  We now have a deal that is better than before.  One server can run continuously each month for free, and you have a couple of hours little left over to spend on your other servers.

That's it!  We hope you will agree that this new pricing will be an immediate benefit to existing users and also provide more flexibility as monitoring needs grow.

  Thank you for using Graphdat!

Sincerely,

"Excessive tracing detected, some data has been truncated and metric X has been disabled"

Huh?  If you are seeing this message it is most likely because you just installed the Graphdat agent on your server along with some instrumentation of either your application or the web server.  If your application is receiving a high volume of requests then this is most likely causing the notification.  The result is that some of your metrics have been disabled.

You have not done anything incorrectly but there is probably something particularly unique about your application.  To help explain it is important to understand how data is aggregated and presented for analysis in Graphdat.  For example, let's say you have an application that exposes the following URL for browsing product details:

http://<your domain here>/catalog/product/:id

(:id represents the unique product ID number)

Because you have instrumented your application every time this URL is accessed timing metrics are sent to Graphdat.  Now let's say that this URL is accessed over time as illustrated in this diagram:

Because you can get many, many requests in a short period of time it is necessary to summarize the requests over various time periods.  The time period displayed depends on the zooming factor being viewed.  Let's say that you have zoomed your Graphdat dashboard such that you are now seeing data summarized at the 15 second interval.  The data would calculated as follows:

We've gone from 3 data points down to 1 which represents the 15 second interval.  In order to come up with the data point Graphdat averaged the time for all the requests made to the URL:

(100ms + 200ms + 100ms) / 3 = 133ms

The key to providing this summarized view is that all of the URLs in the above diagram are the same.  In this case the URL is said to be the 'key',  Now what if we have more than one URL?

As you can see, Graphdat has had to create an additional summary data point for the new URL.  We're still doing good, however, because we've been able to take 4 datapoints and reduce it to 2 to make our graph more readable.  At this point there two very important problems with the way we are visualizing this data:

What if I have many, many different URLs?  In the above example the product ID itself makes up part of the URL meaning that there is a unique URL for EVERY product.  For a busy website you may be receiving thousands of these requests per second which would make for an overly crowded graph.

They are all the same code!  Let's not forget that the fundamental challenge is to pinpoint bottlenecks in the application and in this example it is very likely that the code which handles the request for product ID #1 is the EXACT SAME code that is handling product ID #2 so you probably want to see a single line on your dashboard for these types of URLs.

It turns out that in some cases this problem can be solved automatically for you by using the various Graphdat SDKs.  If you integrated Graphdat into your web framework it may be possible for Graphdat to use the symbolic representation of the URL that you've specified in your code.  For example, if you have used the Node SDK along with the Express web framework and you have set up your product view code as follows:

app.get("/catalog/product/:id", function(req, res, next) { ... });

..then you will only see a single graph line plotted for every product page request with the URL listed as:

/catalog/product/:id

In the case that you either aren't using a Graphdat SDK or the SDK cannot find symbolic URLs in your code there is a simple way to configure the Graphdat agent running on your server to provide symbolic URLs.

First, locate your graphdat.conf file.  You can view the graphdat.conf documentation if you don't know where this file is located (typically in /etc/graphdat.conf for Linux based systems).  In this file we could add the following line:

route_map: GET /catalog/product/[0-9]* /catalog/product/:id

The above mapping specifies the following:

For requests using the HTTP method GET

For all URLs that match the regular expression "/catalog/product/[0-9]*"

Replace with the URL "/catalog/product/:id" for graphing

Done!  Once you restart your agent and you re-enable your metric in the settings dialog above, you will now get a single summary graph line for all your product page URLs.

Another approach to help prevent this situation is to adjust the time threshold for requests.  This sets the minimum amount of time a request needs to take before it is recorded.  In most situations, it is only the slower requests that you want to pay attention to so it is ok to throw this tracing data out.  To change the request threshold you can add a line to your graphdat.conf:

agent_request_resp_time_threshold:  200

(sets the threshold at 200ms)

If you see the "excessive tracing..." message again it means that you may need to add more route_map lines to your configuration or try upping your request threshold.

You asked for it, you got it.  Give a warm welcome to the new Graphdat Python SDK.

The Graphdat SDK for Python lets you instrument your WSGI applications and mix that with a little bit of Graphdat love.

How to Install

If you are using PIP, add graphdat to your requirements.txt file:

graphdat >= 0.2

and install.

pip install -r requirements.txt

(you can install graphdat using pip or easy_install if you want instead)

pip install graphdat

or

easy_install graphdat

Now that you have the graphdat module installed, you need to add it to your wsgi file. You can add the attribute to your application

from graphdat import WSGIWrapper @WSGIWrapper def application(environ, start_response) # your existing application ...

or you can wrap the application instance

application = framework.WSGIHandler() # your existing endpoint from graphdat import WSGIWrapper application = WSGIWrapper(application)

The code will automatically hook into the WSGI object, so if you run this on the same machine which is running the Graphdat agent, you will see which endpoints are being called by your web server in the Request Count and Response Time graphs.

Above is an example of what you will see when you do the most basic instrumentation of your Python application.

Multiple Data Points

Having the http endpoints graphs is very helpful, but the grapdhat sdk also lets you provide more information about the functions running inside each endpoint.

Take a look at this example:

https://gist.github.com/5771637

When you imported the graphdat module and wrapped your application, we include the graphdat property in the environ objects that get generated. You can then use that to instrument your code with begin() and end() statements.

The example above will display a stacked bar graph on your graphdat dill down dashboard, showing you the breakdown between each of the different trace items

Custom Routes Regex

We have some extra love for those of you with some regex of your route URLs. If you want to make your urls a little prettier, you can pass in a list of regex routes and we'll rewrite your URLs for you.

The wrapper object takes in an extra argument:

def __init__(self, app, options=None):

If we add some regex love when we initialize the wrapper (other options are displayed below)

# create a regex that will match our article page /articles/2013/05 options = { 'routes': ['^articles/(?P<year>\d{4})/(?P<month>\d{2})$'] } application = graphdat.WSGIWrapper(application, options)

Start up the server and hit it with some urls like:

http://localhost:1337/articles/2013/05"

http://localhost:1337/articles/2013/01

http://localhost:1337/articles/2011/02

and we will report timings for route "/articles/:year/:" only.

Configuration

Configuration information can be passed to the library on initialization, just like passing in the regex routes.

For other options supported, check out our wiki

E.g.:

options = { 'debug': True, 'messageDump': True, 'routes': ['^articles/(?P<year>\d{4})/(?P<month>\d{2})$'] } application = graphdat.WSGIWrapper(application, options)

The great thing about working on Graphdat is that we get to build tools that we want to use ourselves, everyday.  Being developers for many years we've been exposed to plenty of alarming / alerting tools.  In fact, alarming is usually the second thing we set up after our servers are created.  We can't watch our server monitors constantly so getting told when things are suspect is vital.

The usual suspects of alarming haven't really changed over the years.  At the application layer the alarms will typically test for slow or missing responses from a web request.  At the system level, there are basic resource threshold checks.  One of the biggest issues that surrounds alarming techniques is the ability to produce valid alarms.  The basic problem is that, as we all know, systems fluctuate throughout the day, mostly for known reasons which might cause alarms which are superfluous.  You then must dig through the notifications that are generated in order to find the real problems.  There are various ideas around 'flapping detection', application scoring, and other somewhat arcane techniques so we thought "why not use our existing analytic engine behind Graphdat to make a better alarming solution?"  Here's our approach.

Firstly, you can add alarms quickly and easily from the Alarms tab in the settings dialog.

On the Alarms tab we can see our existing alarms and click to add a new one:

Next we see the alarm settings:

We tried to make alarms as easy as possible to set up. First you give the alarm a name.  This is a simple, short, title that you can create so you can quickly get an idea of what the issue is when you see the alarm. Next is the metric.  This is can be one of:

CPU

Memory

Disk bytes read or write

Network bytes read or write

Application request count

Application response time

These settings are typical with what you've seen elsewhere but now we get to a feature that is a little different;  threshold.

What sets apart Graphdat alarms is that we can not only alarm basic aggregates like min, max, count but we can also alarm for an average.  The aggregates work in conjunction with the time period:

The time period can range from 1 second up to 12 hours.  For example, if I set a CPU threshold of, say, avg > 80% and a period of 1 minute then this translates to:

"Tell me when my CPU average is above 80% for at least 1 minute"

In practice, this technique for detecting actually bad states has been most effective for us at Graphdat.

After setting the threshold and period you can set recipients and add a note.  The note will be passed along at the time the alarm notification is sent.

Now you can choose which servers you wish to alarm.  By default you can simply alarm for all servers.  This setting works in conjunction with the 'notify for every change' feature.

To illustrate how these settings work let's take a simple of example of monitoring 3 servers, A, B, and C at a 1 minute period.  Consider the following scenario: 1:00pm - Alarm A triggered 1:01pm - Alarms B and C triggered 1:02pm - Alarms A and B resolved 1:03pm - Alarm C resolved By default, without the 'notify for every change' option you would receive just two notifications:

ALARM at 1:00pm because at least on of your server alarms triggered

RESOLVED at 1:03pm when all of your server alarms were no longer triggered

However, if you enable 'notify for every change' you would have gotten four notifications as follows:

ALARM at 1:00pm because at least on of your server alarms triggered

ALARM UPDATE at 1:01pm stating that B and C are now triggered, A continues

ALARM UPDATE at 1:02pm stating that A and B are resolved, C continues

RESOLVED at 1:03pm when all of your server alarms were no longer triggered

Speaking of notifications, here's what a typical notification mail looks like:

Notice that you are told exactly which alarms are triggered (or resolved as the case may be), why they are triggered, and a link to the dashboard at the point in time of the trigger.

When you go to your dash you will now some differences.  The first thing you'll notice is that the graph representing the metric that has triggered the alarm now has a flashing red border:

The next thing you'll notice is a red bell icon in the graph title bar.  We can click this to see what has triggered:

You can now click on the triggered alarm to go directly to the point in time of the trigger.

We at Graphdat have worked hard to build a product we can be proud of and although we are in a pretty competitive space we feel Graphdat, for what it provides, performs well and excels.  Up to now we've funded the project ourselves and let everyone give it a try for free so we can get the product right.  We've now hit that part of the journey where we will need to  start charging for our service.

We at Graphdat have a philosophy.  We believe performance monitoring should not just be the privilege of the well funded corporate types, but also accessible by the likes of all startups no matter the size.  We make this distinction because early in our own careers (and in fact, it continues today), we found ourselves needing good tools but were stunned by the high prices most decent performance monitoring tools commanded.  For example:

High per-server pricing - We found that per-server pricing was enough that we couldn't install the monitoring on all servers.

Contracts - We were asked to enter into contracts longer than we felt comfortable entering into.

Steep price gradients - The pricing plans scaled up quickly so the cost went up quickly at thresholds which were not really in tune with the growth of our business.

So given this was the landscape we put our heads together, looked realistically at our costs, and came up with what we believe is the best for the customer.  Here's the deal, starting today Graphdat will cost:

$10 per server, all standard features with unlimited history

1 server is free, forever

Yes, that simple.  Check out more on the pricing page.

For example, if you are just starting up and only have a single server or there is only one server you care about then you can use Graphdat, free, for as long as you like.  However, should you want to start expanding your understanding of your infrastructure by adding more servers to your graphs you can do so for $10 per server, as many as you like.  You pay for your service by attaching a credit card to your account, remove it any time, no obligations.

Next, what about our existing users.  We believe strongly in giving back to the community that helped us make the product it is today, and we wanted to make sure our existing users were in a better place with this change.  We decided to give our existing customers the service for free. We will zero rate all of their running agents and give them all of the benefits of being a standard paying customer.  This is our way of saying thank you.

It's hard to find a balance between providing this kind of service while also making it approachable to the widest possible audience.  We at Graphdat feel that better working apps means better experiences on the web, and for that matter, a better performing web in general which benefits everyone.  With this new pricing we'd like to continue this philosophy into the future.

Regards

The Graphdat Team

As of today we are announcing new pricing.  One simple plan to rule them all.

Existing customers, don’t worry, we have your back.  We are giving you the ‘big thanks for backing Graphdat deal’. The key points are:

All existing customers will continue to enjoy the service for free, for their currently running servers, no matter how many you have.  If you have five agents running today you will won’t be billed a penny until you install your sixth agent on our new pricing plan below (hint, it’s way cheaper).  In the new 'Accounts' tab of the settings you can see how many free servers you have been credited with.  Your plan includes all of the standard Graphdat feature set.

All existing customers will have their data retention on each server increased from 90 days to unlimited.  We heard you loud and clear, don’t delete my data, it’s useful!

We believe strongly in giving back to the community that helped us make the product it is today. In our view the only way to do that is provide a better product and pricing model then when you signed up and we think we have delivered on that promise. There’s more!

After much discussion with the community, we agree that charging circa $50 per month per server for application performance monitoring is simply too expensive relative to the actual costs of the server itself. So we have worked hard on optimising our platform to make it as cost effective as possible to deliver our savings onto you.

Our new pricing model effective immediately for new customers is:

One server free with unlimited data retention and all of our standard Graphdat features

Each additional server is charged at $10 per server per month

Data retention is unlimited for each server

That’s it. A simple, value for money heart rate monitor for devs.

We’re excited about the future of Graphdat and by taking an aggressive stance on pricing we sincerely hope you will agree we are a great value for money proposition for any developer wanting to get the pulse of their applications in a production environment. Plus we wanted to shake things up a little in the application performance monitoring space.

Feel free to send us a mail [email protected] or talk to us on twitter @graphdat

Thanks again to our existing customers and if you are a new customer, time to get on board.

Regards

The Graphdat Team

Today we are announcing a significant milestone for Graphdat - 500 customers are currently using our platform to watch the performance of their servers and applications in real time in order to improve their performance. You can check it out here.

We couldn’t be more thrilled with this achievement. It was 107 days ago we posted a link to Hacker News announcing the arrival of Graphdat to the dev community to whom we built this for and owe many thanks too.

The community has grown us to 500 customers. With no marketing budget to work with we rely solely on the community to spread the word and spread the word you have.

With all the great feedback we receive, Graphdat is really maturing as a product and with lots of new features recently released:

MySQL and Microsoft Server Support.  We automatically profile the queries on your database with zero code changes required by you.  Graphdat is about making your life easier and making you more productive, this is a perfect example of what we are striving to do.

White and Black themes.  It's small I know, but just a little customization can go a long way.  Having the different themes really makes the next feature, Embedded graphs truly amazing.

Embeddable Graphs on any webpage, Graphdat to-go.  This was a monumental achievement for Graphdat.  We give you the ability to display your graphs anywhere you want, even if you are on our Free Plan.  Dashboards, widgets, internal wikis, we want you to have the best experience, paying or not.   

We have so much more in the pipeline.  Alarms, alerts, adding meta data to your graphs, plus Python, Ruby and Java support as well as mobile application performance management.  We are really excited to see what the next 107 days will bring.

I have to say that I’m very excited about Graphdat's future. 

Try Grapdat Today

Getting started with Graphdat is easy. Just signup, install the agent, and start monitoring your servers and web applications in a couple of minutes.

Richard Gillis

Graphdat launches embeddable, real time dashboards that you can add to any web page of your choosing.  We know many of you have your own dashboards, your own way of keeping track of what is going on, and we support that, hell we love it!  So make way for embeddable real time analytics and you don't even have to have a paid account, we love you that much.

  That is a real time graph of one of our production clusters (15 servers).  You can click on the graph to see a legend, drag around to see historical data, zoom out to see larger sections of time (with the mouse wheel), etc... It is essentially everything you get within your Graphdat dashboard, except it is embedded in any web page you like.

How to do it?  Very easy.

Sign up at Graphdat.com.  Fill in a couple of fields to get your trial account rolling. Install the graphdat agent on any of the servers you want to track (once you sign up you will get instructions on the agent setup).

With your Graphdat dash running, just click the embed button on the menu

Here's the embed dialog. Notice we have all our metrics on the left.  The one we want is CPU.

We just need to drag+drop the CPU metric to the upper left of the grid...

Once we've dropped in CPU, let's drag the corner so that it takes up the whole display

For this graph we will turn off the header, legend, and slider

To make it fit nicely onto our page we can set the background to transparent

And finally, because our target web page has a white background we will set the light theme

So look and feel are all fine, let's copy the HTML

And here's the snippet that was copied to my clipboard:

<iframe src="https://dashboard.graphdat.com/embed/d10131ee28?e-w=3&e-h=2&e-pad=5&e-header=0&e-slider=0&e-legend=0&e-light=1&e-bg=none&e-sg-cpu=0-0-3-2"></iframe>

(Note: This iframe will be of the default width/height.  You can add width="your width" and height="your height" tags to the iframe and the graphs will resize accordingly.)

Just paste that into the web page of your choice and we are good to go!

Now that we've got the embed working, we need to make the dashboard public so that the embedding can be displayed anywhere without the user viewing the web page to be logged into Graphdat.  To do this we first open settings:

..then select Project on the left, and finally, click the lock icon in order to make the dash public

So as you've seen you can also add other graphs besides just CPU to this dashboard as well as other features but we'll leave that for the reader.

  So you have jumped on the node.js band wagon and things are going well. So well in fact that you now love node, its awesome, you tell everyone how good it is.

  How being able to only write one language on the front end and the backend (well, kind of) makes you so much faster and more productive.

  How you love spending all day in the terminal in your newly found (or newly re-found) text editor, twitter client, hacker news client, and maybe even IRC and now things are great.  You have chosen your shell, your color scheme and made your terminal tweaks and have turned out a bunch of production code.

  And then you do a new release and things slow down.  You don't know where, you don't know what 'slow' means or even what your past performance was.  You just have an email from some manager or some customer who says "things feel slower".  What do you do now?  Where do you start looking?

You turn to your co-worker, he comes from a C# or a Java background, with a rich set of tools for analyzing everything under the sun and ask him what to do and he says, "don't you have tool X"?  You say no.  He says what about Y?  You say no.  He starts laughing and you start putting logging and tracing statements live in your production code to see what is wrong (doesn't that feel gross), node-inspector and the V8 profiler (in production, because its not the same in staging, do you have a staging?), or you start reading about dtrace and realize your not on a illumos-based systems.  You're a sad panda.

There is a better solution, and we here at Graphdat have felt your pain, we started down the nodejs path and ran into the exact same problems.  So we build Graphdat.  We use it to tune our code and we think its so great, we want to help you optimize your code as well.

Graphdat, is the simplest way to instrument your code and provide you with amazing looking graphs of what is really going on inside your code.  We also correlate that with what is happening on the server so you can view it all together in once place.  Isn't it pretty!

Lets get started.

  First we need to signup with Graphdat  (https://dashboard.graphdat.com/signup) so we have somewhere to put all of those metrics.  It's quick and painless, an email address and a password, how much simpler could you get.

Next, lets get the agent up and running on your server (or your local development box).  The SDK (http://www.graphdat.com/how-it-works) we will be installing will talk to the agent, who then sends everything to your graphs as a single package (server metrics and your code metrics).

Now the fun part starts, we need to start instrumenting your application.

  First order of business is installing the graphdat npm module.  It's nice and simple, just a quick one liner on the command line:

npm install graphdat

Now we are ready to write some code.  The example always given for node.js is the simple webserver, so lets create that:

https://gist.github.com/4662306

run curl against it a couple of times to generate some results (or just use your browser)

curl localhost:8080

And success, we can see the response (graph in the upper right).  With only that one line of code, Graphdat created a hook in your code and now you get results with no extra effort.  Sweet!

Next up, we are going to get a little more crazy, lets add in some FILE I/O and measure how long it takes to serve up that file.

First we need to create a file, we can use some linux foo to create the file (or just create one by hand if you are some inclined).  This will generate a nice 20MB file for us to test with.

yes “we love graphdat, yes we do.  We love graphdat how \'bout you″ | head -c 20000000 > test.txt

Now what we have our large file (things happen so quickly when you are running locally don't they?), we can build our new webserver.

https://gist.github.com/4662537

This time, its probably better to hit the page in your browser as it will fill up your console pretty fast (http://localhost:8080).  Now go back to Graphdat and look at our Application Response Time (same graph in the upper right).  Double click the graph, click on the new peak and your HTTP Request shows up, click on that and you get a break down.

Look a that, a total of 539ms for that request, 424ms taken to read the file and 114ms to write that to the client.  Pretty cool.  

Now if this was your website, you could get a breakdown for how long blocks of code took to run and you would have it historically, so you could compare yesterday to today, or last month to this month.  No more wondering, you now have the hard evidence.

For more information checkout:

Graphdat

We here are graphdat have a bunch of servers that we run and in the end, we always end up needing a VPN service.

We chose OpenVPN because it's pretty easy to setup, and its free!  There are a lot of great guides out there on how to setup OpenVPN, on how to create the keys, but once I connected to the VPN, I could no longer connect to the outside world.

You may have had this problem as well, and for us it was the magic of IP Tables that just made it work.

For my reference, and for yours, here is how we did it.  The adapter of the network interface is eth0, you should change it for the adapter that you are using.  Also, this is running on Ubuntu 12.04.

#setup the IP Table Rules  sudo bash -c "echo 1 > /proc/sys/net/ipv4/ip_forward" sudo /sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE sudo iptables -A FORWARD -i eth0 -o tun0 -m state --state RELATED,ESTABLISHED -j ACCEPT sudo iptables -A FORWARD -i tun0 -o eth0 -j ACCEPT

#save the IPTable rules sudo mkdir -p /etc/iptables sudo bash -c 'iptables-save > /etc/iptables/ip-tables.fw'

# add the rules to rc.local so they get run if the box restarts sudo bash -c "echo \"sbin/iptables-restore < /etc/iptables/ip-tables.fw\" >> /etc/rc.local"

# run the rules and make the VPN work sudo /sbin/iptables-restore < /etc/iptables/ip-tables.fw

How we hacked our landing page conversion rate from 0.5% to 4% for our real time application performance management platform Graphdat

For those with short attention spans the final landing page experience can be found over there https://dashboard.graphdat.com/landing

For those who are interested, here are the steps we went thru to do it. 

When we soft launched our real time application management platform some months back we followed as well as we could best practice landing page design (we have no marketers on staff).

This is what we came up with:

Note, we couldn't follow the best practice landing page design formula to the letter because we didn't have any customers and hence no testimonials. Nevertheless we signed up some customers but our site visit to conversion rate (customer signs up and installs our agent) was only 0.5%.

We believed this to be a poor result by any standard.

As developers building a product for developers the copy + video (demo) + testimonials landing page formula didn't resonate with us and in analysing why our conversions were not meeting our expectations we discovered the number of visitors actually looking at the Demo were only 30% and time on our marketing site was a scant 49 seconds.

We were losing 70% of visitors before they even had a chance to see our product, so for every 1,000 visitors we got 300 demos and 4-5 signups...1.5% of people we got to the demo we got a signup. Glass is always half full around here!

Version 1 landing page key metrics:

Conversions of site visitors: 0.5%

Time on site: 49 seconds

For version 2 we simply made our actual application the landing page. Although logic was telling us this was the 'right thing to do' not one conventional landing page optimisation expert we read suggested it for a software product. If it was recommended, please share the link!

A simple pop-up over the application telling the user what the product does and what the pricing is. Click Close and you are in the Demo.

This simple change had an immediate impact on our sign-ups taking them from 0.5% to 1.5%. It should be noted, we made NO other change, we just stopped losing 70% of users prior to them seeing the demonstration.

Version 2 landing page key metrics:

Conversions: 1.5%

Time on site: 10 minutes 06 seconds

Another key insight was users didn't get how to use our application. It was only when we get real developers using the product, versus test users, that we gained the insight into what was wrong.

Graphdat is a one page application. No scrolling, limited drop-downs, limited clicking. We designed the application to enable users to see graphs quicker than with any other application performance management product. Some features that users didn't pick up on were:

click drag the graphs to go back in time

zoom out of the graphs using the mouse wheel to see a larger time range

double click the graph to drill into it

click the Zoom icon to drill into it

click or drag the slider to scan back in time

we do server and application metrics

The user experience we spent a LOT of time on simply wasn't being used.

For version 3, we went for a totally different experience. On screen writing like something out of Star Wars that the user couldn't click away from except to sign up.

If we had 30 seconds of the users attention we wanted to convey to them exactly what we did, especially around how to use our application so we walked them thru the key features.

Despite ambushing the user with a walk-thru experience the results were what we were looking for and our conversion rate increased from 1.5% to 4%.

Version 3 landing page key metrics:

Conversions: 4%

Time on site: 16 minutes 54 seconds

Probably the most interesting statistic is that showing users our key features led to well over a 100% conversion improvement and users spent almost 70% more time on site versus simply 'clicking around' themselves. This insight was not lost on us because as developers we were slightly concerned about leading users by the nose thru our product, but the results are clear, sometime controlling the experience is a good thing.

For version 4 we will continue to evolve the first time user experience and update you all on how we go.

Cheers