#graphdat

Graphdat has doubled its customer base in the past six months and now 1,000 customers use our platform to improve the performance of their servers and applications.  You can see the demo here if you are new to the family.

We couldn't be more thrilled with this achievement, and we wanted to take a moment to thank you.

A lot happens on the way to a thousand.  For a startup, every customer matters.  Each interaction, each signup, each referral, with every agent you install, you help shape the company.  

We have received fantastic feedback on the product about it's ease of use, the real time animations and the amazing price.  Graphdat is maturing as a product and in case you missed it, we have recently released a ton of new features:

Graphing as a service (Gaas)

The largest feature since the original launch of Graphdat.  Graphing as a Service puts you into the drivers seat and allows you to `will` your graphs into existence.  We created the Graphdat Agent to pull the statistics from your servers that we believed were important.  We gave you an amazing dashboard straight out of the box, but we know you want more.  Redis, memcache, the number of support tickets you receive or even the number of coffee's being made from your coffee machine, GaaS has you covered.

Building your own dashboards

With GaaS you have the ability to design your own Graphs, but what about your dashboards?  With `build your own dashboards` you can alter the Graphdat experience, allowing you to create different views targeted to what you care about.  Everyone can have their own dashboard to visualize what is important to them. 

Partner API

You love Graphdat so much, you want to offer it to all of your customers. With Graphdat Partners we easily allow you to add users to your account through a simple REST API, embed our dashboards into yours, and bring the Graphdat experience to each one of your customers.  Instead of having to roll your own monitoring system, with static, stagnant graphs, you can let Graphdat do the heavy lifting.

Hourly pricing

Why pay for what you don't use?  We had a look at the pricing strategies out there and thought to ourselves, why doesn't anyone get it?  The number of servers you have may increase and decrease based on your load, you add/remove servers in the cloud all of the time, why do you have to pay for the entire month when you only ran it for one day?  Graphdat does the right thing without you needing to even ask.

Alarming your servers and applications

Another alarm went off again, probably a false alarm, it gets ignored again, but maybe this time it was real?  No one knows because they just ignore every alert now.  Alerts are at the heart of what we do at Graphdat, we need to be alerted when something goes wrong, and false alarms don't cut it.  So we built, `Smart alarms` to get rid of the noise and to tell the truth.

What makes us different?

All of these features are available to all of our users.  There is not an upgraded pro plan, or three tiered pricing plans, at Graphdat we give you all of our features.  Pay for what you use, receive great customer service, nothing more, nothing less. 

Thank you again for helping us get where we are today, and thank you for your continued support.  For retweeting our links, for liking us on Facebook, for referring us to your friends, for posting about us on Stackoverflow and Quora, and for contributing to our open source SDKs.

We love you guys!

Sincerely,

The Graphdat Team

For those not using Graphdat yet, try Graphdat Today

The big release announcement is here, Graphing as a Service is released.

 Your data is probably stored in different locations across multiple services, or maybe potentially you are not tracking the data because you do not want to go through the hassle of setting up your own custom reporting service, it's a lot of work, trust us! Let Graphdat help you get everything into one place. 

  No longer are you limited to just what metrics our agent provides for you.  No longer are you limited to only the SDKs that we have available, now you are truly free to graph as you wish.

We have opened up our platform, provided a REST API to add your own metrics and allow you to create your own graphs, with whatever data you is important to you.

Do you want to graph MongoDB?  Maybe Redis or Memcache?  Or maybe you just want to track your disk usage on your server?  With our new Graphing as a Service (GaaS), you can do just that.

This is the first release of many, so let me get a couple of things out there first.

Pricing.  This is a beta release of GaaS, so we are offering it for free to all of our customers while we are in beta.  When we move out of beta, we will offer two separate prices based on what you need.  Same as before, if you are running a Graphdat agent, we will continue to charge the same low hourly pricing (nothing has changed) on a per agent basis.  If you add in your own Graphs (or do not use an agent and only use the custom graphs), we will charge per metric for the custom metrics.  Pricing will be somewhere around $0.25 / 10 Million data points (so about what the current agent is costs right now).  We will still offer 1 free server or the equivalent in data points, so nothing changes there, same sweet deal.

Drill down support.  The first round of graphs do not support the ability to drill into them (like you can for CPU processes or see the code level in Application Responses).  Support for drill downs is coming, but this is just the first version so you have something to get started with.

With those out of the way, lets start creating our first custom graph.  Disk Space is something that we care deeply about at Graphdat, with multiple terabytes of data and growing.  Lets start by graphing the amount of free space on our test server.

Firstly, lets create the metric that we want to graph.  In the Settings dialog, choose the Metrics tab and create a new Metric.

There is a lot of information here, lets go over what it all means.

name: what should the metric be called.  This metric name will show up on your dashboard and in your alert emails.

short name: sometimes we have less space to work with, so we need an abbreviation, the dashboard builder is one example.  disk space is pretty short, so we can use the same text for both.

identifier: this is the primary key for the metric.  It is global across all of Graphdat.  This is how we will identify your metric when you send us data (it will also be used when the plugin system comes into play to make it easier for you to write and share plugins).  Create your own personal namespace and make it easy for yourself.  In this case, all of my metrics will start with SIGURD as they will come from our test environment, code name sigurd.  So lets create this metric in that namespace.

description: we display this along side the metric in the list screen so you know a little bit about your metric.

unit: what type of data are we tracking and how do we want to roll up the numbers so they make sense when the values get larger?  

Count of Bytes tracks things like network traffic.  When you send larger values, we roll it up into K and MB.  ex. 8192

Percentage tracks the percentage of something.  How busy is the CPU or how full is the disk? ex. 0.12

Duration tracks time.  How long did that HTTP Request take in milliseconds?  When you send larger values, we roll it up into seconds and minutes. ex. 1000 

Number is a count.  How many requests did you receive, how many cups of coffee did you drink? ex. 1

default aggregate: if you normally only view things in real time, you may not have even noticed that you can change the aggregates (in real time we are showing you the raw numbers).  If you zoom out with your mouse, or select a larger time span from above, a new drop down appears next to your graphs and let you manipulate the data.  Do you want to see the average of the values in that interval, or maybe you want to see just the min or the max value at in time period, just select it from the dropdown.  In this case, selecting the `default aggregate` lets us pre-select the drop down for you to make you life just a little easier. 

 Now that we have our new metric, we need to create a graph to be able to see it.  Because we are creating a disk usage graph, we'll need to create a dashboard with all of our disk metrics (for a refresher in how to build dashboards, you can check out this post)  

Go into the dashboard builder and drag on your metrics.  Your newly created metric now appears on the left, so lets drag that over and create our disk stats dashboard, give it a name and hit Save.

With our dashboard ready, we can send a couple of test metrics to make sure things are working correctly.  Documentation can be found on the developer site

With a little bit of cURL love we can test it out nice and easy.

curl https://api.graphdat.com/v1/measurements \ -u [email protected]:api.896a98bf4f \ -X POST \ -H "Content-Type: application/json" \ -d ' { "source": "sigurd-db", "metric": "SIGURD_DISK_SPACE", "measure": 0.10, "timestamp": 1378262618 }'

Lets go over what those values actually mean:

email: what email address do you use to sign in with

api-token: this is YOUR personal API Token.  If you are running this as an automated process, create a user for it and use that token instead.  That way you can set that account as a non-admin account, user accounts are free to create, so create as many as you need.  You can find your API token in the Settings -> Account tab.

source: where did the data come from?  In the legend at the top of your dashboard there is a list of servers (sources), what name would you like this metric to be associated with.

metric: what identifier did you give for the metric? What is the metrics primary key.

measure: what value are we sending?  Given this is a percentage, lets try with 0.10 (0.10 == 10%).

timestamp: timestamps are a little tricky, they are Unix time.  If you leave the timestamp out, we do the right thing and use the current date, but when you send a batch of metrics, you need to send a timestamp so we know where to place the metrics.

So lets run the curl command a couple of times and check the graph for the results (you can remove the timestamp field for this test, otherwise you will need to keep updating the timestamps for each request).  Success, we have data!

Now that we know this works, lets write a little node app to run so we can leave it running in the background and report our stats.

https://gist.github.com/6433040

And it worked, now we have our first graph.  Lots of free space on the server, just what we want to see.

  Next, we need to make sure we are alerted when things go wrong.  If the disk fills up, we need to know.  To create an alarm, open Settings -> Alarms and create a new alarm (for a refresher in how to create an alarm, you can check out this post).

With our new dashboard, our new custom graph, our new service and our new alarm, we have successfully extended Graphdat.  

GaaS for the win!

Log into your Graphdat account to see the magic.

For support, drop us an email at [email protected] or on twitter @graphdat

The Graphdat team has been hard at work on building an amazing new feature set, the first is which is the ability to build your own dashboard.

Your Graphdat Dashboard starts with everything we thought could be important to you.  Out of the box we give you our best guess on what  you need.

CPU, Memory, Disk and Network graphs show what is happening on your server.  If you double click the CPU or Memory graphs, you can zoom in and see what is consuming your CPU or Memory (scroll the mouse wheel out a little bit or expand your time range a little if you do not see any results).  We poll this data less frequently than the actual numbers to keep the CPU consumption of the Graphdat agent nice and low.

Application Request Count and Application Response Times show what is happening with your applications.  When you add an SDK (webserver or your chosen programming language) you can see all of the requests that are being made and how long they are taking.  Double click on the graphs to zoom in, and you can see break downs of your code if you have added a little bit of instrumentation.

But, we know this is just our best guess.  The power needs to sit with you, so you can build what you need.  So Build your own Dashboards was created to fill that gap and give the power back to you.

Lets get started.

You will notice there are two new visual elements on your dashboard.  

The Dashboard picker at the top, that allows you to quickly change your dashboards

The Dashboard Edit tool which allows you to create or modify your existing dashboard

Lets modify the existing Dashboard, so click on the Dashboard Edit Tool.

In the editor, you can see the available Graphs that you can drag and drop over to your dashboard, you can click the X to remove any of the Graphs and you can preview your changes before saving them.

There are many customization options that alter the look and feel of your dashboard:

Header: contains the dashboard selector, the pause button and the time range selector.  You probably want to keep these on unless you are using it as a wallboard, auto means we choose based on the size of your browser window.

Legend: contains the list of all of your servers.  It is very handy to turn a server on or off by clicking on it in the legend, and it lets you know where your data is coming from.

Slider: The graph running on the bottom that lets you change your time range, quickly scroll back to another point in time and gives a nice overview of what is happening with the selected metric.

Pad: How much padding would you like between your Graphs?

Rows/Columns: How many rows and columns would you like in your dashboard?

Back Color: What color would you like the background to be?

Light Theme: Graphdat support both a light and a dark theme, which one do you prefer?

Ok, now that we have gone over the basics, lets add some Graphs.

The Graphs have drag handles, so you can expand the graphs to the configuration that you want.

Graphs that have the same type of units can be added together.  So if both graphs show percentages, or both graphs show bytes, or both graphs show a number, we can combine them into a single graph.  By hovering over the top or the bottom of the graph, you can see split. 

In the example dashboard we do not have an SDK hooked up, so lets get rid of the application graphs, expand the CPU and MEMORY graphs and add our disk stats and hit SAVE.

Now I have the dashboard that I wanted.  But I want to have one dashboard with just CPU and Memory because sometimes that is all I want to look at.

So from the Dashboard selector, select create a new dashboard.

Configure the layout with just the CPU and Memory graphs, and give the dashboard a good name.  Click Save and you now have two dashboards that you can select from.

If you want to make the cpu-and-memory graph the default dashboard (so when you come back, this is the one that you see).  You can either check the Make Default checkbox in the dashboard editor, or choose it in the Dashboard tab in Settings.

  Graphdat dashboards, just the way you like it.  

Log into your Graphdat account to see the magic.

For support, drop us an email at [email protected] or on twitter @graphdat

We've been getting some feedback lately that you would like our pricing to be more dynamic.  Specifically, there are two particular scenarios that we hear about time and time again:

Spot use:  You only enable the agent when you are tracking down a problem.  While we encourage you to continually run the agent so you get historical statistics (unlimited retention), sometimes spot usages makes the most sense.  Because you are not using it for the whole month, why should you have to pay for it?

Scaling up/down:  Your server numbers increase and decrease with demand, and you have Graphdat agents running on all of the servers (of course you do).  But when you scale up to 15 servers and then back down to 5, you are billed for all 15 servers and that doesn't seem right.

With our current pricing these scenarios could be more expensive than you would like, because you are paying to monitor a server that is not reporting any data.  You are paying for the entire month, but only using a fraction of it.

We have heard your cries and in light of this new demand, we sought to improve our billing with as little impact as possible to those that are not affected, but solving the problem for you as well.

    Graphdat announces hourly billing.

  $0.014 per monitored server hour per month

  No minimum charges

    What does this mean for you?

If you are running a single server continuously for an average month (730 hrs), this will still cost the same as it does today, about $10.  However, if you stop/start Graphdat agents, you will only be charged for what you use.  All good.

But wait, what about the 1 free server deal that you are offering, how does that work?

With the new hourly billing model it is somewhat problematic to offer a single server for free because we now allow you to stop / start machines at will.  Which machine is the free one?  To solve this problem and keep providing our awesome 'free forever' offer, we are also announcing:

    The first 750 hours of every month are free

    Where did that number come from?

We took a typical month (730hrs) and added in some extra free hours to handle the fluctuation in the length of calendar months for a total 750hrs.  We now have a deal that is better than before.  One server can run continuously each month for free, and you have a couple of hours little left over to spend on your other servers.

That's it!  We hope you will agree that this new pricing will be an immediate benefit to existing users and also provide more flexibility as monitoring needs grow.

  Thank you for using Graphdat!

Sincerely,

You asked for it, you got it.  Give a warm welcome to the new Graphdat Python SDK.

The Graphdat SDK for Python lets you instrument your WSGI applications and mix that with a little bit of Graphdat love.

How to Install

If you are using PIP, add graphdat to your requirements.txt file:

graphdat >= 0.2

and install.

pip install -r requirements.txt

(you can install graphdat using pip or easy_install if you want instead)

pip install graphdat

or

easy_install graphdat

Now that you have the graphdat module installed, you need to add it to your wsgi file. You can add the attribute to your application

from graphdat import WSGIWrapper @WSGIWrapper def application(environ, start_response) # your existing application ...

or you can wrap the application instance

application = framework.WSGIHandler() # your existing endpoint from graphdat import WSGIWrapper application = WSGIWrapper(application)

The code will automatically hook into the WSGI object, so if you run this on the same machine which is running the Graphdat agent, you will see which endpoints are being called by your web server in the Request Count and Response Time graphs.

Above is an example of what you will see when you do the most basic instrumentation of your Python application.

Multiple Data Points

Having the http endpoints graphs is very helpful, but the grapdhat sdk also lets you provide more information about the functions running inside each endpoint.

Take a look at this example:

https://gist.github.com/5771637

When you imported the graphdat module and wrapped your application, we include the graphdat property in the environ objects that get generated. You can then use that to instrument your code with begin() and end() statements.

The example above will display a stacked bar graph on your graphdat dill down dashboard, showing you the breakdown between each of the different trace items

Custom Routes Regex

We have some extra love for those of you with some regex of your route URLs. If you want to make your urls a little prettier, you can pass in a list of regex routes and we'll rewrite your URLs for you.

The wrapper object takes in an extra argument:

def __init__(self, app, options=None):

If we add some regex love when we initialize the wrapper (other options are displayed below)

# create a regex that will match our article page /articles/2013/05 options = { 'routes': ['^articles/(?P<year>\d{4})/(?P<month>\d{2})$'] } application = graphdat.WSGIWrapper(application, options)

Start up the server and hit it with some urls like:

http://localhost:1337/articles/2013/05"

http://localhost:1337/articles/2013/01

http://localhost:1337/articles/2011/02

and we will report timings for route "/articles/:year/:" only.

Configuration

Configuration information can be passed to the library on initialization, just like passing in the regex routes.

For other options supported, check out our wiki

E.g.:

options = { 'debug': True, 'messageDump': True, 'routes': ['^articles/(?P<year>\d{4})/(?P<month>\d{2})$'] } application = graphdat.WSGIWrapper(application, options)

We at Graphdat have worked hard to build a product we can be proud of and although we are in a pretty competitive space we feel Graphdat, for what it provides, performs well and excels.  Up to now we've funded the project ourselves and let everyone give it a try for free so we can get the product right.  We've now hit that part of the journey where we will need to  start charging for our service.

We at Graphdat have a philosophy.  We believe performance monitoring should not just be the privilege of the well funded corporate types, but also accessible by the likes of all startups no matter the size.  We make this distinction because early in our own careers (and in fact, it continues today), we found ourselves needing good tools but were stunned by the high prices most decent performance monitoring tools commanded.  For example:

High per-server pricing - We found that per-server pricing was enough that we couldn't install the monitoring on all servers.

Contracts - We were asked to enter into contracts longer than we felt comfortable entering into.

Steep price gradients - The pricing plans scaled up quickly so the cost went up quickly at thresholds which were not really in tune with the growth of our business.

So given this was the landscape we put our heads together, looked realistically at our costs, and came up with what we believe is the best for the customer.  Here's the deal, starting today Graphdat will cost:

$10 per server, all standard features with unlimited history

1 server is free, forever

Yes, that simple.  Check out more on the pricing page.

For example, if you are just starting up and only have a single server or there is only one server you care about then you can use Graphdat, free, for as long as you like.  However, should you want to start expanding your understanding of your infrastructure by adding more servers to your graphs you can do so for $10 per server, as many as you like.  You pay for your service by attaching a credit card to your account, remove it any time, no obligations.

Next, what about our existing users.  We believe strongly in giving back to the community that helped us make the product it is today, and we wanted to make sure our existing users were in a better place with this change.  We decided to give our existing customers the service for free. We will zero rate all of their running agents and give them all of the benefits of being a standard paying customer.  This is our way of saying thank you.

It's hard to find a balance between providing this kind of service while also making it approachable to the widest possible audience.  We at Graphdat feel that better working apps means better experiences on the web, and for that matter, a better performing web in general which benefits everyone.  With this new pricing we'd like to continue this philosophy into the future.

Regards

The Graphdat Team

Graphdat launches embeddable, real time dashboards that you can add to any web page of your choosing.  We know many of you have your own dashboards, your own way of keeping track of what is going on, and we support that, hell we love it!  So make way for embeddable real time analytics and you don't even have to have a paid account, we love you that much.

  That is a real time graph of one of our production clusters (15 servers).  You can click on the graph to see a legend, drag around to see historical data, zoom out to see larger sections of time (with the mouse wheel), etc... It is essentially everything you get within your Graphdat dashboard, except it is embedded in any web page you like.

How to do it?  Very easy.

Sign up at Graphdat.com.  Fill in a couple of fields to get your trial account rolling. Install the graphdat agent on any of the servers you want to track (once you sign up you will get instructions on the agent setup).

With your Graphdat dash running, just click the embed button on the menu

Here's the embed dialog. Notice we have all our metrics on the left.  The one we want is CPU.

We just need to drag+drop the CPU metric to the upper left of the grid...

Once we've dropped in CPU, let's drag the corner so that it takes up the whole display

For this graph we will turn off the header, legend, and slider

To make it fit nicely onto our page we can set the background to transparent

And finally, because our target web page has a white background we will set the light theme

So look and feel are all fine, let's copy the HTML

And here's the snippet that was copied to my clipboard:

<iframe src="https://dashboard.graphdat.com/embed/d10131ee28?e-w=3&e-h=2&e-pad=5&e-header=0&e-slider=0&e-legend=0&e-light=1&e-bg=none&e-sg-cpu=0-0-3-2"></iframe>

(Note: This iframe will be of the default width/height.  You can add width="your width" and height="your height" tags to the iframe and the graphs will resize accordingly.)

Just paste that into the web page of your choice and we are good to go!

Now that we've got the embed working, we need to make the dashboard public so that the embedding can be displayed anywhere without the user viewing the web page to be logged into Graphdat.  To do this we first open settings:

..then select Project on the left, and finally, click the lock icon in order to make the dash public

So as you've seen you can also add other graphs besides just CPU to this dashboard as well as other features but we'll leave that for the reader.

  So you have jumped on the node.js band wagon and things are going well. So well in fact that you now love node, its awesome, you tell everyone how good it is.

  How being able to only write one language on the front end and the backend (well, kind of) makes you so much faster and more productive.

  How you love spending all day in the terminal in your newly found (or newly re-found) text editor, twitter client, hacker news client, and maybe even IRC and now things are great.  You have chosen your shell, your color scheme and made your terminal tweaks and have turned out a bunch of production code.

  And then you do a new release and things slow down.  You don't know where, you don't know what 'slow' means or even what your past performance was.  You just have an email from some manager or some customer who says "things feel slower".  What do you do now?  Where do you start looking?

You turn to your co-worker, he comes from a C# or a Java background, with a rich set of tools for analyzing everything under the sun and ask him what to do and he says, "don't you have tool X"?  You say no.  He says what about Y?  You say no.  He starts laughing and you start putting logging and tracing statements live in your production code to see what is wrong (doesn't that feel gross), node-inspector and the V8 profiler (in production, because its not the same in staging, do you have a staging?), or you start reading about dtrace and realize your not on a illumos-based systems.  You're a sad panda.

There is a better solution, and we here at Graphdat have felt your pain, we started down the nodejs path and ran into the exact same problems.  So we build Graphdat.  We use it to tune our code and we think its so great, we want to help you optimize your code as well.

Graphdat, is the simplest way to instrument your code and provide you with amazing looking graphs of what is really going on inside your code.  We also correlate that with what is happening on the server so you can view it all together in once place.  Isn't it pretty!

Lets get started.

  First we need to signup with Graphdat  (https://dashboard.graphdat.com/signup) so we have somewhere to put all of those metrics.  It's quick and painless, an email address and a password, how much simpler could you get.

Next, lets get the agent up and running on your server (or your local development box).  The SDK (http://www.graphdat.com/how-it-works) we will be installing will talk to the agent, who then sends everything to your graphs as a single package (server metrics and your code metrics).

Now the fun part starts, we need to start instrumenting your application.

  First order of business is installing the graphdat npm module.  It's nice and simple, just a quick one liner on the command line:

npm install graphdat

Now we are ready to write some code.  The example always given for node.js is the simple webserver, so lets create that:

https://gist.github.com/4662306

run curl against it a couple of times to generate some results (or just use your browser)

curl localhost:8080

And success, we can see the response (graph in the upper right).  With only that one line of code, Graphdat created a hook in your code and now you get results with no extra effort.  Sweet!

Next up, we are going to get a little more crazy, lets add in some FILE I/O and measure how long it takes to serve up that file.

First we need to create a file, we can use some linux foo to create the file (or just create one by hand if you are some inclined).  This will generate a nice 20MB file for us to test with.

yes “we love graphdat, yes we do.  We love graphdat how \'bout you″ | head -c 20000000 > test.txt

Now what we have our large file (things happen so quickly when you are running locally don't they?), we can build our new webserver.

https://gist.github.com/4662537

This time, its probably better to hit the page in your browser as it will fill up your console pretty fast (http://localhost:8080).  Now go back to Graphdat and look at our Application Response Time (same graph in the upper right).  Double click the graph, click on the new peak and your HTTP Request shows up, click on that and you get a break down.

Look a that, a total of 539ms for that request, 424ms taken to read the file and 114ms to write that to the client.  Pretty cool.  

Now if this was your website, you could get a breakdown for how long blocks of code took to run and you would have it historically, so you could compare yesterday to today, or last month to this month.  No more wondering, you now have the hard evidence.

For more information checkout:

Graphdat