#graylog2

After many years of hard work, graylog2 latest version 1.0 is now available for download, give it a try!

In this post, I will show you how to implement semantic logging with graylog tools.

Introduction If you have already read my other posts about ELK stack, you will be familiar with graylog. Graylog is another set of tools that are similar to ELK in some ways. (but still very different…)

Graylog comes in several parts, there is:

Graylog server: which is intended to receive and manage logs

Graylog web interface: which is intended to give the users a browser experience and navigation within elasticsearch and graylog server configuration

When you install graylog, you need to have a few things up'n'running!

Mongodb

Elasticsearch

Java JRE

You may say, but why graylog? If it is built on top of elasticsearch, why use graylog instead of kibana & logstash? If you are asking that, it is a good question. But still, graylog is a good platform to implement semantic logging. I've been using graylog profesionnaly for almost two years and it is working perfectly. It is enough for the needs of my business actually. But… logstash and kibana are offering a few more things that graylog do not have:

More inputs

More flexibility

More logs analysis dashboards

…

And, if you want, in a professionnal environment, you can have ELK and graylog collaborating to leverage the logging of your enterprise! (see this schema http://www.flickr.com/photos/jpmens/7709398562/sizes/l/in/photostream/)

GELF If you read my other posts about ELK setup & configuration, you already heard about GELF. GELF stands for Graylog Extended Log Format. I love GELF. It simplifies a lot the management of the data structures of the logs. And, as you can see, GELF input is available in ELK stack, proof that it is a really good input!

In the earlier versions of Graylog, you had to install and configure every graylog components manually. In the latest version, there is a graylog2-setup tool that we will use today to simplify the process!

Enough talking! Let's create our graylog services!

Prerequisities

Virtual machine hypervisor (personnaly, I’m using Hyper-V provided for free with Windows 8+, but VMWare or VirtualBox or other hypervisors will do)

Ubuntu Server 14.10 x64 ISO (site link | download link)

Note: I had some issues with my virtual machines running the graylog-setup. Be sure to have at least 1Go RAM + and 2 cores+. If not, graylog-setup may not be able to complete successfully.

Server installation

Ubuntu server 14.10 install

Fresh & Clean Install of Ubuntu Server 14.10

Install openssh server in the installer

Note that this installation is for testing purposes only.

We need to download and install all dependencies for graylog, let's do this (easy way with debian packages)!

sudo apt-get install mongodb sudo apt-get install openjdk-7-jre

Now that we have all the dependencies installed, we can install and configure graylog.

wget https://packages.graylog2.org/releases/graylog2-setup/graylog2-setup-0.92.4.tar.gz tar -zxf graylog2-setup-0.92.4.tar.gz

Start graylog setup with this command.

cd graylog2-setup && ./graylog2 setup

Goto http://<ip address>:10000/ and go through all the configuration steps (see documentation there: …)

When the setup is completed, you have a last thing to do to be able to use graylog. You have to create the GELF input to be able to send GELF messages through UDP to graylog. Let's do this!

Go to http://<ip address>:10100/, 

Login to graylog

Go to system

Go to Inputs

Select GELF UDP, launch new input

Title: GELF UDP INPUT

Launch!

Test & enjoy!

You are now ready to use your GELF input and your graylog setup! Congratulations. If you need code to test this setup, take a look at my other post "Integration Time" or take a look at my code available in Github in this repository.

Links

How to: logstash (or graylog?) vs nxLog to collect event logs and csv logs #development #computers #fix

logstash (or graylog?) vs nxLog to collect event logs and csv logs

I am current investigating the possibility to consolidate logs from multiple servers using logstash (or graylog2).

I am still a bit confused about the difference logstash and graylog. So far I appreciated the ease of use of logstash, but I would be interested in hearing the experience from other people.

Futhermore, it appears…

Misc

GrayLog2

As Splunk-like tools lover I was very happy to see that Graylog2 was opensourced and is free to use. In case you are seriously thinking about application monitoring the Graylog2 could be the right choice. Must have.

Arduboy

This week was also in sign of arduboy project which looks fantastic. I'm very curious of the future practical usage of toys like this.

iOS

Alcatraz

And here are great news for the all Xcode users. Xcode plugin manager alcatraz was released for Xcode5! Plugin manager itself is great but the project website is also deserve my respect. Nice!

Shimmer

I have been quite busy for a long time doing work that I can not write about here but lately I have found time to explore a bit.

When it comes to logging on Windows and especially central logging it can be quite a bit of pain. There are excellent products out there like Splunk that has plugins to nearly every product you care about but it can get very expensive very fast. There is a free version that people can use that is limited to 500mb of logs a day, for some systems that is plenty but as soon as you start to push logs from more than one system those 500mb start to run it very quickly.

There are other commercial options out there that live in the same space, Envision and Arcsight are the biggest players. My favorite of the small players is Kiwi Syslog Server, easy to set up and very cheap.

In the open source area there are two products that everyone loves and I can understand that very well because of the improvements they have made this year.

Graylog2 can easily compared to Splunk in many ways but it runs only on Linux.

Logstash on the other hand is the product I have fallen in love with. Its a java application which in this rare occasion is a good thing. Being a java application means that the only requirement is the latest version of java, yes this runs easily on Windows.

Logstash as a solution is built on three products Logstash, Elasticsearch and Kibana that last week were finally joined under the Elasticsearch company.

Logstash handles incoming logs, parses them, filters them and outputs them to Elasticsearch

Elasticsearch stores the logs in such a way that it can scale out to insane levels. There are companies gathering over 100gb of logs a day and storing them for months. This is also a Java application.

Kibana is the web front end to Logstash data stored in Elasticsearch. The latest version is plain html+js that talks directly to elasticsearch requiring only a simple webserver to run.

The only problem I have with Logstash is its lack of access control. If you have access to the elasticsearch ports you can access all the data stored there. You need to do the access control on the webserver level if you need to. This is the only area where I feel Graylog2 to be better than Logstash.

Graylog2[http://www.graylog2.org/]

Graylog2 enables you to unleash the power that lays inside your logs. Use it to run analytics, alerting, monitoring and powerful searches over your whole log base. Need to debug a failing request? Just run a quick filter search to find it and see what errors it produced. Want to see all messages a certain API consumer is consuming in real time? Create streams for every consumer and have them always only one click away.