#semantic-logging

After many years of hard work, graylog2 latest version 1.0 is now available for download, give it a try!

In this post, I will show you how to implement semantic logging with graylog tools.

Introduction If you have already read my other posts about ELK stack, you will be familiar with graylog. Graylog is another set of tools that are similar to ELK in some ways. (but still very different…)

Graylog comes in several parts, there is:

Graylog server: which is intended to receive and manage logs

Graylog web interface: which is intended to give the users a browser experience and navigation within elasticsearch and graylog server configuration

When you install graylog, you need to have a few things up'n'running!

Mongodb

Elasticsearch

Java JRE

You may say, but why graylog? If it is built on top of elasticsearch, why use graylog instead of kibana & logstash? If you are asking that, it is a good question. But still, graylog is a good platform to implement semantic logging. I've been using graylog profesionnaly for almost two years and it is working perfectly. It is enough for the needs of my business actually. But… logstash and kibana are offering a few more things that graylog do not have:

More inputs

More flexibility

More logs analysis dashboards

…

And, if you want, in a professionnal environment, you can have ELK and graylog collaborating to leverage the logging of your enterprise! (see this schema http://www.flickr.com/photos/jpmens/7709398562/sizes/l/in/photostream/)

GELF If you read my other posts about ELK setup & configuration, you already heard about GELF. GELF stands for Graylog Extended Log Format. I love GELF. It simplifies a lot the management of the data structures of the logs. And, as you can see, GELF input is available in ELK stack, proof that it is a really good input!

In the earlier versions of Graylog, you had to install and configure every graylog components manually. In the latest version, there is a graylog2-setup tool that we will use today to simplify the process!

Enough talking! Let's create our graylog services!

Prerequisities

Virtual machine hypervisor (personnaly, I’m using Hyper-V provided for free with Windows 8+, but VMWare or VirtualBox or other hypervisors will do)

Ubuntu Server 14.10 x64 ISO (site link | download link)

Note: I had some issues with my virtual machines running the graylog-setup. Be sure to have at least 1Go RAM + and 2 cores+. If not, graylog-setup may not be able to complete successfully.

Server installation

Ubuntu server 14.10 install

Fresh & Clean Install of Ubuntu Server 14.10

Install openssh server in the installer

Note that this installation is for testing purposes only.

We need to download and install all dependencies for graylog, let's do this (easy way with debian packages)!

sudo apt-get install mongodb sudo apt-get install openjdk-7-jre

Now that we have all the dependencies installed, we can install and configure graylog.

wget https://packages.graylog2.org/releases/graylog2-setup/graylog2-setup-0.92.4.tar.gz tar -zxf graylog2-setup-0.92.4.tar.gz

Start graylog setup with this command.

cd graylog2-setup && ./graylog2 setup

Goto http://<ip address>:10000/ and go through all the configuration steps (see documentation there: …)

When the setup is completed, you have a last thing to do to be able to use graylog. You have to create the GELF input to be able to send GELF messages through UDP to graylog. Let's do this!

Go to http://<ip address>:10100/, 

Login to graylog

Go to system

Go to Inputs

Select GELF UDP, launch new input

Title: GELF UDP INPUT

Launch!

Test & enjoy!

You are now ready to use your GELF input and your graylog setup! Congratulations. If you need code to test this setup, take a look at my other post "Integration Time" or take a look at my code available in Github in this repository.

Links

In this post, I’ll present how to setup a new machine running Ubuntu to make the ELK stack work!

Prerequisities

Virtual machine hypervisor (personnaly, I’m using Hyper-V provided for free with Windows 8+, but VMWare or VirtualBox or other hypervisors will do)

Ubuntu Server 14.10 x64 ISO (site link | download link) 

Ubuntu server 14.10 install

Fresh & Clean Install of Ubuntu Server 14.10

Install openssh server in the installer if you want to connect via SSH

Note: This installation is for testing purposes only. This setup should not be directly for production use. Please, refer to elasticsearch documentations for production use installations.

Preparation 

Let me be personnal here, I mostly use TAR packages instead of DEB packages when I start with something new. I prefer mastering a technology manually to understand how to install, what is the prerequisities and how the configuration works instead of letting the OS do all the job with DEB pacakges! With these words of wisdom, lets begin!

We need to download and install all dependencies for ELK, let's do this (easy way with debian packages)!

sudo apt-get update

sudo apt-get install openjdk-7-jre

Go to user's home folder to download ELK

cd

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.tar.gz

wget https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz

wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.0-beta3.tar.gz

 Decompress all files downloaded.

tar -zxf elasticsearch-1.4.2.tar.gz

tar -zxf logstash-1.4.2.tar.gz

tar -zxf kibana-4.0.0-beta3.tar.gz 

At this step, you have all the files ready to start the ELK services. We will use some commands in the following steps, let's review them:

nohup: enable a command to live after a session is closed

&: silenced command (will not interfere with active session)

Elasticsearch

Documentation link: http://www.elasticsearch.org/overview/elasticsearch/

Let's begin by starting elasticsearch service. 

nohup elasticsearch-1.4.2/bin/elasticsearch &

To see if everything works fine, wait the first time for up to a minute to let everything set-up properly and connect to elasticsearch web api. Go into your favorite browser and type the following address:

http://<ip address>:9200 

You should have something like this:

{

  "status" : 200,   "name" : "Marrina",   "cluster_name" : "elasticsearch",   "version" : {     "number" : "1.4.2",     "build_hash" : "927caff6f05403e936c20bf4529f144f0c89fd8c",     "build_timestamp" : "2014-12-16T14:11:12Z",     "build_snapshot" : false,     "lucene_version" : "4.10.2"   },   "tagline" : "You Know, for Search" }

You have now a working elasticsearch node! 

Note: The name of the cluster changes each time you start the elasticsearch service.

Logstash

Documentation link: http://www.elasticsearch.org/overview/logstash

Next step, we will prepare and start the logstash service. But, before we begin, we must define a configuration file for logstash. For my sample, I will configure a tcp and a gelf udp appender (I will talk about my lovely GELF in another post =P) that will output to our elasticsearch node.

vim logstash-1.4.2/bin/logstash.conf

Place this content in the configuration file

input {

   tcp { port => 19200 }

   gelf {}

}

output {

   elasticsearch_http { host => localhost }

}

  To start logstash agents, run the following command

nohup logstash-1.4.2/bin/logstash agent -f logstash-1.4.2/bin/logstash.conf & 

The first time, it may take a few minutes to set-up properly. The reason I set-up a tcp input is to be able to see something in elasticsearch. 

After a minute or two type in the command promt the following command, it will send a message to logstash and be stocked in elasticsearch

echo "Hello World!" > /dev/tcp/127.0.0.1/19200 

To see if everything went fine, go into your favorite browser and ask elasticsearch for status information. 

http://<ip address>:9200/_status?pretty=true 

You should have something like this: 

{   "_shards" : {     "total" : 10,     "successful" : 5,     "failed" : 0   },   "indices" : {     "logstash-2014.12.31" : {       "index" : {

… 

To see if it worked well, you should have an index called logstash-<date of the day>. If the index is created successfully, everything should be fine! 

You can also try this URL:

http://<ip address>:9200/<logstash index name>/_search?pretty=true

You should have something like this:

{

  "took" : 3,   "timed_out" : false,   "_shards" : {     "total" : 5,     "successful" : 5,     "failed" : 0   },   "hits" : {     "total" : 1,     "max_score" : 1.0,     "hits" : [ {       "_index" : "logstash-2015.01.07",       "_type" : "logs",       "_id" : "AUrB4z5qWtbYLymf7Zg_",       "_score" : 1.0,       "_source":{"message":"Hello World!","@version":"1","@timestamp":"2015-01-07T00:55:02.607Z","host":"127.0.0.1:40875"}     } ]   } }

Kibana

Documentation link: http://www.elasticsearch.org/overview/kibana

Note: I've downloaded a beta version of Kibana 4. The reason is mainly for the simplicity of the set-up & use. If you want to know how to configure and use kibana stable releases, refer to kibana installation documentation. 

Let's start the kibana web server instance

nohup kibana-4.0.0-beta3/bin/kibana &

It may take a minute or to set-up the first time, be patient ;) After that waiting time, try that URL to see if Kibana is loading.

http://<ip address>:5601

After you load kibana the first time, in elasticsearch, you should have a new index named .kibana. If so, it works ;)

Try this URL:

http://<ip address>:9200/_status?pretty=true

You should have something like this

{   "_shards" : {     "total" : 12,     "successful" : 6,     "failed" : 0   },   "indices" : {     ".kibana" : {       "index" : {         "primary_size_in_bytes" : 2243,         "size_in_bytes" : 2243       },

…

Kibana configuration

The first time kibana starts, it will need a few configurations to work.

For example, you will be asked for a Time-field name, just put the default value @timestamp. 

And… you're done. You have now a complete ELK stack running in gelf udp mode in inbound with an elasticsearch_http outboud. 

If you want to configure more inputs/outputs, please refer to the links provided below. 

Summary 

In this post, we have downloaded and configured all components required to make an ELK stack work. Do not forget that we are in a testing set-up, be sure to read the documentation provided for production use setup. 

In the next post, we will develop a .NET solution that will integrate our GELF UDP appender and see the results in Kibana! 

Semantic logging links

Introduction: Semantic logging

Part 1: What is semantic logging?

Part 2: Technically talking… how can I use it

Part 3: Creating the ELK machine!

Part 4: Integration time!

Part 5: Pitfalls

Links

https://www.graylog2.org/resources/gelf

https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-logs-on-centos-7

http://logstash.net/docs/1.4.2/

In this post, I will present the technicalities that can hide behind logs and how we can resolve it using a technology that I use a lot for semantic logging (ELK stack).

Problem solving - Let's begin simple

When you face a problem in an application, there is a multitude of parameters that need to be analysed in certain situations to pinpoint the real problem. It may be:

an operational issue (server memory, cpu bottleneck, misconfiguration in server, ...)

a network issue (firewall rules, communication lost with party, network capacity exceeded, ...)

a web server configuration issue

a development issue (commonly called: a bug)

a user issue (user entering wrong data: ie.: form data validation)

...

Woah... this is a lot and... I may have forgot some ;) Anyway, some problems are simple to solve. But, in many cases they are way more complicated to be solved without good tools.

Problem solving - First try

Let's think for a minute that you need to solve a complex problem that can be related to more than one option. What does that mean? It means in simple that:

if it is an operational issue, you may need server event logs

if it is a network issue, you may need firewall syslogs

if it is a configuration issue, you may need web server logs

if it is a development issue, you may need application logs

if it is a user issue, I hope you have application logs also! =P

If you are in a "enough big" company (I'm not saying that bigger is better), each of those bullet points is one or more person related. Do you imagine how hard it is to work with, lets say, 6 people, to resolve an authentication issue. It's COMPLICATED!

Problem solving - let's find a tool to rule them all!

It is pretty easy to understand that, without good toolings, it is hard to resolve the issue presented above. Let's present something that come to rescue! ELK stack.

It's a free set of tools that, together, are awesome!

ELK stands for:

Elasticsearch: search & analytic engine

Logstash: log parser, log keeper, log presenter (json)

Kibana: data visualization engine

At first, on their website (elasticsearch.org), it seems simple to set-up but I'll present you a complete set-up using Ubuntu 14.10 (at the date of writing). This sample will takes you up and running with a ELK stack server and a .NET solution using ELK.

Semantic logging links

Introduction: Semantic logging

Part 1: What is semantic logging?

Part 2: Technically talking… how can I use it

Part 3: Creating the ELK machine!

Part 4: Integration time!

Part 5: Pitfalls

Links

http://www.elasticsearch.org/webinars/introduction-elk-stack/

I’ll try, in this post, to simplify and clarify what is semantic logging.

If we decompile these two words, we have:

semantic: give a meaning/signification to something 

logging: in an IT world, to make this simple, logging is related to the information gathering of an IT process

Semantic

Let's say one of your friends is saying to you: "I'm exhausted...". Its not clear at all at this moment what he is exhausted about. It seems an easy process but it is not. What your brain will begin to do is asking for more information to put a real meaning to this phrase.

What you will do is ask for more information to have this kind of answer: "I'm exhausted [because my daughter is always sick]."

Semantic is far more than only that. But, for now, it is enough for you to be able to understand the basis of semantic logging.

Logging

It is hard to explain in which cases we have logging. Logging can appear in many ways:

console logging

event viewer

rolling file logs

syslogs

database logs

... and so on

As I said previously, logging in an IT world is linked to one or more IT processes. For example, you can have logging for a web server (Apache). In this case, you will have lots of processes that will be logged into syslogs rolling files (by Apache default settings). Here is an example:

[Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/htdocs/test

In this log, you have plenty of information:

DateTime of the event

Type of event

information of the requester (ip address)

and a message telling what is the kind of event triggered (and the error that occured)

If you take this event alone, it's pretty easy to understand what it is. It seems that a client has been denied access to a resource.

Summary

Ok... and you may ask, what it semantic logging after all?. It is quite simple, it is something like this: "Give a meaning to logging provided by one or more IT processes".

It is not an easy process of gathering all logs provider in rolling file to see a possible hacker attack for example. With semantic logging, it is the process of simplifying these logs to make them understandable by humans. When logs are well understood, IT decisions can be made with a better overview of what is happening.

This concludes the introduction of semantic logging. I have provided links below for further readings. Feel free to propose related links.

Semantic logging links

Introduction: Semantic logging

Part 1: What is semantic logging?

Part 2: Technically talking… how can I use it

Part 3: Creating the ELK machine!

Part 4: Integration time!

Part 5: Pitfalls

Links

http://msdn.microsoft.com/en-us/library/dn775006.aspx

Hi everyone!

I’ll begin my first posts with something really cool, semantic logging. But, lets do first things first. Let me present myself.

Mainly involved in the .NET platform for the last 6 years, I’ve developed myself around C#, ASP.NET MVC & WebApi and SharePoint. In the last years, I've developed a few “big projects” that were missing a “little something” from a “semantic” point of view.

A year ago, a new fellow came into our team and talked to me about semantic logging: “a power delivered from the gods =P”. Anyway, we tried it and we loved it.

Semantic logging links

Introduction: Semantic logging

Part 1: What is semantic logging?

Part 2: Technically talking… how can I use it

Part 3: Creating the ELK machine!

Part 4: Integration time!

Part 5: Pitfalls