#logmonitoring

Glances is an open source real time Linux server monitoring system that aims at providing as much as information possible in as minimum as space possible. Glances is like top command but more refined with many more features & much more real time information of your Linux system. Glances provide information regarding memory, CPU, Disk IO , file system, Uptime, processes, interfaces, alerts & many other system information. One of the best features of Glances is that it can work in server/client mode. We can perform remote monitoring of the systems using a Web interface or through terminal/CLI & we can also export stats to a file. In this tutorial, we are going to discuss the installation of glances monitoring tool on CentOS/RHEL as well Ubuntu machines & also how to use it for real time Linux server monitoring of our system. Recommended Read: Complete guide to install Zabbix on CentOS7  Also Read: Easy guide to setup Nagios monitoring using SSH (check_by_ssh)

Prerequisites

Glances is written in python, so it requires that we have python installed on machines. If its not installed, we can install it using the following command, On CentOS $ sudo yum install python python-devel On Ubuntu $ sudo apt-get install python python-dev Also read our detailed article on How to install python 3.6 on Linux machines.  

Installation of Glances monitoring

There are three ways with which we can install glances on CentOS/RHEL & Ubuntu systems. Either using system repositories or we can use pip (python package manager) or we can also use an official script for installation. PIP method & official script method is same on both Ubuntu & CentOS/RHEL. Its recommended that we either use PIP or Official script to install Glances as repositories might hold older versions but none the less, let's discuss all methods one by one... CentOS/RHEL installation On CentOS, glances packages are available with the EPEL repository. Install EPEL repository with the following command on your machine, RHEL/CentOS 8 $ sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm RHEL/CentOS 7 $ sudo rpm -Uvh https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm RHEL/CentOS 6 (64 Bit) $ sudo rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm RHEL/CentOS 6 (32 Bit) $ sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm Once installed, run the following command to install glances tool, $ sudo yum install glances This will install glances tool on your machine.   Ubuntu Installation Glances is available with the default Ubuntu repositories, so we can install glances easily on Ubuntu systems with the following command, $ sudo apt-get install glances   Installation of Glances monitoring tool using PIP If you have PIP installed on your system, than install glances on both CentOS/RHEL & Ubuntu systems using the following command, $ pip install glances If you don't have PIP installed on your machine, read our detailed article on How to install PIP on Linux systems.   Installation of Glances monitoring tool using the Official script This is another way with which we can install the latest version of glances on our CentOS/RHEL & Ubuntu systems. Open terminal & execute the following command, $ sudo curl -L https://bit.ly/glances | /bin/bash This will automatically install the glances on our system. We can also use following command in place of the one mentioned above, both achieve the same goal, $ sudo wget -O- https://bit.ly/glances | /bin/bash   Using Glances in standalone mode Once the glances tool has been installed on our machine, we can use it for real time Linux monitoring by opening the terminal & running the following command, $ glances

  Running Glances remotely To remotely monitor the server from another machine, open terminal & run the following command, $ glances -s you should get the following output Glances server is running on 0.0.0.0:61209 Now open the remote machine from where you want to monitor the server & run the following command from the terminal, $ glances -c 192.168.1.100 where 192.168.1.100 is IP address of the server to be monitored.   Running Glances in Web server mode To start the Glances in webserver mode, execute the following command from terminal, $ glances -w Once done, open the web browser & enter the following URL to access the Glances monitoring using web browser, http://192.168.1.100:61208

새로 올린 배치 서버는 aws ec2 인스턴스에 올라가 있고, 그 인스턴스에 docker로 filebeat을 띄어놓았다. 그리고 기존 상용 서버에는 docker가 아닌, 그냥 데몬으로 filebeat을 실행해놓았다. 

일단 docker로 올린 filebeat은 임의로 종료되는 케이스가 아직까지 없다. 반면 데몬으로 돌고 있던 filebeat은, 퇴근 전 실행중인것을 확인하고 다음날 출근해서 확인하면 죽어있다. 

-> 이것도 확인해봐야 하는 문제 

그리고, 다시 filebeat을 띄우면, 그 동안 수확하지 못하고 쌓여있던 로그들을 긁어서 조금 많은 양의 데이터를 kafka로 보내기 때문인지, filebeat은 아래와 같은 로그 메시지를 찍는다.

2019-11-21T10:32:37.813+0900 INFO kafka/log.go:53 producer/broker/1 maximum request accumulated, waiting for space

정확히 어떤 의미일까? 구글링 구글링.

일단, 에러는 아님.

This is not an exception but an info message. kafka client's buffer is full, and it's probably waiting for some ACKs before sending more messages.

filebeat의 송신 버퍼가 full이라는 뜻.

The message is logged because the send buffers are full. The kafka client is waiting for ACK from kafka for an older batch in order to flush it's buffers. So network of kafka itself might create some backpressure. All in all data will not be lost, as filebeat will retry until there is enough space. 

내 설정을 확인해 보자.

1. producer인, filebeat 설정.

output.kafka:    ...    required_acks: 1    compression: gzip    max_message_bytes: 1000000

2. kafka broker 설정. (server.properties)

num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM) socket.request.max.bytes=104857600