Syoncloud Logs 0.2 - Community Edition
Copyright (C) 2012 Syoncloud
Syoncloud Logs processes log files from various applications and many servers. It enables to capture business relevant information from everyday log files generated by web servers, business applications and back office applications. It uses Flume sinks that run on the machines that produce log files. This data is filtered and relevant events channeled to HBase. HBase NoSQL database is used for actual data analysis. The number of HBase nodes depends on the amount of processed log files. Syoncloud Logs has easy to use installer that includes all necessary components such as Hadoop, Flume, Hbase and Zookeeper.
Syoncloud requires Redhat (Fedora, Centos) style operating system.
It requires Oracle Java JDK 1.6 and higher.
There should be at least 1GB disk space on local machine.
Root or sudo access is required just for installation.
Make sure that no HBase, Hadoop or Zookeeper instances are running on the local machine. In case you want other instances to be running you have to change default ports during installation of Syoncloud.
Fixes of installer and scripts regarding SSH configuration.
Installation of Kerberos.
Make sure that Oracle JDK 1.6 is installed and path is set to java binary. You can use command "which java"
Under sudo or root start installation by: java -jar ./syoncloud-0.2.jar . If you get error message that xterm can not be open you have to use root account.
Accept license and select path to local JDK.
Select installation path for Syoncloud. All necessary files will be installed in this directory.
Select pseudo-distributed installation to run all servers on local machine. This mode is intended for evaluation and debugging. For production distributed mode should be used.
Syoncloud servers can run as a daemon (service) or regular applications.
Log processing is done by default by HBase but there is an option to use Hadoop Map/Red as well.
Enter operating system user name that will be used to run all Syoncloud servers.
Enter and repeat secure password for given Linux user.
Enter host name of the local machine and name of local node of Syoncloud.
There is a list of default ports for HBase, Hadoop, Flume and Zookeeper. You should change the ports in case some of the ports are already used by other running instance of some of the applications.
Continue to next page and select what type of log files will be processed. There is an example of apache log file with certain structure ($INSTALL_PATH/syoncloud_data/example_log/access_log). Syoncloud can process given log file and store record in 'apacheExample' table in HBase.
Make sure there are no errors during "Processing" stage of the installation.
You can start all Syoncloud servers by logging in as Syoncloud user and in its home directory there is a start.sh script. You will be asked to permanently add generated RSA key to local machine to known hosts list . You should answer yes and hit enter.
You can find index.html file in the installation directory. If you view this file in browser you can see links to webUIs of installed servers.
You can check log files of all Syoncloud servers in installation directory under logs sub-directory ($INSTALL_PATH/logs).
During the first start of Syoncloud configuration and data are created for Flume, Hadoop and HBase. You can run given process manually by script populate_syoncloud.sh .
Testing, Implementation, Production
By default apache example log processing is activated. You can look in HBase table 'apacheExample' and examine records. "hbase shell", "scan 'apacheExample'"
