tagomoris in English

Logstash is an open source software for log management, which is widely known and used as a part of the ELK stack. There's a great repository collection with many plugins for Logstash to collect, filter and store data from many source, and to many destinations, but it doesn't have a plugin to store data into Treasure Data Service.

So, I created it.

Repository on Github

Released gem

How to install / use the plugin

You can install that plugin very easily if you're already using Logstash.

$ cd /path/of/logstash $ bin/plugin install logstash-output-treasure_data Validating logstash-output-treasure_data Installing logstash-output-treasure_data Installation successful $

Next, configure Logstash with Treasure Data services. It requires name of database and table to be inserted, and API Key which can be checked on Treasure Data Console.

input { # ... } output { treasure_data { apikey => "0/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" database => "dbname" table => "tablename" } }

Then, launch Logstash with that configuration file.

$ bin/logstash -f your.conf

You'll get rows on Treasure Data console. Log message texts are stored in message column, and some additional columns will exist (ex: time, host and version).

Specifications / Configurations

Currently, this plugin store all logs into just a table. Use 2 or more sections for treasure_data in configuration file if you want to insert data into 2 or more tables.

And this plugin buffers data in memory buffer in 5 minutes (in longest case). Buffered data will be lost if Logstash process crashes. (For further option, see "Combination with Fluentd" section.)

This plugin have configuration options listed below. The plugin will work with default values for almost all cases, but some of these might help you under unstable network environments.

apikey (required)

database (required)

table (required)

auto_create_table [true]: plugin will create table if not exists on Treasure Data

endpoint [api.treasuredata.com]

use_ssl [true]

http_proxy [none]

connect_timeout [60s]

read_timeout [600s]

send_timeout [600s]

Combination with Fluentd

For now, logstash-output-treasure_data has very limited feature, especially for buffering, stored table specifications and performance.

There's an another option to use Fluentd for more flexible and high performance transferring. We can use logstash-output-fluentd to do it.

Github repository

Released gem

[host a]-> (logstash-output-fluentd) -> [host b]-> (logstash-output-fluentd) -> [fluentd] -> (fluent-plugin-td) -> [Treasure Data] [host c]-> (logstash-output-fluentd) ->

Many Logstash can be configured to send these logs to a Fluentd node, and that Fluentd stores whole data into Treasure Data.

# Configuration for Logastash input { # ... } output { fluentd { host => "your.host.name.example.com" port => 24224 # default tag => "td.database.tablename" } } # Configuration for Fluentd <source> @type forward port 24224 </source> <match td.*.*> @type tdlog apikey "0/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" buffer_path /mnt/fluentd/buffer/td </match>

Fluentd tdlog plugin can store data into many database-table combinations by parsing td.dbname.tablename. So you can configure any database/table pairs in Logstash configuration files if you want.

Conclusion

Last weekend, I went to Europe again to attend [EuRuKo 2015](http://www.euruko2015.org/) and to do a presentation which was accepted. My trip was sponsored by Treasure Data, and organizer payed my hotel charge :D

It was really well organized, developer centric and impressive conference. I talked with many people, developers from various area in Europe, and all over the world.

My talk was about "Data Analytics Service Company and Its Ruby Usage".

Data Analytics Service Company and Its Ruby Usage from SATOSHI TAGOMORI

I talked about what modern data analytics company is, and how/where we're using rubies (CRuby and JRuby). I got some feedback that I should have talked more about our requirements and motivations to make audience to be known why we're creating various middleware. At next RubyKaigi 2015, I'll do the revised one in better way :P Anyway, there were some discussions about data analytics and its platform designs, middleware and performance problems with many developers.

As a participant, I enjoyed the event, many great talks and presentations, including which isn't releated with ruby :-) Every conferences what makes all audience happy have great organizers as far as I've seen. EuRuKo 2015 does too.

I must mention about the great city, Salzburg. It was extraordinary impressive city with tiny beautiful old town and large unforgettable castle.

2 weeks ago, I went to Germany, and attended to JRubyConf.EU and Eurucamp 2015 in Potsdam, to have a talk session. It's my first trip to Europe, and my longest flight ever.

http://2015.jrubyconf.eu/

http://2015.eurucamp.org/

The conference (JRubyConf.EU is actually a part of Eurucamp) was organized extraordinary well. I was welcomed well, and had no troubles in whole trip.

JRuby with Java Code in Data Processing World from SATOSHI TAGOMORI

I did a presentation about OSS, Norikra and Embulk, which uses JRuby as a plugin layer language on a top of Java middleware/library. My English speech is not so good, but great audience are highly motivated, and got known what I want to tell, and asked some questions to me after session. That's really great experience for me.

JRubyConf.EU concentrated about tech points around JRuby, and Eurucamp seems to make itself to be for developer community and culture around Ruby developer community. It's reflects the rich diversity of Europe countries, and looks unique for me from Japan.

At 3 Jun 2015, we had a meetup about Norikra at Shibuya, Tokyo. There were about 100 attendees who are interested in stream processing with SQL, and skillful 4 (+me) presenter talked about their experience to deploy & manage Norikra server for their own workloads.

メルカリでのNorikraの活用、 Mackerelを添えて from Masahiro Nagano

mercari is the company of C2C market app service on mobile device. @kazeburo talked about their experience about Norikra and Mackerel.

pixiv provides a SNS service, based on communication over illustrations. @harukasan is an user of Norikra in very early stage, and talked about why stream processing is needed, and how we can use Norikra.

Kayac is the very unique company, and they develop/release many web services and games for a long time. @fujiwara talked about their usage of Norikra, especially for spam detection and reaction with stream processing.

gunosy is a news app on mobile device, and they also have their own Ad service. @shunsukeaihara talked about why and how they're using Norikra for their Ad service improvement.

Norikra Recent Updates from SATOSHI TAGOMORI

And then, I talked about recent features and updates of Norikra, especially between v1.0.0 and v1.3.1 (latest), for example, suspended queries, NULLABLE fields, listener plugins and dynamic plugin loading.

I got a report that there is a vulnerability for Man-In-The-Middle attack(MITM) for fluent-plugin-secure-forward. I updated and released fluent-plugin-secure-forward v0.3.0. It is strongly recommended to update this plugin for all users of it.

UPDATE: Please use v0.3.2 or later, and update shared_key when you update your deployment, to prevent replay attacks for shared_key_digest. It's not so serious problem because fake data make no sense for many users, but to be considered.

This update newly requires one of these below:

a server certificate issued by public trusted CA, private key and its passphrase

a private CA certificate, private key and its passphrase

fluent-plugin-secure-forward v0.3.0 has incompatibility about configuration parameters, and fails to start with configuration for older versions. Please read this article carefully, and update both of plugin and your configuration as soon as possible.

Using certificate issued by public CA

If you have valid SSL server certificates issued by public and trusted CA, you can use it on input plugins.

In this case, output plugin doesn't require any shared files (like private CA certificates), but required to be configured to verify FQDN (fully qualified domain name) of server for input plugins.

https://github.com/tagomoris/fluent-plugin-secure-forward/#using-ssl-certificates-issued-from-trusted-ca

Using private CA and automatically generated server certificate

fluent-plugin-secure-forward provides SSL communication to users who doesn't have certificates issued by public CA (and valid domain name for fluentd nodes), by using private CA.

Updated plugin includes a utility command to generate private CA certificate and key pair. So once you deploy private CA certificate on server for output plugins, they can verify certificates from servers for input plugins.

https://github.com/tagomoris/fluent-plugin-secure-forward/#using-private-ca-file-and-key

How to configure plugins for insecure communication

Using certificates requires to change how to deploy your Fluentd server. So fluent-plugin-secure-forward allow users to communicate in insecure way. In fact, that is almost same with older versions of fluent-plugin-secure-forward.

This feature requires to be configured as secure no for both of input and output plugins. With this configuration, input plugin generates self-signed server certificate and use it, and output plugin doesn't verify certificate at all. This configuration has serious MITM vulnerability. So it's strongly unrecommended.

After the release of Norikra v1.0.0, we are continuing to develop Norikra to add many good features and to improve to help performance problems.

Today, we released Norikra v1.3.0. This relase has many improvements and updates just like previous versions.

Now, Norikra have many improvements after v1.0.0, like below:

Pluggable Listeners to process output events from queries directly

Suspending running queries and resume them

NULLABLE(field) specifier to process missing fields forcely as NULL

Dynamic plugin reloading by SIGHUP

Update bundled Esper to 5.2.0

This March, I left LINE Corp, and joined Treasure Data as a Software Engineer. I'll try to improve Treasure Data platform, and also continue to develop Fluentd & many OSSs.

As many people know, parsing user-agent string is very confusing work. There are many parser library, and these return different results from each other.

So, we wrote woothee. It is a brand-new user-agent parser project, that have a common testset written in yaml, and also have many implementations in each languages.

The most important point is that these implementations return just same result as far as a single testset ensures.

http://woothee.github.io/

https://github.com/woothee/woothee

Woothee implementations are:

Java (and Hive UDF)

Ruby

Perl

Python

PHP

Node.js (browser / node)

Go

Woothee's concept is very simple: argument is a string of user-agent, result is a object of key-value pairs(Hash, Map, Object, ...).

For example:

// in Java // import is.tagomor.woothee.Classifier; // import is.tagomor.woothee.DataSet; Map r = Classifier.parse("user agent string"); r.get("name"); r.get("category"); r.get("os"); r.get("version"); r.get("os_version"); # in Ruby require 'woothee' Woothee.parse("Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)") # => {:name=>"Internet Explorer", :category=>:pc, :os=>"Windows 7", :version=>"8.0", :vendor=>"Microsoft", :os_version=>"NT 6.1"}

We've maintained this library for few years, and will keep continuing from now on too.

Last weekend, I visited Taipei to perticipate in HadoopCon 2014, and to do two talks, including a keynote talk.

At first, I submitted my proposal for afternoon tech talk. And then, the organizers invited me as a keynote speaker, and I accepted. It was a kind of big challenge, but also a great experience for me at the same time.

I think that I talked in keynote session successfully, with presentation slides that my co-workers of LINE corporation cooperated with me to make it.

Lambda Architecture Using SQL from SATOSHI TAGOMORI

In that afternoon, I did a tech talk about Norikra. That was the first presentation about Norikra outside of Japan!

Norikra: Stream Processing with SQL from SATOSHI TAGOMORI

I've got some questions after presentations, so I think that my presentation was not so bad at least :)

Conference

HadoopCon 2014 was very well-organized conference. And many developers perticipated in each sessions with enthusiasm (even in morning session!).

As same with Japanese Hadoop community, there were not so many talks about Hadoop itself in HadoopCon 2014, and many spearkers' topics (including mine) were next steps of Hadoop/MapReduce.

I chouldn't see many of talks, but I chould talk with many developers in Taiwan. I knew about many points in common between Japanese and Taiwanese developer communities, and also about many points of difference. That was really fantastic experience.

Sightseeing

Before/after the conference, I visited many sites. Some local developers showed us (I & repeatedly) around. Really thank you for great guide!

I saw many monuments like Taipei 101 and National Chiang Kai-shek Memorial Hall, ate great foods,

ate great foods,

and ate great foods.

Of course, I got many chances to drik Taiwan beer :)

Thank you!

I really enjoyed 3 days in Taipei. In fact, I'm very sorry that 3 days are too short to see many wonderful places in that city.

I visited Singapore last week for RedDotRubyConf 2014, and did a talk about Fluentd.

Fluentd: Data streams in Ruby world #rdrc2014 from SATOSHI TAGOMORI

That was my first talk in English, but I did it successfully (for me :-) with great helps by organizers. It was a really great experience to talk about ruby and its products between skillful rubyists from all over the world, in front of many rubyists in Singapore and from many area of the world.

After that talk, I accepted some positive response and tweets for Fluentd! I'm looking forward to see their impressions and feedbacks. I want many more people to use Fluentd, and I believe that Fluentd can make their life more simple and fun :)

RedDotRubyConf is a really fantastic conference, well organized by @winstonyw and other great rubyists.

I've just released Norikra v1.0.0, which is a server software to provide schema-less stream processing with SQL.

I believe that this is the first software which realize user-friendly stream processing. In fact, we can deploy norikra server in 3 minutes! (How long we must struggle for the first test program on Storm?)

http://norikra.github.io/

We can send any events (key-value pairs, can be nested deeply) to Norikra server, and can execute SELECT queries for these events like standard SQL.

Any programmings on frameworks are not required. We should only write SQL and register it (on WebUI, over RPC or by CLI). And then, send input events into norikra server. And we can add/remove queries anytime.

Norikra supports complex events which have nested lists and key-value pairs, which is accessed on SQL directly, without any schema definitions.

And Norikra v1.0.0 supports many more features what you wants! (User-defined functions, JOINs and subqueries, loopback query group, ...)