#marahon

Cô gái chạy suốt một ngày đêm hoàn tất 45 vòng quanh hồ Gươm

[ad_1]

Yến sinh năm 1990, làm quen với môn chạy bộ đã 3 năm. Như mọi cô gái làm việc công sở với nhịp sinh hoạt đều đặn, Yến muốn thay đổi để cuộc sống mới mẻ, vui và tốt cho sức khỏe nên chọn môn chạy bộ. Ban đầu, cô gái không chạy nhiều, thường 1-2 vòng công viên đã rất mệt. Không nản chí, Yến vẫn tiếp tục chạy hàng ngày để rèn luyện sức bền. Dần dần cô kết nối thêm với những người bạn cùng…

So, when netflix released vizceral,  I immediately wanted to have the same ability to monitor our system. 

We’re running a ~100 node  mesos cluster with a bunch of microservices, for a POC I’ve used my 10 node rasperry pi 3 cluster

Since all our microservices are started using marathon and are in docker and mostly doing web We decided to create a system which will automaticly monitor running dockers started by marathon and display the animated graph on vizceral. 

Some guys already made similar system but on docker swarm so We’ve decided to take a similar route. 

Which is.. launch an docker  on all slaves (or agents from 1.0)  which monitors which dockers are running, If the docker is started from marathon start a packetbeat docker to monitor it. 

At the end all slaves were sending packets from all marathon started services which were tagged as task name from marathon into the elasticsearch. 

Then comes the create a json for vizceral part. Agh. 

Since packets were tagged by marathon task id I knew what came into the docker but I didn’t know where it came from (since many services live on the same host and apps were behind docker NAT)

When reading release logs from mesos 1.0 I saw It now support CNI which could come in handy - start containers on separate ip’s and then I would know where the packes came from.  Docker also has this ability now: to create macvlan networks - so we went this route.   In 1.12.1 docker version macvlan is not in global scope, so recompiling was needed but after that It all came together. Tasks were started all on separate ips. 

Then We needed to create a “scraper” service which creates a json for vizceral in order to display the graph.  We took all apps from marathon and query elasticsearch for packets which had a direction “IN” and group by service name.  Then All client_ips that came into that  service we could match by service name from marathon. 

In the video below you can see how traffic starts flowing when I refresh the web app.