Logon Job

Hi, thanks for joining we'd, be talking today about parco scope and how understanding on your own, your spark jobs on the operating system called the operating system works. While you run your spark jobs how to visualize those data and get better insights when you are planning for your infrastructure needs, so I'm Janice. This is the mandatory. Who am I slide. I'Ve been working in IBM research in Dublin since 2012 was research software engineer. It'S a fairly new lab established in 2011, we're celebrating five years 2016. So, basically, my work is focused around analytics foundations, middleware whatever that means. So it's basically work around distributed frameworks. 

Anything with having to do with Hadoop spark HBase. I love Java and Scala based technologies. I like working on those and, of course getting excited with a web-based proof of concepts. That'S why these kids were needed also to develop a sparkle scope, as you will see. Basically, I'm the UI guy for the high performance systems group led by kostas. That'S why his first andreia hallett Michael mr., find peer and three ok, so the I'm gon na explain the how we get to start developing around spark and trying to improve spark and specifically the the monitoring part of spark, so we just enjoyed so much writing our Analytical work loads on spark. We tried Hadoop and MapReduce, of course, in the beginning, but the performance and the ease of development made it really obvious that we should spin switch quickly to spark. We have fully embraced internally in IBM research in Ireland since the very first versions of 1.0, actually before even the official announcements by a vm, and so so. 

The the target views case that we had was mostly had to do with energy related central data with a big customer in the u.s.. So we have been developing our spark jobs on this large volume of energy-related central data. So, of course, as the type of this huge case required that once we have developed the day required the required jobs right, they were executed, then in a recurring fashion. That'S it because the data were coming. In every day we were receiving a new batch of data. Every day and we had to apply this once we have reached a level of maturity on these jobs, we had to apply the same transformation, the same filtering on a new set of data, so this data, these jobs were executed on a daily basis. So therefore, we had to plan ahead of the we can cut plan ahead for our infrastructure needs, because the data you can, as you can imagine, they were accumulated and they were growing so therefore we're starting to read some the limits of our hardware. 

So, of course, in the process of developing spark, jobs were hitting different types of bugs right. Of course, there was some bugs on our code and now pointer exception, something else which we could very clearly and easily see and discover on the spark default web UI right. Sometimes was also because the data weren't really is strictly structured, they were on one column. It was supposed to be a number and was a string and so on so forth. So using the the web UI the default spark web UI. It was very easy to track down to what exactly states failed in which line number in our source code failed. As I said, we're, starting to experience some slowdowns that had to do with our infrastructure right. We didn't have infinite resources to drown this job right. 

The problem that we had was that by using this, the spark webui we couldn't easily identify at which part of the code, at which stage the bottom link was caused because of because of our hardware because of our of the limitation of our hardware. Also, we couldn't really trace back and see what was the part of the code that were causing this button like what operation were we trying to run when this boat link was happening? Spark, of course, has its money in has its monitoring framework. One of the options that was available to us to get some better understanding on what's happening under the hood in in the in the system level was, of course, the the gangly extension. There was an option for us that we considered in the beginning, but we felt that there are some drawbacks to that. We had we had to maintain and configure yet another external system, and also there was no association of the matrix exported with the actual job or stage or source code of the of the spark job. 

And in the end we like. We really liked the spark web UI the default web UI. We find it hadn't. We wanted to consolidate all of our matrix there. So, as I mentioned, the default spark monitoring framework supports an operation that exports all the all the metrics own CSVs. So this CSVs are exported. You can get metrics about the HDFS, read right about the number of tasks running and there were always in the default. And since we seek rights rights, the those metrics into a local file system - and we found it that it was a bit cumbersome to call all those differences, revised aggregate them make. The Association make the temporal association with the jobs that we're running on our spark lasted at that point, so the whole procedure seemed a bit cumbersome again. It was tough to get those associations with a source code of the job and data, as I explained before, this is very rough. 

Outline of the current monitoring architecture. Hug looks like on spark. You have the separate the separate workers on different nodes and the difference. The different executors being loans on those machines and by using the executor source, which you can also attach a different one, jvm source, which just generates a different type of matrix. It'S written then to a CSV file on the on the local file system. Just to give you so basically, if you look, i run i run a job and specified that i want to use the defaults. Is we think it will look like this on your local file system right? It will have all those different csv files and they will inside that there is the time series and the value there is the time stamp and the value for each of those metrics like the HDFS right bites and, of course the app is the application ID part. So that's why we started developing Sparkle scope and, as my friends make fun of me, that I claim that every word comes from Greek. It'S it's actually true for this. It'S a combination of spark and scoping, which is the Greek for look out. 

So it's pronounced like stethoscope, but instead it's a sparker scope, okay, so what what is it exactly right? So the what we have built is an extension to enrich the spark existing monitoring framework with operating system level. Metrics for that we used so one module that we developed was the cigar source which basically utilizes the hyperic cigar library to get the operating system level metrics and basically this libraries just it - offers a java java api and supports all the major operating systems. You can get the cpu utilization, the the network, the network bandwidth being used, the disk read write, and so we developed this guard shows too attached to every executor, and on top of that we also develop the HDFS sink. So it's the same concept as the CSV sink, but I thought with a difference that now it writes on the HDFS instead and by utilizing those two by combining those two, then we pass the the web UI to actually pull out from HDFS all these available metrics And visualize them, and so you would get you get a view also of the new of the new operating system level metrics that we developed, but also the existing ones. 

The like, I showed before the HDFS, writes the operations and so on. So this is the slightly modified and architecture that we designed for this to work. Again, you have your separate on every node. You have your spark workers which launched the executor and now, in this case our cigar source is attached to the executor. But this is not really required. The scar source is also required. If you want to have the operating system matrix as well and now by using the HDFS link, the user would specify a directory on HDFS in this case it's its custom metrics, let's say, and for every application that you run on spark. We would create another sub folder with the application ID and under that there would be. There would be the HDFS Inc right for its executor, those metrics that were the default, one, the default ones, but also the the ones that we created using cigar and by having all these all these metrics stored on its DFS, then the spark web UI could consume 

That and plot on the on the same web UI that you would use to see to see all the other statistics for your jobs on the default spark web UI um, just some visual, but so just some very basic instructions for how you can stole it. The project is open source under the get are the github of IBM Research Island. It also has some other projects, some other open source projects that we did around tae-hyun illusion now and some MPI projects. Basically, you need to build spark as the as the official distribution as well and the the things that you need to do to get the monitoring part and the HDFS ink is just specifying on the matrix dot properties. The same way that you would the same parameters that you would use for, the CSV sync would be used for the HDFS thing. Now you can specify the the polling period, how often to obtain those metrics and what directory on the HDFS should be used to store. 

Those metrics and then, on the other hand, for the spark web UI to consume that you just need to have the logging enabled. So you would get to that screen. You would be able to see that screen in general, and also you need to specify the the matrix directory which, from which the the spark well be. I should pull the matrix from. So if you want to see your mattox, you have to specify the same directory that you specified on the direct on the methods of properties and the same would be on the spark default configuration file. And now it's a bit more cumbersome to install the operating system level metrics. Basically, you just need to download the hyperic cigar library to all of the slave nodes, extract it and we're on the file system. And then you need to modify the matrix dot properties file, to attach the cigar source, to the executor and also, let's park, know of the exact on the library path where, where are those by now? Where are those library files stored on your disk and then it should work without problem? Ok, so now it's a demo time. 

Ok! So what I have here, it's the it's a just, the cluster that we have in Ireland and we can you consume the historical applications that there is a there is one job that I ran and we can visualize. So if you select that this is the bit that we have added is the executor this pulldown menu lets you select what what type of matrix you want to plot, and as I said so, it's then what you can see there. It'S the default ones that were already there on the file system level, with the HDFS, the read/write, the operations and so on, but also, as you can see, we have added the cigar ones. Basically, currently what we have there is the CPU and the network bandwidth used. The discrete read and write and how much ram was utilized during that job. So, let's try to plot the sdfs rig bites, for instance, and this nice plot saw job and you can immediately see that there was so every line is a different is a different spark. 

Executor, so you could clearly see that in this case it was the executor 0 doing most of the heavy lifting it was reading most of the bytes, as you can see up to 3 gigabytes. In the end, all the others, one, the other ones - were a bit unutilized. We can see now the operating system matrix like the CPU, so you can see again that one executor again was doing most of the heavy lifting up to in the beginning was up to eighty percent, the the cpu utilization and just to show a bit of what's Happening under the hood, basically, if you navigate to the HDFS on the director on the custom metrics directly that i have specified, there is one entry for every application and we, if we have a look inside, you will see that there is a one JSON file. / executor stored there, so this is how a session would look like. I don't think it's very clear but, as you can imagine, with just the matrix being written there, ok so the as a road map we want. We want to make it more pluggable. The the storage mechanism we want to allow different type of backends to store the the matrix. 

We won't support 8 space or MongoDB, which would be fairly straightforward, given data matrix are in position four month. We write them in JSON format in either case and then, when you store everything into HDFS the matrix and then you pull doubt for the webui, there are some scalability issues there. Maybe if you run have a very long running job and you log the matrix for every second, then you would have some problems on the web UI. Another improvement that we want to make is the we want to provide the user. Some smart recommendation on what is the infrastructural needs and we have. We have to focus on the on the patterns derived from this research utilization of the previous jobs and come up with recommendations on that. 

Of course, we want to continue working with the open source ecosystem to improve to improving target mortgage cases and not just the internal but see what other people. How can other people use it? For instance, there was someone on the kid hub just yesterday, pinging me about having headless version of this library in the sense that he wasn't interested on the visualization part, just wounded, centralized storing off of those metrics to be so. He can retail them direct from its DFS, so we're working on that as well having just the the sink and the cigar source to write on nate's they face without any visualizations on the on the web, UI. Ah so yeah, that's it thank you and any questions. Alright excellent: let's get some questions. We'Ve got two mics set up and just a reminder please get closer to Mike. 

If you have a question you can use, can you shown the same screen the matrix and the stages of a of a job so on the on the demo? So here ya? Is it possible? You know to see the dag of the execution dec 02 to correlate, which stage may have some impact or may have something? Yes, so the if you notice, so this job has only two. There are two stages here right, so if you notice this does vertical line, it notifies that was the other one that started the second one, and if you have a mouse over here, you will see that the state changes right. So it up on the right. It gives the host right, so I adjust just to be clear on the on the operating system level. Metrics. We don't display executives anymore, because it's a we are plotting based on the we group them by the note right. So you just see the holes there on the upper right and, of course you will get the job and stage ID while you my shoulder on every on every point on every data point: where are you showing the IDS, the data upper right, the IDS of the Jobs, okay, yeah upper right, so the mouse over, if you notice here it's the UC and okay, this does shows that there is a different stage. 

Okay, thanks thanks yeah, I am I thank you for the thought. It was really interesting things um. I was wondering if you have two applications running at the same time in one knows, is: is there any way to like this late? This is the cpu usage of this application and this dispute the business of the other one so them basically, what the different dots right is that it gives you a snapshot so that the operating system level metrics right so the on the cigar related matrix. What it gives you it's basically an overview of what was happening on your system. While you were running that job, it doesn't imply that the job caused that CPU that ramp utilization. So basically, if you were running, let's say this application and you were doing some Python or some other operation on the operating system, then you would see spike there without having anything to do with sparks. But for us it was useful as well to have any better and saying what it's happening, but we ate.