ZoomCharts Co-Founder and CTO Viesturs ZariĆĆĄ gave a highly successful presentation at XXVII DevClub.lv on January 15th, 2015 at the Microsoft Latvia office. DevClub.lv is a community of Latvian IT specialists that gather at monthly events to host free talks and presentations. Through them, the local IT community can share knowledge and experience, and can network and communicate on everything information technology-related.
 ZariĆĆĄ talked about the challenges that must be faced and overcome in developing JavaScript SDK. His discussion sparked a lot of interest from attendees, and concluded with a flood of questions and talks of deals within the IT community.
 Take a look at this overview of ZariĆĆĄâ PowerPoint presentation, and discover why, unmatched in both functionality and speed, ZoomCharts is the worldâs most interactive data visualization software:
  Itâs really hard to talk about ZoomCharts until you see how itâs working. So, we are doing charts. But we are doing interactive charts. You can take the chart and move it around, you can zoom in, and itâs very interactive and shows the data. Basically, we provide visualization for large amounts of data and in a very interactive way.
 Usually, with normal charts, what you get is a picture. We start with a picture and go into the data. To show some more examples, we have a pie chart, a data-driven chart. So we can drill down and see more data, and we can go even deeper, and we can go back. We are building charts for data exploration.
 ZoomCharts is not another HTML5 charts library. We are doing interactive stuff, we are doing it fast, and focusing on touch, and working with big data. Thatâs the product.
 How did we get here? It all started 20 or so years ago, when there was DOS6. The story was pretty simple back there. You had a 320x240 screen with 80 bits per pixel. And you just wrote whatever you wanted on those pixels. Windows came and basically everything went away. It was tricky and hardware-dependent.
 Web changes the scene again. Graphics is fun again. We can do what we did in the old times, but we can do it a lot faster because we have graphics acceleration on pretty much every device.
 With new stuff comes new challenges. There are many devices and every device has its own browser and capabilities and resolutions, and performance varies a lot. Thatâs what we are working with right now, and thatâs the main challenge.
 A bit about development setup: We write in CoffeeScript because itâs very neat. Itâs easy to read and easy to write. We use GitHub, which is pretty common nowadays. We build systems in JavaScript. Chrome is our best friend. We do all the debugging, testing and profiling in Chrome. They are really helping to bring web development to masses. We run automated tests. In SDK, automation is very useful. We are not at the level where we have 15 devices to test on, but maybe we will get there. We like WebStorm and Vim.
 In terms of graphics, we work on three things: Canvas, SVG, and WebGL. Canvas is very fast on every device. SVG is very nice because of CSS. You can style it, you can animate it because it scales nicely, but it is slow. It doesnât work with interactions. For the chart to work interactively, you have to keep the frame rate at least 30 frames per second. Performance is the main target weâre looking at. WebGL is very nice but still under 50% of devices support WebGL.
 We chose Canvas. Itâs a very simple interface. You get a pixel buffer, and you can just paint on it. There are three kinds of operations: you can paint shapes, text, and images. It is very simple, and all devices implement it the same way. So far we havenât encountered any device that has implemented canvas in a way that differs from the others. Basically if a device supports Canvas, then we are good. Canvas is very mathematical. You have coordinates everywhere. You have to do your own animations. Itâs kind of tricky because you need to do a lot of the housekeeping yourself, but it is also a benefit. You can do interactivity very easily. Itâs very easy to control animations. Another thing thatâs easy on Canvas is graceful degradation. You can remove shadows, and performance will go up. Itâs easy to measure the frame rate because youâre painting every frame. You measure frame rate and you decide what you want to paint to get the performance you need. All the changes in the view are free. Thatâs different if you compare it to SVG or WebGL where you have to add or remove elements from the scene. Here if you want to change something, you just execute or donât execute some of your code, and that is a big benefit for interactivity, because in interactivity you basically do big gestures, and it completely changes the scene, and you donât want your chart to freeze for a few frames to update the changesâyou want it to be really fast.
 We chose a few third party libraries. Raphael is for vector graphics, Hammer.js is for input, like mouse and touch detection, Leaflet is for maps, and Moment.js is for printing and parsing date and time. As we developed, we wanted as few libraries as possible. We already got rid of the first two and implemented them ourselves. The trick is, as you want the functionality to be as seamless and good as possible, you quickly get into limitations of the library because the library is written for the use case of the developer. The library and your use case will most certainly differ in subtle ways and these ways get into your development process as you go. Currently we are looking to rewrite Leaflet, but not at the moment. Our experience is that third party libraries are a great way to start early on, but as you go, you usually replace them.
 A few challenges I would like to talk about: responsive designâso there are lots of things we can solve with responsive design, but itâs still a struggle. Because this is Canvas, we have to do everything ourselves, and we have to rely on CSS. This is the downside of Canvasâthat you really have to do everything yourself. We made a module that would make the charts adjust to the size of the container. We used a very weird hack. It works, but it is not easy.
 Another thing is that Canvas is a pixel buffer, so if you zoom in, you get something very pixelated. We have solved half of the problem using devicePixelRatio value that is supported on every browser we have tested. One thing is that we donât support Internet Explorer below version 9 because Canvas is not supported there. So thatâs our biggest limitation. You can use X Canvas but itâs very slow, and you donât want to use X Canvas. On every browser that supports Canvas, you have this devicePixelRatio and you can just upscale the canvas, and it looks very nice and native, like vector graphics, basically. That works for desktop. For mobile and touch zooming, it doesnât work. And so far, there is no way to detect touch zoom on browsers. There is one website that has about 10 variants on how to test device zoom, basically how to test what the size of the actual screen pixel is, and none of these methods really work on touch zooms. Another thing is that if you zoom in really big and if you have retina screens, you can get a canvas size beyond 2048. At that point, hardware acceleration just switches off. Thatâs pretty consistent on Macs. On Windows devices, it differs. And on mobile devices it is pretty consistent. At this resolution, hardware acceleration switches off and goes back to software rendering. Obviously we donât want to do that. Currently we just limit the number of pixels in the canvas. The canvas can get bigger, but the real number of pixels is smaller, and the browser just upscales. That way we keep the hardware acceleration.
 Safari is a pretty weird browserâdifferent from other browsers in one very interesting aspect. If you get close to 100% CPU in JavaScript, the input events get blocked. In other browsers, the input events will be processed regardless of the CPU utilization. Usually you would just browse a chart, and as you load the CPU, most motions donât register anymore. The animations still work and everything looks like itâs working, but it looks like your mouse shuts down. Safari has some different input architecture. Sometimes it goes into browser lockups, and results in the beach ball being displayed, and the page going unresponsive. Sometimes Safari just crashes. There is something fishy going on with Safari JavaScript engine. Basically, Safari is the worst browser ever for us. So letâs go on.
 DOM is very slow. If you want to go at 60 frames per second, then updating DOM elements in every frame doesnât really work. Currently because of this info popup on time chart, if you want to update it on every frame, it takes 30% of the full load. 70% is painting the chart, 30% is generating DOM. Not just regenerating everything, but just updating the position. We try to avoid doing DOM updates as much as possible. Almost everything is Canvas. We have a toolbar thatâs DOM, and this info popup and context menuâthatâs DOM. For all browser text, we have our own HTML markup parser that understands simple things like italic, bold, new line, and we just render it with our own renderer on Canvas. Caching it is pretty fast.Â
 We have been in the market for one year. The customer usually comes this way: first they get a trial, then they ask a few support questions because our library is pretty new and the documentation is still a bit lacking, and after they go and say, âyeah, we will buy.â The support is very important. We are really trying to do 1-day issue resolution and mostly it works, sometimes it doesnât. The customer is usually happy if we get issues resolved in one day. Thatâs about the timeframe the customer is usually willing to wait. The number one support question is usually âTell me what I did wrong.â Number two is âcan you do somethingâŠâ So far the support hasnât been very busy. We really donât have a support person. Usually we distribute support questions between the developers. It has worked fine so far.
 Testing is pretty important for SDK because you donât want a customer to say, âhey, this doesnât work.â Thatâs pretty embarrassing. We have our own automated tests that we run on every GIT push. We wrote our own thing. We originally considered using Jenkins, but we have a very big focus on visual stuff, and we figured that it is easier to write our own. We have something like 160 webpages distributed in various chart types, and each webpages has basically a page with some charts. We run it using Phantom, render the image, check the error logs. We compare images, we record the performance, and we record the errors. For the test to succeed, the images need to be the same. We do pixel-by-pixel comparison. There should be no errors in the console. Performance is recorded and we review it before release. Comparing images looks like a very aggressive way. The reason we do this is because this is a visual library, and if you get a customer, he would integrate your chart, add some styling of his own, and he really wouldnât like that you get an update and suddenly everything looks different. Thatâs why this is the first thing we did, and we want to stick to this.
 For the future, this is what we want to do next in testing: we want to add really interactive testing. For this we have a built-in record and playback system. This way, all the input events will be recorded, and instead of just rendering a webpage, you will render a webpage and then run back input events that were recorded, and check if the events are the same that are going back to the user, and if the final image looks the same. Interactivity is a big part of our product, and we want to automate everything here, or, as much as possible, and to free the developers to do interesting stuff.
 We used Phantom for simple testing, but Phantom is just like a browser thatâs rendering on a page. We want to test every device, and we donât have these devices in our office. Currently we are working with BrowserStackâa service that will provide you with lots of devices, and you can just give them a URL, and they will give you either an interactive modeâjust like a video screen where you can interact with the deviceâor you can interact with the device in an automated way with Selenium API. We are currently integrating with them to make automated tests on as many devices as we can. So this is a different approach than buying 50 devices. Weâll see if it works, but currently we have high hopes and we are working with BrowserStack. They have some issues that we have encountered, but they are making good progress.
 Our stuff is pretty complicated, and we spent 50% of the time debugging and understanding whatâs going on. For that we use Chrome Dev tools. So with the debugger, insert break points, otherwise the local variables runs the program step by step, and thatâs the basic tool. You can put break points wherever you want, and if you really have some conditions then you put the debugger in your code and the browser will stop there.
 Profiler gives you a quick overview about the CPU load and a nice overview about the hottest functions in your code. You go in and check whatâs going on. Profiler has a text view that can show you exactly which function has spent the most time, and you can really go and drill down in your call stack and see where the issues are.
 Timeline is nice, too, for seeing your frame rate and whatâs going on. Chrome does nice triangles that show warnings that there may be some issues. Itâs another nice debugging tool to see where the problems are. Chrome is doing as much as it can to show whatâs going on under the hood.
 Another nice feature is that you can use all these debuggers on a mobile phone. You just connect your mobile phone and can go on and debug mobile devices. You can do this for Android, you can do this for iOSâwith a Mac, not with a PC, thatâs a bit tricky. So far we havenât really had an issue that is not reproducible on Android or iOS, but if you have an issue on Mac, Blackberry or Nokia phone, then I donât know how to debug that.
 For the future, of course we are working on more charts, and we are working on an extension API so users can add functionality to the chart itself. So we have had a few feature requests where people want a very narrow niche feature like a specific interactivity or specific overlay of their data, and we want to do it via API. Currently we are stabilizing the internal architecture and the API should be available in a year or so.
 One thing that Chrome Dev tools doesnât give you is to give a quick overview of how much memory was allocated and compare it between graphs. Thatâs one more thing weâd like to add to our automated tests, to understand how much memory allocations are going on. Memory allocations are not nice because you need to run garbage collector to free that up. And garbage collector just freezes animations. So you want garbage collector to run as little as possible. We are currently looking into tools that would track these allocations and give us a number.
 We are thinking about experimenting with WebGL because it is the future, and weâll get there when WebGL is supported and itâll give us maybe not bigger performance gains, but nicer graphics because these charts are looking pretty simple, and if you add more effects, performance just goes down dramatically. You can do a lot more with WebGL in terms of nice visual effects.
 Another thing about the future: we have grown pretty big. We have more than 20K of CoffeeScript code. That translates into something like 40K of JavaScript code. JavaScript is a nice prototyping language, but itâs not nice for big projects. CoffeeScript removes some of the common issues like comparing by string value, not by real value, but it doesnât really work long term because refactoring is very hard. You usually spend two days if you have to refactor something thatâs not trivial, and tests donât catch everything.
 With a statically typed language, you get error checking, you get better performance because the language knows data types for simple things, and you can really tweak. Basically, the browser tries to optimize for you. If you have a type language, you can optimize it beforehand. You can fun minifiers, but if you look at the minified code, there is still a lot to be minified after. We are happy with CoffeeScript because itâs easy to write and read, and itâs easy to call from JavaScriptâthatâs very important for SDK library.
   Check out ZoomCharts products:
Explore linked data sets. Highlight relevant data with dynamic filters and visual styles. Incremental data loading. Exploration with focus nodes.
Time navigation and exploration tool
Browse activity logs, select time ranges. Multiple data series and value axes. Switch between time units.
Amazingly intuitive hierarchical data exploration
Get quick overview of your data and drill down when necessary. All in a single easy to use chart.
Scrollable bar chart with drill-down
Compare values side by side and provide easy access to the long tail.
http://www.zoomcharts.com
The worldâs most interactive data visualization software
   #zoomcharts #interactive #data #interactivedata #datavisualization #interactivedatavisualization #chart #graph #charts #graphs #ViestursZariĆĆĄ #Microsoft #Latvia #DevClub #DevClub.lv #JavaScript #SDK #JavaScriptSDK #IT #informationtechnology #PowerPoint #datadriven #bigdata #HTML5 #DOS6 #Windows #graphicsacceleration #CoffeeScript #GitHub #Chrome #WebStorm #Vim #Canvas #SVG #WebGL #CSS #gracefuldegradation #thirdpartylibrary #Raphael #Hammer.js #Leaflet #Moment.js #responsive #responsivedesign #pixelbuffer #devicePixelRatio #InternetExplorer #touchzoom #Mac #Safari #CPU #DOM #GITpush #Phantom #BrowserStack #API #SeleniumAPI #ChromeDevtools #Profiler #Timeline #Android #PC #iOS #Blackberry #Nokia #extensionAPI #garbagecollector #SDKlibrary
