#s4tf

GSoC is coming to an end. I’m glad to say that I’ve met all the goals as planned in the beginning. You can find out about it in my previous blog posts.

This post is going to be a very brief report of all the things implemented, goals accomplished, some stuff that’s left to do and future plans.

My project was to implement a Data Visualization library in Swift (similar to matplotlib). Before this project, no plotting framework had been written in Swift that could work on Linux, macOS, iOS, or the the other platforms Swift can target. A truly cross-platform Swift plotting framework was needed to fill a gap in the Swift data science ecosystem.

GitHub link to project: https://github.com/KarthikRIyer/swiftplot

What work was done?

The following features were implemented:

1. LineChart #4

2. Plotting functions in LineChart #11

3. SVG Renderer and AGG Renderer as backends. #3

4. Secondary axis in LineChart. #16

5. Bar Chart (both vertical and horizontal, with hatching). #21 #23

6. Scatter Plot (with various scatter patterns). #25

7. Histogram #30

8. Improved aesthetics and text rendering using FreeType. #33

9. A set of examples, to make sure addition of code doesn’t break the existing code. (initiated in #7 with many updates in subsequent PRs).

10. A pure Swift library to generate Jupyter display messages. It is used to display base64 images in a Jupyter notebook. #67

What’s left to do?

Although I’ve completed all the goals as planned, I have started work on a CoreGraphics rendering backend for the iOS and macOS platforms. The PR is currently under review, and little work is left before it is ready to be merged. #35

Also the updated documentation is yet to be merged. This PR has also been submitted for review. #36

Also, the latest work doesn’t work wih Jupyter Notebooks becuase it isn’t able to find the FreeType install on the system when in Jupyter. I haven’t been able to find a fix for this yet.

Future Work

These are some things I plan to work on, post GSoC:

1. Implement an OpenGL Rendering backend.

2. Work on adding support for real-time plotting. For example updating the plot while a machine learning model is training.

3. Implement more complex plot types, such as Contours and Fields.

Highlights and Challenges

A flexible architecture: This approach was slightly new to me. I had to implement an architecture that kept the rendering of primitives as far away from the plotting logic as possible. This provided the necessary flexibility to add more rendering backends when required.

Implementing a pure Swift display library for swift-jupyter: It took me some time to understand how Jupyter worked. I hadn’t ever used Jupyter notebook before, let alone write a library for it. Once I understood how Jupyter worked it was relatively easy to get it to work. But I still faced a few hurdles which were a little tricky to debug. Like, when I was sending a serialized c-string to the Jupyter kernel, it had a next line delimiter at the end. This caused problems with the Jupyter kernel. I required help from my mentors to solve this issue. I also got to explore SHA256 hashing while working on this. It was necessary for message signing. I used IBM’s BlueCryptor library for SHA256.

Allowing more input data types: Deciding which method to use to allow more input data types was time consuming. In the end we settled on generics. Implementing this also took some time, and it was a bit tricky to get the FloatConvertible protocol to work. FloatConvertible was a Swift protocol I wrote which took Numeric input which could be converted into Float. This isn’t perfect yet. It still has some trouble working with the Int Data type.

Working with FreeType: Working with FreeType was a little tricky. I needed a method to draw custom fonts to improve the aesthetics of the plot. Although AGG provides methods that uses FreeType to draw text, I needed a way to use FreeType with Swift Package Manager. One way was to include FreeType with SwiftPlot, but I couldn’t get it to build with Swift Package Manager. FreeType had it’s own build configuration and also had some Python files. The other option was to make Swift Package Manager find the FreeType install on the system, which came with the necessary headers and the compiled binaries. After a lot of legwork I found a way to do that. The next hurdle here was to test it on the macOS platform. I didn’t have a Mac and when I got access to one I wasn’t able to install FreeType on it. After some more legwork I was able to build FreeType on my own and install it from there. This section is also not perfect yet. It isn’t able to find the FreeType install on the system when in a Jupyter Notebook.

Conclusion

I got to learn a lot of things during the summers. I learnt a new language (Swift) and its conventions, approach to building flexible frameworks, how Jupyter works, and many more things. In other words I’ve had a great and productive summer, working under the guidance of the best mentors possible. Brad and Marc (my mentors) were very motivating and their solutions to my problems always worked! 

It was a great feeling to see my project take shape from scratch and even greater to see the community showing interest in it right from the start. I will definitely keep working on this project post GSoC and would like to see a community growing around it, and people contributing to it.

I hope that this project comes to the next GSoC too so that it keeps on maturing and its reach increases more. I would be more than willing to help people get started.

Hey everyone!

It’s been a long time since I last posted. Since then I’ve passed the second evaluation, and am pretty close to the end of this years’s GSoC. The final evaluation period starts tomorrow.

This post will be a very brief account of the developments since the last time, and the future plans.

The first topic to address is how we plan to handle data other than Float. After some discussion it was decided that using Generics would be a good option. I wrote a FloatConvertible protocol, and wrote Float and Double extensions conforming to it. The protocol is as shown below:

public protocol FloatConvertible : Comparable{   init<T: FloatConvertible>(_ x: T)   init(_ other: Float)   init(_ other: Double)   init(_ other: Int)

  func toFloat() -> Float   func toDouble() -> Double   func toInt() -> Int   static func +(lhs: Self, rhs: Self) -> Self   static func -(lhs: Self, rhs: Self) -> Self   static func *(lhs: Self, rhs: Self) -> Self   static func /(lhs: Self, rhs: Self) -> Self }

extension Float: FloatConvertible {   public init<T: FloatConvertible>(_ x: T) {self = x.toFloat()}   public func toFloat() -> Float {return Float(self)}   public func toDouble() -> Double {return Double(self)}   public func toInt() -> Int {return Int(self)} }

extension Double: FloatConvertible {  public init<T: FloatConvertible>(_ x: T) {self = x.toDouble()}  public func toFloat() -> Float {return Float(self)}  public func toDouble() -> Double {return Double(self)}  public func toInt() -> Int {return Int(self)} }

Now, each plot class would have generics accepting values conforming to the FloatConvertible protocol. I’m using Floats for calculations at most places. As the input data conforms to FloatConvertible we can convert each value to a Float before any operation. This method also avoids type erasure. Most of you might notice that there’s no extension for the Int data type. That’s because I’m facing some issues with it. Whenever I use Integers I get a fatal error while execution, the reason to which I haven’t been able to figure out yet. So, currently we can use Float and Double.

The next issue I came across was when I was testing plotting different values. I had forgotten to consider a case when the range of values was less than 1, i.e. the plot markers had to have decimal values. So what I did was if the range was between 1 and 2, I plotted each marker with an increment of half times the inverse of the range. And when the range was less than 1, each increment was one-tenth of the range. I also rounded the values to two significant digits. This worked well for most cases.

Brad pointed out that the current plots weren’t as aesthetically pleasing as those generated by currently existing frameworks. The following needed to be changed:

1. Font

2. Line Size

3. Spacing of numbers from axis hatching

4. Enable a grid

Changing the line size, spacing and enabling a grid were simple tasks and didn’t take a lot of time. The tricky part was using a custom font. Let’s take a look at the approach used with both AGG and SVG Renderers.

AGG provides a way to draw text using custom fonts using the FreeType library. Either we need to include FreeType with SwiftPlot or we need a way so that Swift Package Manager can use a version of FreeType installed on the device. I initially wanted to include FreeType with SwiftPlot, but FreeType has its own build system and it also had a few Python files in its source. So I ruled that out. In order to use the install on the device I needed a module that lets SPM know which library to look for. I found this C module on github which with a few changes I got working. Then I changed the old text functions to use the FreeType functions provided by AGG. The default font I used was Roboto-Regular. I had two reasons for choosing this font. One, it was very clean and professional looking. Second, this font was available in the Google Fonts API(This came in useful for SVG).

I couldn’t find any simple way to use custom fonts in SVG that worked everywhere. The only method I found was to use CSS, which worked only in a browser. Brad said that people would use SVG primarily in a browser so supporting that could be a good beginning. The next problem was how to set a default font in SVG? We couldn’t include the font in the SVG file, and including the ttf file along with the SVG file would be very inconvenient. Here the Google Fonts API came in useful. We just need to specify the font family and it would fetch the font whenever the image was loaded into the browser.

This almost concludes my work on aesthetics. The only problem left is to get FreeType to work with Jupyter. Currently, It isn’t able to find the FreeType installed on the system.

Histogram with updated font and grid

Apart from this my main objectives are complete. In the leftover time I have started work on a CoreGraphics Renderer for macOS. This would be a great addition to the library and increase its audience.

The next thing to do was to get Jupyter notebooks to display the plots using pure Swift. Currently this was being done using the EnableIPythonDisplay.swift library bundled with swift-jupyter. The image was being passed to a display function after encoding it to the base64 format. The display function then used IPython’s display function along with the image format(i.e. PNG) to generate a display message, which was then sent to the Jupyter Kernel. And voila! The image was displayed!

For those who don’t know what a Jupyter Notebook is, quoting the Jupyter Notebook website:

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.

The way Jupyter displays thing is using messages. A message contains all the data that Jupyter requires to display anything.

The general format of a Jupyter message(as mentioned here) is :

{  # The message header contains a pair of unique identifiers for the  # originating session and the actual message id, in addition to the  # username for the process that generated the message.  This is useful in  # collaborative settings where multiple users may be interacting with the  # same kernel simultaneously, so that frontends can label the various  # messages in a meaningful way.  'header' : {                'msg_id' : str, # typically UUID, must be unique per message                'username' : str,                'session' : str, # typically UUID, should be unique per session                # ISO 8601 timestamp for when the message is created                'date': str,                # All recognized message type strings are listed below.                'msg_type' : str,                # the message protocol version                'version' : '5.0',     },  # In a chain of messages, the header from the parent is copied so that  # clients can track where messages come from.  'parent_header' : dict,  # Any metadata associated with the message.  'metadata' : dict,  # The actual content of the message must be a dict, whose structure  # depends on the message type.  'content' : dict,  # optional: buffers is a list of binary data buffers for implementations  # that support binary extensions to the protocol.  'buffers': list, }

To display images/graphics the message type(msg_type tag in the header) is display_data. 

So what I had to do was, generate such display messages using Swift and send them over to the Jupyter Kernel.

Initially I found it pretty difficult to understand how messaging worked in Jupyter, so I had next to no idea about what I had to do. I decided to give it a few days and i finally started to understand what it all was. I made a naive implementation of the message generating code but only to fail. I got some error which seemed like gibberish to me. Even Googling didn’t help me here. Then Marc came to rescue. In swift-jupyter we send the message to the kernel in a serialized form(a utf8 CString). Marc found that while converting to this form, an extra null terminator was added to it, which created an error with Jupyter. Also in my implementation the message wasn’t an optional. So if a message was empty, it was still getting sent and this caused some errors. So, Marc removed the null terminators and made the message an optional. 

But just sending messages wasn’t enough. Jupyter also needed message signing. A unique signature was generated for each message which was verified by Jupyter and if successful, only then we could see the result appear in the notebook. All this time we wore working by disabling message signing in Jupyter like this:

jupyter notebook --Session.key='b""'

The signature is the HMAC hex digest of the concatenation of:

A shared key (typically the key field of a connection file)

The serialized header

The serialized parent header

The serialized metadata

The serialized content

The hashing function to be used was sha256.

The problem that I faced here was, how do I get the HMAC hex digest? Implementing it on my own was out of question, because there was no way to guarantee its security. There are no cryptographic functions in Swift available on Linux. On Mac we could use CommonCrypto. It is an Obj-C library that one could import using bridging headers between Obj-C and Swift. The obstacle here was that using CommmonCrypto wouldn’t make it cross-platform, and also I had no idea of how to use bridging headers with swift-jupyter. 

Brad suggested that we could use IBM’s BlueCryptor library. This was a cross-platform Swift cryptography library. On the Mac platform it used the CommonCrypto module and on Linux it used the libssl-dev package.

So to use the pure swift library the user had to install BlueCryptor before, or disable message signing.

Using BlueCryptor was pretty straightforward. The exact way to get an HMAC hex digest was given in the documentation. I had a few hiccups doing this initially, because I did not know that the key and data for hashing were to be provided as hex strings. I was using simple strings before and therefore wasn’t getting any results. Also the Swift toolchain I was using had some problems due to which it couldn’t compile BlueCryptor. The only Swift toolchains compatible with swift-jupyter were the S4TF nightly builds, which for some reason didn’t have python3.6 support on that specific day, which was a must for swift-jupyter. So I had to build it myself. This took a lot more time than I expected. 

After all these hassles, finally I got it working. Yipee!!

You can find the implementation here: EnableJupyterDisplay.swift

Then I submitted a PR, which was reviewed many times. The comments were mostly about code formatting and style. I still hadn’t gotten used to the Swift style. I did what was asked and the PR was merged. There is still work to be done on this, like supporting more file formats, maybe include audio playback, etc. But all of that can come later. Right now the focus is just on making the data visualization library.

When I had taken a break from thinking about messaging in Jupyter I got to wotk on fixing the plot dimensions issue with the AGG Renderer.

How the AGG Renderer works is that, on initializing an instance of the renderer creates a buffer that holds all the pixels of the image. One could easily say that to have a custom size, one could pass in the image dimensions in the initializer/constructor of the respective class and allocate a buffer of the required size. But on doing this I got a segmentation fault. On investigating the issue using gdb, I found that the error had to do something with the row_accessor not being able to access a pixel. Essentially we were trying to access a wrong memory location somewhere. But even after going through the code many a times I couldn’t figure out where i was going wrong. The previous implementation had the buffer declared globally. It worked that way, but not when the buffer was part of the class.

I decided to dig a bit deeper and tried to find out answers online. There seemed to be no such answers anywhere but, I found an AGG mailing list, that had been inactive for more than two years. I did not have high hopes, but there was no harm in giving it a try. I’ll add the link to the mailing list at the end if anyone would need it in the future.

class Plot{  public:    agg::rasterizer_scanline_aa<> m_ras;    agg::scanline_p8              m_sl_p8;    agg::line_cap_e roundCap = agg::round_cap;    renderer_aa ren_aa;    int pngBufferSize = 0;    unsigned char* buffer;    agg::int8u*           m_pattern;    agg::rendering_buffer m_pattern_rbuf;    renderer_base_pre rb_pre;    Plot(float width, float height, float subW, float subH){      frame_width = width;      frame_height = height;      sub_width = subW;      sub_height = subH;      buffer = new unsigned char[frame_width*frame_height*3];      agg::rendering_buffer rbuf = agg::rendering_buffer(buffer, frame_width, frame_height, -frame_width*3);      pixfmt pixf = pixfmt(rbuf);      renderer_base rb = renderer_base(pixf);      ren_aa = renderer_aa(rb);      pixfmt_pre pixf_pre(rbuf);      rb_pre = renderer_base_pre(pixf_pre);    }

This was the part of the code I was using to initialize the renderer. I sent this snippet to the mailing list. Fortunately someone called Stephan Abmus responded to my mail and pointed out that all the objects required for rendering needed to stay in memory while rendering. I won’t bore you with excessive details but here is a brief summary. I was declaring the pixf and rendering buffer in the constructor and as soon as it went out of scope those declarations went out of memory too and thus we got an error. I am still not sure why making the buffer global worked. Even declaring the buffer and pixf outside the constructor didn’t help. Actually pixf is a typedef for pix_fmt, which holds the format of the pixel to be used in the image, for example RGBA. In fact I couldn’t declare pixf outside the constructor because it didn’t have a default constructor. I had to pass in the rendering buffer while declarinf the pixf object. This meant I couldn’t declare the rendering_buffer outside of the constructor. The only option left was to declare the rendering buffer object and the pixf object inside every function that renderer something onto the image. This method worked!I had finally fixed the bug I was racking my brains on from the first day!

The next task to accomplish was to provide a better way to give data input to the plots, than the Point type. I was still trying to think of a way to do that. I had tried out generics but at the time it didn’t seem like a viable method. This topic is still under development and once something is finalized I’ll write about it.

The goals for my first milestone were done. 

On June 28, the results of the 1st evaluation were declared. I had finally passed the first evaluation. But I did know beforehand that I’d be passing the evaluation as I had already spoken about it with Brad and Marc. Both of them had given me positive feedback regarding it.

Nonetheless, I’ll describe very briefly what the evaluation was. Both the stuent and the mentors were to fill out an evaluation form. The mentor had to give an evaluation of how well the student had performed, and if he/she should pass or not, and the student had to answer a few questions about their GSoC experience and how well they were able to communicate with the mentors.

Phase 1 went pretty smoothly. There were a few ups and downs, but I’ve the got possibly the best mentors ever who guided me through the whole process. I got to know about Swift, it’s coding format and style. I learnt a lot about building frameworks through my mentors’ answers to  my questions. I observes=d how they think about the features to implement and what kind of implementation would be best for both the developer and the end user of the product.

This was the feedback I received from my mentors. It was a really great motivator for me and I want to give my best now more than ever. 

I hope the rest of the summers stays on to be the great experience I’ve had till now.

I’ve worked on implementing Histogram since then, and I also seem to have made some progress on the Data Input front. I’ll talk about any further developments in the next post.

Last time we left off at where I’d implemented some code to display plots in a Jupyter notebook. The next task was to get an independent y-axis working in the Line Graph. I looked at other plotting frameworks’ implementation of this feature. What matplotlib did was that it had different kinds of plots added to an axes, ad the axes was then plotted. To get independent axes, it was twinned with one of the axes. For example if you have a primary axis and twin a secondary axis with it, they will be plotted opposite to each other. ggplot required you to explicity set the scale for the secondary axis relative to the primary axis. After some thinking I decide that having a primary and secondary axis with series of data being added to either one of them would be a good options. The scaling would be done automatically of course. But how would you differentiate between a plot on either axis? A simple solution for now could be to make the secondary plot lines dashed, so that’s what I went with. Remember I had told you that setting up the tests in the beginning helped me with a bug? While implementing multiple y axes, I gave negative input points for the first time and noticed that I hadn’t though of this while implementing the logic. I could easily check the difference in old and new plots by using git. If the newly rendered image was different it would show up as updated in git. I had lifted the logic directly from Graph-Kit the android library I’d made and it seems I’d made that mistake there too. So I spent some time rectifying that.

The next task was to implement Bar Graph. This seemed like an easy enough task as I distinctly remembered it to be less work even when implementing it in Graph-Kit. Bu this time I didn’t repeat the same mistake as I did with Line graph. Although I did initially lift the logic again I immediately checked for loopholes such as negative coordinates. I had to rewrite some of the code to get that done. Additional features planned for Bar Graph were plot hatching and stacked plots. Hatching is the patterns you fill inside the bars, something like shading the bars. For implementing hatching there were two options. One was to use primitives to generate the patterns. As this would be implemented using primitives everything could be done using Swift. This would make all the renderers support hatching. The other option was to have an enum/list of possible hatching patterns and pass a unique value for each case to the renderer and all the pattern generation will be handled independently by the renderer. This meant that each renderer implementation would need to have some logic to generate the patterns, and maybe some renderers would not have out of the box support for this. The former had some disadvantages. If we generated all the pattersn in Swift using primitives we’d have to handle the edge cases i.e when half of a pattern was inside the rectangle and half of it outside, we’d have to clip it. Then there would also be the logic for drawing the pattern in the bound area of the rectangle. Basically we’d have to reinvent the wheel. Even if this was done, in the case of SVG, a lot of statements to render primitives would be added to the file. This would take a lot of time to parse and therefore delay the final image generation. Both the renderers I was using had the pattern generation feature built right into them. They would easily handle all edge cases and filling the pattern in the required area. So I went with the second option. I just had to implement some logic to generate the shapes to be used in the hatching pattern. The patterns I implemented were forward slash backward slash, hollow circle and grid to name a few. Then I went on to implement stacked bars. This could be used when multiple series of data were to be plotted on the came graph. The next series of data was placed over or below the previous series’ bar depending on whether the data point was positive or negative. There were no specific challenges to this other than the usual debugging one does to get any feature working. The last part left in Bar Graph was to allow horizontal bars too. I just had to switch the x and y values if the graph was to be plotted horizontally.

I went on to implement Scatter Plot. This wasn’t a very difficult task as the logic was pretty much the same as that of Line Graph, the difference being that the line weren’t to be joined by lines and instead be replaced with shapes. I added functions to the renderers to draw triangles and polygons. Generating the points to draw shapes like triangle, hexagon, stars was simple trigonometry and rotation of points. I also added a helper function to rotate points about a center. This wrapped up Scatter Plot.

While implementing Bar Graph, the plot had to accept String input also. So I had to add String variables to the Point type. This raised a question. Was the current way of using Point to accept data the correct way? We had to think of other ways to support other data types such as Double, Int, etc. Is generics the right way to proceed? This is a topic that’s currently under discussion. 

Hello everyone!

A lot has happened since the GSoC results were declared! I’ve got quite a few things implemented in my project, I’ll be breaking the discussion of the first part of the coding phase into two or three parts. So lets get into it without further ado.

According to my proposal here, I had one week of community bonding, during which I had to make sure that I had everything I needed to begin with the project, and discuss with the mentors what i should be doing ahead of time. I received a mail from my mentors, Brad and Marc welcoming me to the program. After some discussion it was decided that I should modify my milestones a little bit. Swift for TensorFlow is being used in the Fast.ai course. And there’s a lot of interest in displaying plots in Jupyter notebooks, which is being driven by this. This was to be moved to the first milestone. I have never worked with Jupyter notebooks before let alone editing code that communicated with a Jupyter Kernel. Marc guided me through this. It was decided that for an initial implementation I could use the Swift-Python interoperability to display base64 images in a relatively straightforward manner. Once I implemented some of the planned plots I could work on a pure Swift implementation.

One of the most important parts of building a framework is that it functions as expected. There will definitely be many revisions and changes to the code later on. This warranted a need for the presence of some tests included in the project repository. This would help in making sure that new changes did not break the previously working plots. (I am really glad that we decided to include this in the first milestone itself. It helped me find a really important bug! We’ll come to it in later on)

I have been a little vague in my proposal about implementation of Sub Plots. For those who don’t know what Sub Plots are, they are multiple graphs included in a single display/image. They can be of any type(Line Graph, Bar Graph, etc.). It was necessary to include Sub Plots in the first milestone itself because each Plot would have to be coded in a way that it could be part of a Sub Plot. Implementing all the plots independently and later adding Sub Plot support would be a lot of extra work!

So this is what was decided. In the first milestone I would do the following:

Make a simple Line Chart implementation with Sub Plot support.

Setup tests that saves images.

Get a base64 encoded PNG and use it in Jupyter notebook. Later work on python free implementation.

Complete line chart implementation in the leftover time.

The rest of the stuff for the first milestone according to my proposal were to be moved to the second milestone.

It didn’t take long for me to complete the simple line chart. I used most of the code from the prototype I had made with a few changes.

Let’s look briefly at the LineGraph implementation. All the further discussion will be applicable to Linux (I am using Ubuntu 18.04 LTS) unless otherwise specified.

The first step was to set up the Swift Package. For absolute beginners, this is how you initialise a Swift Package using the Swift Package manager:

Execute this command in the terminal.

swift package init --type library

This will initialise a package that is primarily meant to be a library. If you want a package with an executable as the build product, you can change the type flag to executable.

Before implementing the plots I had to set up the renderers because they were the entities that would handle all the image generation. The plan was to have almost no plotting logic in the Renderers. They would just allow you to draw primitives such as Lines, Rectangles, Text, etc.

One of the Renderers part of the project is the Anti-Grain Geometry C++ library, developed by the late Maxim Shemanarev. I wrote some code to render simple primitives necessary for a Line Graph. Although Swift Package Manager can compile C++ code, C++ functions aren’t directly accessible from Swift code. So I had to write a bridging C-headers. You can call the C-functions from Swift directly which in turn call the C++ functions. You can find the implementation here.

One other aim of implementing different rendering backends was to facilitate adding more backends in the future. This required all the Renderers to have some main stuff in common. So I made a Renderer protocol that included the main functions that every Renderer must have. Each Renderer will have to conform to that protocol. 

The AGGRenderer worked fine apart from one thing. The plot dimensions and and therefore the buffer size were hard coded. This meant that the user couldn’t change the size of the image rendered. This was obviously a big handicap to the end user. But for the moment I decided to focus on implementing the plot and getting the basic structure up and running. I could deal with it later on.

The other Renderer I planned to implement was a simple SVGRenderer written in Swift. The implementation is pretty simple and straightforward just like the SVG format. It has a String variable that will describe the image. Whenever you need to draw a primitive you pass the data to the SVGRenderer and it concatenates the relevant tag to the String. In the end the Renderer saves the String into a .svg file. 

We’re talking about passing the plotting data to the Renderer, but how does that happen? I have defined a Point type which is a struct. It contains two Floats, x and y.  You can pass the plotting data to the Renderer in the form of Point variable, or Point arrays. But the end user need not worry about this. All this will be handled by the Plots. Which brings us to the LineGraph implementation.

What I noticed first was that each plot would have to have the support of being a SubPlot. Therefore the renderer would need each image and plot to have separate dimensions in case of a SubPlot.  Lets take an example of two SubPlots stacked horizontally. An easy way to go about it would be to do all the plot calculations of each plot in its own independent co-ordinate system and the shift the origin of each plot as required while drawing it.So what I did was create a Plot protocol with a PlotDimensions type that held the image size and the dimesions of the current plot being rendered, and two offset variables, xOffset and yOffset respectively. In this case the xOffset of the second SubPlot will be a positive number and the yOffset will be zero for both of them. The plot dimensions will be equal divisions of the net image space available to all the Sub Plots. The Renderer will just shift the origin of each SubPlot by (xOffset, yOffset). This did the job. 

The Plot protocol has just one more method called drawGraph(). This was because each Plot had to have the functionality to just draw the plot in memory irrespective of what mode of output(like saving images in case of AGG, or displaying an image in a window in case an OpenGL implementation was written) the used Renderer would have. Also this facilitated drawing each SubPlot separately to the image before generating the final output. 

Then I took the plotting logic from my prototype and the basic Line Graph was done. 

The next step was to set up the tests. I created an examples directory with individual executable modules, each demonstrating a single feature. In this directory I made a Reference directory with two separate directories for AGG and SVG renders. So that anyone could run all the tests easily in one go, I made a simple bash script with the commands to run each example like so:

swift run <Executable Example Module Name>

Then came the time to let the users show the plots in a Jupyter Notebook. Initially the way I did this was, save the image as usual using the AGGRenderer, re read it from the disk encode it to base64 in C++ code, and send back the String to Swift code. But there was a better way that my mentors suggested. The library that I was using to encode PNGs, lodepng, allowed you to encode the image in memory and not save it to the disk. I could return a pointer to a buffer with the encoded bytes, to the Swift code and use some functions under Foundation to do the base64 encoding in Swift itself. This could come in handy sometime later if another Renderer could generate images that coudl be encoded to base64. I did the encoding using a function like this:

public func encodeBase64PNG(pngBufferPointer: UnsafePointer<UInt8>, bufferSize: Int) -> String {    let pngBuffer : NSData = NSData(bytes: pngBufferPointer, length: bufferSize)    return pngBuffer.base64EncodedString(options: .lineLength64Characters) }

To display the image in Jupyter I added these lines to the EnableIPythonDisplay.swift file in the swift-jupyter repository:

func display(base64EncodedPNG: String) {  let displayImage = Python.import("IPython.display")  let codecs = Python.import("codecs")  let imageData = codecs.decode(Python.bytes(base64EncodedPNG, encoding: "utf8"), encoding: "base64")  displayImage.Image(data: imageData, format: "png").display() }

To display the plot the only thing the user has to do is to include this file in their jupyter notebook, get the base64 image from the plot object and pass it to the display function.

This completed all the main stuff I had planned for my first milestone well before the deadline. By this time the official coding period hadn’t started yet. The first deadline was June 24 and I had almost a month left. I could cover a lot more stuff in my first milestone itself, so I decided to complete the Line Plot and keep at least the Bar Chart implementation in my first milestone.

You can find all the code here.

This post has already gotten pretty long, so I’ll sign off here. I’ll be discussing the rest of my Line Graph implementation, Bar Chart implementation and how setting up the tests beforehand helped me avoid a bug, all in my next post.

Stay tuned!

PS: Don’t forget to subscribe to the Swift for TensorFlow newsletter to stay up to date with the work being done and the happenings of the S4TF community!

Here’s the link: https://www.s4tfnews.com/

Hello everyone!

I’ve been selected as a Google Summer of Code developer under TensorFlow. Yaay! 

It’s been a long time since the results came out, and I’ve finally got around to writing about it... so my next few posts will be about my GSoC experience with TensorFlow.

Let’s start at the very beginning...a very good place to start...

January 2019

It was that time of year when everyone was gearing up for GSoC. The organisations for 2019 were to be announced on February 27.  There were many who had already started to contribute to open-source organisations that came to GSoC in the past years, to get a head start. This was the most frustrating experience of my life, because I was having a hard time choosing an organization. I had been exploring computer graphics for the past few months and really wanted to do something in that area...But which org to choose...I was also in doubt if the organizations that came last year would come this year or not...

February 2019

Finally on February 27, the list of orgs was out...I had almost given up, but that day I received a message from a senior at college, that TensorFlow had come to GSoC this year and there was a project that would interest me, something related to Data Visualisation. I was a bit confused if I would be able to contribute or not, considering that TensorFlow was an organisation working in Machine Learning. There was no harm in taking a look at the projects, so I did...

The idea my senior was talking about caught my eye. The idea was:

A library for data visualization in Swift

Once data has been cleaned and processed into a standard format, the next step in any data science project is to visualize it. This project aims to provide a Swift library similar to matplotlib. Desired features:

1. Two-dimensional plots: horizontal and vertical bar chart, histogram, scatterplot, line chart 2. Support for images, contours, and fields 3. Support for detailed subplots, axes, and features Desired languages: C, C++, Swift

I immediately decided to apply for this project. I had made a plotting framework for Android before and this seemed to be something similar but way more involved. I’d have to work with Swift, a language I hadn’t worked with before. It included graphics work that I wanted to do and I would be working under the guidance of people from the TensorFlow team! This seemed like a great opportunity to learn something new.

It turned to to be exactly that. I enquired about the project and guidelines on the Swift for TensorFlow mailing list. I got excellent guidance from the project mentors Brad Larson, Marc Rasi and other community members. I did some digging on my end, got a prototype working and shared it with the S4TF community.

March 2019

Then came the time to work on my proposal. The projects aim was to make a Swift library that works cross-platform. Currently available Swift plotting frameworks worked only on Mac and iOS. So the library had to have multiple rendering backends so that the end user may choose one that worked best on their platform. It was important that I implement at least two rendering backends, so that the library would be architectured in a way that made it simple enough to add more backends later on. After several reviews and revisions from the mentors I finalised my proposal and made the submission on March 25.

Then next month went into preparing and appearing for my end semester examinations, but GSoC results were always on my mind.

May 2019

My exams were over and the results were to be declared in 6 days. 

On 7 May I received a mail:

Thus began my GSoC journey with TensorFlow.