Cutting-edge Web Technologies

Jump to the end to view links to our app.

Final Product

Our final product is the GradeStat web app, a tool to extend the existing functionality provided by bCourses. Course staff can use this site to manage and monitor their teaching staff with less dependence on messy Excel spreadsheets and charts. GradeStat is designed to both save time, as well as enable more consistent monitoring of grades in a large course.

Walkthrough

This walkthrough describes the basic page of GradeStat. However, we expect most of the user time to be spent on the Course and Assignment view pages which allow staff to manage and view detailed information.

Splash Page

The splash page has a simple scrolling layout. It currently contains simple marketing text and a basic description of the site. In the upper-right corner, there is a link to log in with a Google account.

Home Page

The home page, visible after logging in, has a welcome message and a brief description of how to use the site. On the left side of the screen, there is a navigation bar with links to the “Courses” and “Settings” pages.

All Courses Page

The first time this page is viewed, it prompts the user to enter their bCourses authentication token (click the “?” for instructions on how to retrieve this token). Once the user has updated their authentication token, the courses that they teach will be loaded from bCourses and listed. The user has the option to “Add” or “Remove” each of these courses from GradeStat (essentially opting to store the data at GradeStat for faster retrieval and analytics). Each course listing also links to a course profile page.

Settings Page

Currently this page acts as a space to update a user’s bCourses token at any time. bCourses tokens can be optionally revoked through the bCourses site and this page has a form to upload a new one.

Course Details Page

The course page contains basic information about the course as well as a listing of all assignment and staff. In the staff listing table, the user can update the number of hours per week that each staff member should work on grading assignments. The assignments listing table contains links to the profile pages for each of the assignments.

Assignment Page

The assignment page is the most feature rich page in the app. This page contains six drop down sections. At the top, the user will find basic information about the assignment including due date, date of creation, the number of points possible and the number of submissions still ungraded. The next section contains a form to assign staff members to grade assignments. The user chooses a number of submissions to validate and a number of submissions to allow overlap between graders. Upon submitting the form, GradeStat’s algorithm uses this information and the relative number of hours assigned to each staff member to assign specific submissions to be graded by each staff member. The third section contains a table with information and links to grade all of the submission assigned to the current user. The fourth section contains a list of all submissions for this assignment. The next section contains statistics on grader participation and results. There is a table showing the average grade for each grader as well as the overall average grade for the assignment. There is also a side-by-side box-and-whisker chart showing the distributions of each grader’s grades alongside the assignment total. Another chart shows each reader’s progress in grading their assigned submissions. The chart shows what percentage of assignments they have completed with timestamps. The last section contains statistics on the overall distribution of the grades for this assignment including a table with relevant statistics and a histogram of grades.

Lessons Learned

Meteor

Pros:

Meteor has a lot of reactiveness built into its design. This capability was useful for working with API calls. A reactive framework ended up being an ideal setup for running API calls on the server to access data from bCourses and then update the views. This way, we could load most of the data on a page quickly and fill in the rest of the data as the data was returned from bCourses. Of course, this functionality was only relevant when we were not loading course data directly from the database.

Meteor also makes development really easy by integrating the dev environment with a simple deployment strategy all by using just the meteor command line tool. We feel like this is a nice advantage, but it does leave us with questions about hosting our app on alternate services should the need arise.

Cons:

Meteor takes a little more work than we were expecting. We are both used to doing more web development work with Rails where there is perhaps a slightly steeper learning curve, but a lot more work is done for you. It seemed to us that a significant amount of our code ended up being much more verbose than we thought necessary. It seemed that in order to achieve some tasks, we would often need to write a function on the server, a function on the client to interact with the server and then possibly another handler to retrieve the data. Some of this perception can no doubt be attributed simply to the fact that javascript is a more verbose language than Ruby, but we found that we were often frustrated with the ways that Meteor chose to implement features. The initial setup also took longer than expected because the default app created by meteor is extremely minimal. We spent a while just getting our app configured in a way that was helpful to us.

Additionally, because of a lack of standard configuration, we ran into problems through development that caused bugs because it wasn’t clear exactly how meteor was compiling our app. Another good example, is that Meteor has an odd way of supporting traditional npm packages, despite the fact that it was implemented in nodeJS. There were times were we wanted to use regular npm packages (like a bCourses / Canvas library, we had written) but were unable to easily use these tools which cost time.

Since Meteor is a more recent framework, the documentation and support on sites like StackOverflow are not as developed yet. Even when we did find helpful tips online from other developers, they often used tools that had since been deprecated. Meteor has changed rapidly since its first release, so it was sometimes confusing what the “correct” way to do something was.

Styling and Libraries

When we started writing code, we decided that we did not want our app to have any obviously recognizable traits of a site that was “Bootstrapped” or used another popular UI framework. We decided to use Yahoo’s Pure CSS because of its good support for grid layout, but minimal effect on the overall theme of the site. We ended up struggling a little more because of this decision. We spent more time trying to make our site look presentable than we should have; especially because neither of us have very much design experience (and perhaps just lack an intuition for aesthetics). We learned that it is ok and probably even a good idea to use libraries to fill in the gaps in your knowledge or skill. If we had started by using a library like Twitter’s Bootstrap, we could have spent less time struggling with evasive UI bugs and more time adding/fixing broader features.

For the graphs, we used a version of HighChartsJS which was designed to work with Meteor, though this too required some customization.

In Retrospect and Moving Forward

As is perhaps apparent from the relative lengths of our pros and cons sections on Meteor, we were not the biggest fans of Meteor. It was a great learning experience and gave us a chance to work on our Javascript skills, but overall we would not implement this app again in Meteor. The plan for this project is that it will be an ongoing project that will hopefully be useful to future course staff at UC Berkeley. This means that the code base will need to be built upon by future collaborators and we fear the the use of Meteor may preclude that possibility simply because it is not as accessible.

The biggest lesson here is: Project Documentation. Meteor does indeed have many awesome features, but we felt that we spent too much time wrestling with Meteor. Perhaps, when the framework is more mature and well documented it will be a better contender in the app development space. In the meantime, Rails (version 4) + React or something like SailsJS and node seem like more stable alternatives to Meteor, unless you need the deeply reactive features Meteor provides. 

Related Links

Proposal link: http://webtech-cs294.tumblr.com/post/111183958936/proposal-gradestat-by-lambda-lambda-lambda.

Milestone 1 link:

http://webtech-cs294.tumblr.com/post/113788229828/gradestat-project-milestone-1-team-lambda-lambda

Milestone 2 link:

http://webtech-cs294.tumblr.com/post/116986546335/project-milestone-2-gradestat

Application Link:

gradestat.meteor.com

GitHub: https://github.com/gradestat/gradestat

Screencast:

Here is a link to the current code base: https://github.com/jesca/powerly

Here is a link to the current deployed site: powerly.meteor.com

Here is our video demo: https://www.youtube.com/watch?v=bstwBrNHZd4

Our group (Jessica, Paymon, Junwei) successfully created a hybrid app that tracks a user’s AC usage through through post requests sent from a device. We completed everything we had in our milestone, as well as added some functionality (such as the live graph), and we have a working proof of concept. We built a working end-to-end system that receives post requests, keeping track of the total power from multiple devices. In the end, we not only got it to work with a simulator, but tested it with the live device as well.

We used Meteor and coupled it with Ionic and Cordova to access the phone’s built in functions (push notifications) and to give it a native feel.

This is the first time any of us used Meteor, Mongo, Ionic, or Cordova (this was a really new project for us).

Here are some major skills that we learned from working on this project:

We learned how to use a new framework that was written entirely in Javascript

how template-based frameworks function

Figuring out how to send push notifications without running the application in the background, using Google GCM

Designing how to advantageously structure a no-sql database (we had a total of 4 collections)

Using the Meteor accounts packages

Using the front-end library Ionic

Use d3 to have a live graph of the data streaming in through post requests

Using Meteor’s Blaze

Using Meteor reactive variables

Paymon made a pull request to Meteor when he found a bug in the source code, and it got accepted!

Go-Beacon Final Report

For our project we developed a set of components which enable developers to quickly bootstrap a bluetooth beacon interaction on Android. Specifically we created a simple example Android application which reports interactions with various bluetooth beacons to our Go-beacon service (which we created using a Ruby on Rails Application, hosted on Heroku). These device interactions can then be used to trigger any sort action the developer desires, from logging a user’s location to sending them a text about nearby commercial deals.

Deliverables

In our proposal we listed the following deliverables, which we are happy to say we completed.

Client stub

Management/Interaction API

Backend Code to support API

Application Demo demonstrating (beacon -> client -> report interaction -> callback) interaction.

Technology overview

We used native android code with the altbeacon library to develop the phone client. It was necessary to use native code in order to gain access to the bluetooth sensor.

The Go-Beacon interaction routing service is developed as a standard Rails application which is deployed on Heroku. There is no user facing front end on the Rails app, it only provides a simple JSON api which is leveraged by the other components of our application/demos.

Demo

In order to demonstrate the kind of responses to interactions we imagined possible, ie sms, email, etc.. we built several simple external services in PHP.

Our in class demo was created as a node.js server + a react.js frontend. We used node.js because it was the simplest way to store some very simple state on the server and serve it up to the react frontend.

We also used a Kinivo BTD-400 USB dongle and the linux-ibeacon library to create a transmitter in order to demo our application. (https://github.com/dburr/linux-ibeacon)

Git Repos

Android Client: https://github.com/go-team/go-mobile

Rails Backend: https://github.com/go-team/go-beacon

Demo Support: https://github.com/go-team/go-misc

Github: https://github.com/EzzySri/CalDash

Website: caldash.herokuapp.com

Webcast: http://youtu.be/ko8UHuMdacM

Everyday, one has mandatory and flexible events. Mandatory events can be things like class, meetings, or work which requires us to be in specific places at fixed times. Flexible events, well, are more flexible. For example, a person might want to get a haircut, but doesn’t really care when in the day he gets one. Thus, among all the events we face in one day, what is the best way to schedule these flexible events in order to save the most time possible? We created Caldash to solve this problem. Caldash is a calendar app manages your flexible and mandatory events and finds the best schedule that minimizes your walking distance throughout the day.

Currently, we have built out the core of this idea. Caldash can currently optimize events on a current day to minimize distance traveled. However, we cannot currently optimize on any other heuristic, such as minimizing gaps between events or minimizing driving distance as opposed to walking distance. In addition, by the nature of our algorithm, we can only schedule events on every half hour to reduce the number of total possible events and thus the search tree we need to explore in order to find the best schedule. In addition, we tried to generate multiple schedules so that the user can pick the one he likes the best, but this also made the algorithm too slow to run on Heroku. However, Caldash is still very good at what it does currently. It can almost always find the best schedule to save you from walking up those treacherous Berkeley hills.

We used Rails for the backend. At the core of our app, we need to create and then find an optimal schedule events. Creating events is really easy in Rails due to its MVC principles. However, optimizing schedules proved to be the hardest part of the backend. We initially thought we could reduce our problem to the Traveling Salesman Problem since we were essentially trying to find the shortest path between a bunch of locations. However, we had the additional problem that the Traveling Salesman Problem assumes that you can travel from any node in the graph to any other node. In our situation, we couldn’t because only some events can happen after other events, and placements of events were highly dependent on where other events were placed. After realizing this, we tried a different approach. We tried every combination of events that had a distance less than the best schedule’s distance we had found. For example, if we are trying to schedule five events and we found a schedule that has a distance of 500 meters, then we know we can immediately stop exploring any placement of events that has a distance greater than 500 meters. We used euclidean distance to calculate the distance between any two locations.

If we were going to restart the project, we would not have used Rails. While Rails allowed us to quickly create the CRUD portion of our application and provided a bunch of awesome libraries to help us deploy onto Heroku, the major bottleneck for our project was the speed of the optimization algorithm. Because we do explore a significant portion of the search tree that grows exponentially, we had to place a lot of limits that impedes on the user’s experience, such as only allowing the user to place events on every half hour. If we used a faster language that allowed for concurrency (Ruby has the GIL that only allows one thread to execute at a time) such as Scala or Java, we could potentially make the algorithm faster and thus the app better.

We also deployed our final app onto Heroku. Heroku made it insanely easy for us to get something on the web for free. We had to do a bit of refactoring so that Heroku knew how to host our app, but after that, we only had to type git push heroku master to deploy to production. We didn’t explore other features of heroku such as various addons or dyno scalings because those features required money. In future projects, we would use Heroku again, especially if we had a budget that allowed us to use the additional features.

For the front end, React and Flux are the pillars of this portion of the project. Essentially, React is the UI layer while Flux is the data layer. We learned that the benefit of React in reducing complicated program flow is at its best when we strictly maintain the design as the following. The UI side registers actions with the data layer as a set of API endpoints. The triggered actions cause the states at the data layer to change. After all changes are finished, the changes present themselves unidirectionally to UI components from the top down, from which point the React Diff Algorithm takes over.

We add on to the tested benefits, continuing from the discussion we had in the last milestone report. Firstly, it reduces redundant calls to setState. As the web app grows larger, we found that stores become interdependent on computation of data. For example, the Google Map needs current event form input to render the location on the map. Without a store layer, we would need to manage the changes in state in the UI layer itself by passing and receiving states. This complicates program flow and we would end up with more calls to setState simply because it is hard to see the best course of actions in terms of achieving a set of computation. With stores, we are isolated from changes in UI and the computation becomes a traditional program with object representations, class hierarchies and modules and not a web app mixed with css and DOM manipulation, thus allowing us to reason in the same way as when we first learned programming. As a future goal, we will adopt stronger software engineering principles to the front end such as utilizing object factories and class inheritance in Javascript to allow the codebase to scale as we implement more complicated features.

Secondly, isolating the data layer to approximate a traditional program allows us to apply familiar design principles that are not easily relatable in web programming. For example, we did our best to separate computation from service calls. This is because although there are many service calls, the underlying computations tend to be similar. Hence, these computations involving data structures are abstracted out as services themselves to achieve a layered design. Later in the project, we found that many features that we decided to add did not involve refactoring existing codes at all but just adding a few lines of service calls to the same underlying computations. This is not necessarily the case in other frameworks such as Ember, in which actions are contained in controllers when developers are more in the mindset of delivering the result of an action than applying traditional design principles. To summarize, the separation of concerns allow developers to take consideration of the data layer as a whole when delivering an action.

Background

uSee Data is a data visualization platform built to allow anyone to do more with their data.  We assume no previous technical knowledge on the user to provide them with beautiful data visualizations that they would normally not be able to create.  The idea is that anyone can land on our site, and have a meaningful visualization within a few minutes of uploading their data.  We do this by providing an intuitive user interface and experience that throughout the entire process of creating a visualization makes sense.  In the end it’s also incredibly easy to inject this visualization into the client’s site.

Target Users

The target users for this web application is largely those that don’t have much technical knowledge but still want to create dynamic, cutting-edge data visualizations.  Most of these types of people want to create D3 charts, but don’t have the time to spend learning how to code.  Our product is oriented towards non-technical researchers, journalists, and civilian scientists. Other than that, the website UI/UX is made in such a way that it allows anyone with a data set to create a cool exportable data visualization.

Technical Specification

1. Backend

uSee Data uses Node.js as the backend framework. One of the key features, which has been proposed and implemented, is to enable a user to upload a CSV (comma-separated values) file or a JSON (javascript object notation) file containing the dataset the user would like to visualize. We are able to have a user select a file from their local machine, and upload it to our node server. As soon as user authentication feature is implemented, the uploaded CSV file will be managed in a user’s data repository.

uSee Data uses MongoDB and Mongoose for schema-based data modeling. The data model currently implemented processes the metadata of the user-uploaded dataset (csv file). The server keeps the dataset in a file structure, but Mongoose module of the server manages metadata that is useful for generating each visualization. One of the most significant parts of the metadata is the header (the first row with field names) of the dataset. The data header is utilized in the interface when the user selects what to visualize. Another significant portion is the datatype of each field. This information is not provided by the user. The server parses the user-uploaded csv dataset and detects the datatypes of the fields. The current version of uSee Data classifies them into numeric data and non-numeric data. uSee Data will filter out unavailable chart types based on the datatypes of user-selected fields. The idea is that a user will upload the data they want to see, and based on which columns are interesting to them, we will give them options of various visualizations. Third party libraries used for backend are below.

Express: web application framework

Multer: file upload feature

Mongoose: MVC controller & database driver

CSV-streamify: CSV parsing

2. Frontend

The entire frontend was done with Google’s new framework, Polymer.  We used several Polymer elements including paper tabs, core toolbars, core sliders, paper toasts, custom popout elements, and several other.  Additionally, all the data visualization was done using D3.js and C3.js.  These are both data visualization javascript frameworks and industry standards.  All frontend followed the user interface and experience standards employed by several other cutting edge websites.  The front end scales fairly well on mobile, and is supported on all browsers.

Features

1. Chart Rendering

uSee Data takes a CSV (comma-separated values) or JSON (javascript object notation) file as user input. The server parses the dataset and generates a metadata object that is used to render a chart; the metadata object contains the number of fields, the number of rows, the data types of each field, etc.

2. Configurable Data Visualization

Once uSee Data renders a chart, it provides options for a user to configure the chart. The options are to turn on/off a data field, to shift X-Y axes, and to label the axes. The configurations are dynamically applied to the chart in real-time. For example, if a data field is turned on/off, the chart updates its graphical elements to include/exclude the column, and the axes of the chart automatically scales accordingly.

3. Embeddable SVG code

A visualization object users generates on uSee Data is a SVG (standard vector graphics). uSee Data supports an “Embed” feature that copies a code snippet of the SVG element to the clipboard.  This way when users create beautiful visualizations on our site, they can take it to their own blog, website, or dashboards!

Technical Challenges

A technical challenge for the SVG exporting feature is that of converting styling rules outside the SVG element into inline styling attributes of a graphics object. There are some open source libraries to support this feature, but none of them appeared to work perfectly in every case. Currently, SVG-exported visualization sometimes doesn’t have all the CSS rules, so that it cannot style its element correctly. To address this issue, XML parsing in more detailed level is necessary. The basic strategy is to fetch an existing CSS rule and to attach them a corresponding graphics elements.  This is a problem many others in web design are facing, and that hasn’t been completely solved.  The New York Times built a tool that in a ‘hacky’ way fixes this problem, but our team found it to be incredibly inconsistent and buggy.  

Github: https://github.com/stlim0730/u-see-data

Youtube: https://www.youtube.com/watch?v=w2MNEz4aWXc&feature=youtu.be

Overview

We created a beautiful and informative web application, which displays a set of user-selected stocks with financial metrics measured in days, months, quarters, and years. We provide the core information needed to give users useful updates about the stocks that they are interested in. This simplifies the user interface and removes the numeric clutter that our target audience usually disregards. The goal of this tool is to help our users make decisions in long-term investing.

Technologies Used

Polymer

Flex

Heroku

Polymer

We’ve build a system composed of Polymer elements. A toolbar on the side houses a custom stock-list item which populates itself with stock-ticker elements that are an overlay on top of paper-buttons. stock-card elements sit in the main body of the page, and they employ the use of paper-tabs and a custom stock-chart element. All these elements communicate with each other using core-signals, and they access data using core-ajaxs implented in stock-services.

Let’s talk a little bit about each of these elements.

Stock-Card

Stock-Card elements are composed of paper-tabs, and a stock-chart element. Events are passed into the card via core-signals, and data is pulled from our API through the stock-price-service element.

Stock-Chart

Stock-Chart is a Polymer implementation of Nvd3’s graph API. With a support for multiple date ranges, the stock-chart element handles data visualization through trend lines and colors corresponding to positive or negative growth.

Stock-List

Stock-List is a list of stock-ticker elements. Data is pulled from the API through the stock-service element, which is then used to populate the list.

Stock-Service

Stock-Service is an implementation of core-ajax that pulls data from the API. A list of stock symbols can be provided to get back a list of datapoint for those stocks.

Stock-Ticker

Stock-Ticker is an implementation of paper-buttons. It uses the Flexbox to beautifully layout the elements.

Data

Our application is built on top of Flask. We built a RESTful API that services the stock information for our front-end to visualize. This API pulls from both Yahoo finance and Google finance, but it cleans up the data nicely for easy access.

Lessons Learned

We’ve gained a lot from diving into polymer and attempting to use this new technology. We found that a lot of documentation is still lacking given its limited use, and we had to figure out a lot just by spending time and trying things out. Cross-browser compatibility is still an issue; it looks best on chrome, and buggy on Safari. If we had to redo the project, we would’ve spent more time earlier in the process to dig through resources to understand development practices and paradigms (of Polymer). We wasted a lot of effort trying to build workaround for features that we later found out were expressible through polymer elements. All in all, Polymer is a great tool to use, and it will change the way websites are built. Not only does it scale well between mobile and non-mobile browsers, but it looks clean and slick.

Check It Out

Github

Demo

Last week Eric Baldeschwieler talked with us about Big Data with a focus on Hadoop. This week, groups mainly focused on two aspects of Eric’s talk: Hadoop and its strengths, and the growing big data technology stack.

Hadoop is built on top of:

Common - the common libs that Hadoop uses

HDFS - the Hadoop Distributed File System, used for making many disks across many machines work together

MapReduce - the computing aspect of Hadoop, takes a lot of data and does work on it

Hadoop has a few strong points:

Cost effective - built to work on commodity components, Hadoop (and HDFS) was described by Eric as one of the cheapest storage options, with a free compute cluster thrown in

Scalable - it is very easy to add on more machines to a computing cluster which can Hadoop can immediately take advantage with little additional work

Reliable - Hadoop is failure tolerant, if a node goes down Hadoop can redistribute the task assigned to that machine

Open Source - As an open source project, Hadoop has many contributors and additions are built on top of it by many different companies and groups

However, Hadoop has a few weak points as well:

Single point of failure  - if the master node goes down, all the information on jobs is lost

Writing to disk - after each step in MapReduce, the results are written to disk which provides good durability and recovery but is also slow

Boilerplate code - to get a job up and running, you often need to write the same boilerplate code and it gets even worse when you have to run many jobs for a single data processing task

In the wider context, Hadoop has been the mainstay in the data processing stack, but there are many additions and alternatives to Hadoop that is machine processing more efficient, intuitive, and scalable.

Pig and Hive - SQL-like syntax built on top of Hadoop to provide a more familiar and expressive abstraction to the developer

Spark - an alternative to the MapReduce framework was takes advantage of in memory speed and streaming

First, A Brief History of Hadoop-

In the late 1990’s and early 2000s, with populations of internet users skyrocketing and data-collection pulling in more information than ever before, ‘big data’ had become a big problem.  Particularly for internet giants like Yahoo and Google, whose data-centers needed to perform huge numbers of computations in real time while leveraging vast sets of data, the scalability of resources, both for computation and for storage, became more important than ever.  Paving the way for improved scalability, Google published a seminal paper on MapReduce in 2004, suggesting that simple abstractions of mapping and reducing functions could be used to express rich computational models while being lending themselves well to scaling and reliability.  The paper was not accompanied by an implementation, but as excitement grew about the ability of MapReduce to greatly simplify cluster computing, a project called Hadoop, developed first at Yahoo! and later open-sourced, rose to provide a de-facto standard.

Hadoop was originally intended to provide a solution to in-house problems at Yahoo!, and consists of three major parts.  Hadoop MapReduce offers a relatively simple way to execute computations which had typically been run on clusters of 15-20 nodes across as many as 4500, drastically improving speed of computation.  Further, it allows for the use of off-the-shelf machines rather than expensive specialized hardware, severely cutting costs where deploying traditional solutions would be prohibitively expensive.  The Hadoop Distributed Filesystem (HDFS) manages the distribution of data across distributed storage, including the same off-the-shelf machines used for computation.  Redundancy is used as the main means of reliability here, rather than specialized machines: the ability to use off-the-shelf storage solutions cut the cost per unit of storage by up to 100x.  Since the commodity computers usable with Hadoop offer few fail-safe guarantees, failure recovery and redundancy are both major components of Hadoop’s solution.  These two complementary services were bound together by a resource management platform called YARN, which worked to schedule tasks across clusters and generally manage the use of computational resources.

The benefits to Yahoo! were significant in two ways.  First and most obviously, the process of crawling and indexing webpages benefitted from the improved speed of computation that came with working across larger clusters.  But Yahoo! also saw improvements in their research pipeline.  The scalability and speed of Hadoop, coupled with the storage capabilities that it offered, meant that researchers at Yahoo could run experiments that had once taken weeks in a matter of hours, drastically increasing the speed of iteration and the productivity of the research department as a whole.  

That’s not to say Hadoop is all roses and sunshine: there are a number of drawbacks to employing Hadoop as well.  First and foremost there’s a steep learning curve to using Hadoop - it is a non-trivial process to educate workers on how to use it in their workflow.  This is partially ameliorated by some recent solutions, like Pig or Hive, which allow for the use of SQL-like syntax to query data stored in HDFS systems, but can still be a major obstacle for companies investing in using Hadoop.  Realistically, it may only be worthwhile to work with Hadoop when traditional data-management methods no longer suffice - small businesses with simpler needs may be better served by simpler solutions.

Now open-sourced with Apache, Hadoop has grown from the fundamental functionality described above to an entire ecosystem of open-source projects, many also under the Apache license.  Some examples include Hive, which allows for SQL-like query of HDFS systems, Pig, which is similar but offers functionality for user-defined functions, Mahout, which is specialized for machine learning, HBase, which allows for the use of non-relational data formats, and many others.

This is not to say that Hadoop is the only game in town.  Recently, the project Spark, begun in the AMPlab at UC Berkeley and now also under the Apache label, has gained a significant amount of interest and open-source contribution.  It is also a platform for dealing with distributed computation, but with a focus on streaming computation, keeping data fresh in memory and only writing to disk when computation has completed.  This streaming based approach offers significant performance improvements over typical Hadoop jobs, and is particularly useful for realtime processing of large streams of data.  Spark also supports SQL queries with Spark SQL, and offers APIs in common languages like Java, Python, and Scala.  It can work with a number of data formats, including HDFS, and is thus relatively flexible.

Other solutions offer even more user-friendly environments for users who might be daunted by Hadoop or Spark.  Splunk is one recent contender, offering services like log management and attractive, automatically generated analytic dashboards.  Splunk even works with Hadoop systems, allowing for synergy connecting users who might be less familiar with systems like Hadoop with those who are comfortable using it.

Demo: http://file.pizza GitHub: https://github.com/kern/filepizza Screencast: https://vimeo.com/127437277

We are enabling private, significantly faster P2P transfer of large files over the web.

By using web protocol WebRTC, we are able to eliminate the slow and costly initial upload step traditionally required when sharing files via Dropbox/CloudApp/etc. Instead, the recipient simply downloads the file directly from the sender’s computer. Because the file is never actually saved on our servers, the transfer is incredibly private and secure. For large files, this is a big deal.

When using our service, the user would drag a file into their web browser, immediately receiving a shareable and unique “tempalink.“ When a recipient clicks this link, the download of that file will begin. Once the sender closes that web browser tab, the shareable link becomes dead, ensuring it can’t be accidentally found by a nefarious party in the future.

We’re incredibly excited about this service and hope to significantly improve file sharing on the web.

Technologies Used

Stylus and CSS3 for styling and animations.

PeerJS and WebRTC for peer-to-peer connectivity in the browser.

Socket.IO and WebSockets for connection signalling.

React and Alt (Flux) for isomorphic user experience.

Node.js and Express for the HTTP server.

Browserify and Babel for ES6 features in node and the browser.

What We’ve Learned

Now that we’ve completed the first version of FilePizza, we’d like to reflect on a few lessons learned while creating it.

We learned that JavaScript’s ability to run on both client and server allowed us to de-duplicate code and simplify our codebase. The server renders the page’s HTML using the exact same React code as the browser uses, sends this compiled HTML along with all store state, and the browser picks up where the server left off. This technique drastically reduces load times and is optimized to work with static page analysis for search engines. However isomorphism also presents unique challenges. The first is that there are minor differences between the Node.js and browser runtimes which result in errors when libraries are not tested against both. In particular, the Socket.IO and PeerJS client-side libraries are not designed to run in the Node.js environment, requiring us to create shims for them. Other challenges with isomorphic JavaScript include bootstrapping detesters correctly, serializing custom datatypes, and handling error pages properly.

We also (re-)learned that premature optimization truly is the root of all evil. We experimented with different ways to recursively build the downloading file in memory, using clever tricks like fire-and-forget and recursive concatenation to achieve speed-ups. We soon realized, however, that these tricks were in fact slowing down overall throughput and causing intermittent errors on slow connections that complicated our handling of large files. We ended up sticking with the simplest scheme of them all in our finished project because it was the only one which consistently performed well and correctly in our testing.

JavaScript is a beast of a language. There are an incredible number of “gotchas” sprinkled throughout its syntax and semantics. Fortunately, ES6 features like the new classes syntax encapsulate commonly used patterns while remaining backwards compatible with older browsers. Babel makes it easy for developers to use ES6 features today without waiting for native implementation of these time-saving and bug-preventing improvements.

Finally, we learned that working together to solve design challenges is the best way to get a clean and intuitive user interface. The original name for FilePizza was WebDrop, which was rather boring and uninspiring. When looking for another name, we went through all of the new TLDs available with different variations, before finally choosing “file.pizza” because we were delivering files. The name had character and stuck. We changed our graphics to reflect a more fun and lighthearted theme, changed our URLs to be a combination of pizza toppings, and changed the tagline to “Your files, delivered.” Sometimes all it takes is a bit of creative thinking to uncover a unique brand and style.

What We Would Do Differently

Overall, we were really happy with our technology stack.

A lot of the logic for serializing and deserializing state in stores is hairy because we used custom data types to represent files. This is a fundamental limitation of JSON: it’s exceedingly cumbersome to represent custom element types without creating your own parsing algorithm. Instead of using JSON as a serialization format, we might consider using EDN or another extensible serialization instead.

Additionally, React currently renders the entire page, starting with the HTML element itself. Although idealistically more pure, this seems to be unnecessary since the “body” element is the only thing that ever changes. This also causes hard-to-diagnose problems when stores are based on transient information resulting in checksum errors on the client. In the future, we’ll only use React to render the <body> element for isomorphic applications.

Now that the semester has ended, we have completed everything we proposed to do for each of our milestones. We created an open source library called Dynamic3 that enabled users to easily make dynamic graphs featuring real-time data. We created the three main graphs that show off Dynamic3’s potential. These graphs emphasize our library’s focus on displaying dynamic data through animation. The future of data visualization is in displaying dynamic data, not static data. Dynamic3 is a step in this direction by providing users with an easy way to get started with dynamic data visualizations.

As we began our project, we learned that d3’s library is complicated, intricate, nuanced, and that it takes a while to learn the basics. In that sense, we really found a purpose for our project in making it a lot easier for anyone to make simple data visualizations on top of d3. We confirmed this when we began coding our library. Although setting up an initial graph in d3 does not require too many components, the d3 calls to do so were not entirely intuitive. Thus, this motivated us as we were building Dynamic3 to make our library functions self-explanatory and easy for the user to use.

But something we also should’ve done — and would do if we were to redo the project — would have been to research some other open-source data visualization libraries earlier on to figure out the features we would want to implement in Dynamic3 to differentiate it more from other libraries. Another thing we would do differently is start on components related to the library earlier, such as our website and documentation in order to create a better presentation of our library and its use cases and to find bugs.

As we worked on this, we also learned a lot about what it takes to build an open source library. We had to write code that complete outsiders would be able to easily understand. We also had to create documentation that outside users can easily reference to build custom dynamic graphs.

Github: https://github.com/jeffreychan637/dynamic3

Website: http://jeffreychan637.github.io

Screencast: http://youtu.be/Y1qgQO46Vqs

Go reminds me a lot of the late Zune HD.

The Zune was a great piece of technology that was engineered as a direct competitor to Apple’s iPod Touch. The screen was a gorgeous OLED, the interface ventured into flat design years before it was a fad, and under the hood, it had a surprising amount of power. I repeat: it was a great device. And yet, it never caught on.

Similarly, Go seems like a powerful alternative to Python, Ruby, and C++, and yet it doesn’t seem to have the same pervasiveness throughout the programming space. There could be a lot of reasons for this slow adoption (we’ll just leave Roger’s Diffusion of Innovations curve right here.)

Academic jokes aside, this really is a surprising state of affairs, since Golang is highly successful as a language on a number of fronts. As discussed in Rob Pike’s talk, the language is fairly readable for anybody with sufficient programming knowledge; it’s literally built to scale to massive levels - even Google; as a result, it’s highly suited to these scaling problems, and can interface with other languages; finally, Go has a single way of expressing most operations, meaning that tools like gofmt can easily format a document written in Golang, and there’s even interesting work in programmatically rewriting and modernizing old code.

Although “finagle” generally has a negative connotation, it’s fun to see how Twitter is playing with the idea of deception to make distributed computing on large systems more efficient. It’s framework for dealing with concurrency, Finagle, works to make networked systems - referred to in this talk as “the dismal computer” - into something more efficient, and a little less frustrating.

The problem space Finagle is working to address feels massive; large-scale infrastructure is massively complicated, with various machines working alongside each other, constantly shifting and morphing network topologies, and unreliable nodes. This makes a process that would be (relatively) straightforward on a single machine enormously complicated.

A number of technologies are available to solve this problem; Finagle offers a low-level solution, with one of its most attractive features being its protocol-agnostic design. In heterogenous networks, it’s able to facilitate machines that might be working under different protocols, providing a tidy solution to one of those massive issues. Additionally, it’s distribution mechanism allocates tasks based on failure rates, latency, load, and other metrics for health. This intentional distribution has obvious implications on efficiency, and companies like Tumblr and Pinterest have taken notice and adopted the flexible technology.

As noted on the AI Computer Learning blog, Akka by TypeSafe is a solid alternative (or a potential complement) to Finagle; however, its inability to communicate over multiple protocols make it a little more rigid. Communication between nodes needs to be simple, and infrastructure benefits from being relatively homogenous. Still, Akka seems like a fairly popular solution across industry, and introduces a mindset about how processes should be handled with its actor model, which protects information about state and the data an actor is handling under most circumstances.

This week, Eric Baldeshwieler came to talk to us about the development of Hadoop. Hadoop is an open-source big data framework. Eric is the co-founder of HortonWorks, a Yahoo spin-off devoted to developing and improving Hadoop for use in the commercial enterprises. Although Hadoop was first created at Apache, Eric describes himself as the ‘midwife’ of Hadoop, helping to make it relevant in a modern marketplace.

Structure

Hadoop is composed of 4 main “parts” which have various roles in data processing:

Hadoop Common

These are shared tools for the other 3 parts of hadoop.

Hadoop Distributed File System (HDFS)

HDFS is designed to store data on clusters of commodity hardware, handling failures and providing high bandwidth.

Hadoop YARN

YARN is one of the tools that makes hadoop particularly interesting because it handles the process scheduling and makes working with a cluster of computers much easier.

Hadoop MapReduce

MapReduce is the part for which hadoop is most famous. MR abstracts away much of the details of the other parts of hadoop and handles many intermediate steps of the data processing process making programs easier to write.

Pros

Reacts well to failures: One of the greatest advantages of this system is the built-in expectation of failure. No one wants to spend lots of money on valuable AWS time just to have their whole job fail because of crashing nodes at the last minute. With Hadoop, you can package code for different nodes and relaunch them in reaction to failures. With the Hadoop Distributed File System (HDFS), data is stored on three computers so it can easily replace lost data or nodes. This “self healing” system helps keep jobs going even after encountering hardware failures.

Cost: The relative cost of running a Hadoop cluster is considerably cheap when compared to many alternatives. Hadoop’s storage methods can make hardware costs up to 100X cheaper.

Cons

This past week’s blog posts were very centered on Finagle, and the idea of scaling. Marius Eriksen made this the focus as he talked about how twitter handled their scaling problem.

A common theme among the blog posts were the idea of data centers and horizontal scaling. Unlike vertical scaling, horizontal scaling is most focused on the idea of adding new computers to the data center. One thing Marius made clear was that data center computers really are not meant to handle modern day applications, as very little has been done to improve them in years. The problems with data centers were a clear focus, with a common theme being addressing the fact that data centers often involve partial failures and painful abstractions.

Another common theme was the idea of RPCs, and using Finagle as a RPC as a means of avoiding the headaches commonly seen in data centers. This idea communicated that data centers could use RPC as a means of creating a more natural way of communication between the user and the data center. Using finagle’s model of fixtures, services, and filters, the use of an RPC becomes much clearer.

Hadoop and it’s family of tech

Hadoop was created to solve the problem of reliable large scale data processing. Hadoop not only provides a solution that enables massive parallelism on clusters of machines, but it provides self healing mechanisms, ie in the event a machine goes down data is already replicated by the filesystem layer and the machine’s work can and will be automatically restarted elsewhere.

Strengths

In addition to the benefits mentioned above, Hadoop provided a common opensource platform, which enabled major industry players to get behind the same source and this has provided leverage Hadoop/the community.

Weaknesses

Hadoop is built for large-ish jobs, it actually is much less efficient to run smaller jobs on a Hadoop system, speaking from experience, especially if the network interconnects are slower. This is become the system first replicated the dataset onto a large number of nodes before processing it. Other notes

This week, we had to pleasure of hearing Eric Baldeschwieler talk about the history and development of Hadoop. On the surface level, Hadoop is an open source framework for distributed storage and processing. It was first envisioned by Doug Cutting and Mike Cafarella in 2005, and the name Hadoop actually comes from Cutting who named it after his son's toy elephant at the time. Although Hadoop was an open source project started by Apache, it wasn't until a team at Yahoo took the prototype and made it into what it is today when it released what it claimed as the world's largest Hadoop production application. This application was the basis of the Yahoo web crawler that is still used today to scan through billions of web pages within less than a second to bring up the vital page this we all use in our everyday lives.

Hadoop is made up of two distinctive layers. The first layer, being the computation layer, is capable of processing many jobs across many computers, and the second layer, the HDFS (Hadoop distributed file system), which stores all the data on commodity machines. The beauty of Hadoop beyond its simplicity is that its easily scalable because any increase in sizes only requires more machines if any additional machines are even needed, and these machines are often times cheap and easy to acquire as there are no special requirements that limit what can be used for these computations. With that in mind, there will be a lot of failures across multiple clusters of these machines, but Hadoop was built with a lot of flexibility in mind as well so that if a machine fails, its jobs are merely picked up by another which is working and the process continues in parallel as if nothing happened. A standard Hadoop cluster is made of commodity computers with 4-6 hard drives each SATA 4 TB, 64 GB of ram, and commodity switches can prevent rack level failures by switching other racks.

Eric Baldeschwieler: Hadoop in Context

Eric Baldeschwieler, now a private technical advisor in the Valley, led the team at Yahoo who brought Hadoop to the commercial stage. Baldeschwieler described Hadoop as a product built with two primary layers including a computation layer and a distributed file system later. He offered insights into the origins of Hadoop by providing the reason for its conception. Yahoo, one of the major internet networks and largest advertising giant on the web, created Hadoop to crawl and index the internet. The required capabilities for such a robust and mission-critical task included an innate ability to grow in processing power in a flexibly, yet economical manner. Baldeschwieler repeatedly noted that the open source aspect of Hadoop cost quite a bit of money and management resources early on, but serves as a key idea in Hadoop’s scalability and multi-purpose features.

One of main benefits of Hadoop is that it reduces the interaction with databases, the continuous availability of which are essential for both enterprise and consumer-facing applications. A comparison that particularly illustrates the strength and limitation of Hadoop is one with Elasticsearch.  Elasticsearch, for example, relies on indexing and retrieving search results while interacting heavily with the database. Hadoop runs clusters that collect and analyze log data directly from servers so that the load is scattered across all servers instead of concentrated on the database. Elasticsearch excels in small amounts of data, low latency and high availability while Hadoop focuses on tasks involving enormous amount of data and high analytics complexity, such as scientific research and testing of new advertising metrics.

The other key feature that allows Hadoop to prosper is its cost-efficiency. Its processing capabilities are built on cheap Intel commodity servers, which are stripped from all unnecessary accessories except raw computing power. For example, the cheap x86 servers only cost about $4k for each node. This is compared to other relational database deployments which run at a cost of over $10k per terabyte.