CrowdProcess

Ok, we admit it, we haven’t been posting enough, but for good reason: we’ve been working on running Excel, in parallel, in JavaScript.

Three questions arise: “what?”, “good God why?”, and “how?”

Regarding the first one, we’ve been compiling Excel to JavaScript, and running it on multiple nodes using our JavaScript Distributed Computing Platform.

As for why, we’ve found that there are a number of applications such as sensitivity analysis which require more computing power than a single computer can provide in useful time. Bringing the computation into JavaScript helps a bit, distributing it over multiple nodes helps a lot.

Finally, the “how” question. We decided to build a new compiler and use the open source library formula.js (started by the awesome guys at Stoiku, which we forked) to have a JavaScript implementation of Microsoft Excel formulas.

On top of this, we built a new application, which does stress testing of Excel Models, but this is the tip of the iceberg for what this can do. 

David Sousa and his team research advanced MRI techniques, which help diagnose diseases such as Alzheimer's. We are proud and honored to see our technology aiding such a noble pursuit. We will leave the rest of this blog post to David Sousa's explanation of their work. Our deepest thanks and best wishes to David, and the whole IBEB research team. The following is a Guest Post by David Sousa, IBEB Researcher

________________________________________________________

Magnetic resonance imaging (MRI) has been used for decades to investigate the anatomy and physiology of the human body. This technique, allied with a deep knowledge of biological tissues, has a wide range of applications, specifically, medical diagnosis and staging of diseases like Alzheimer.

  A special MRI technique, called Diffusion-weighted imaging (DWI) can inclusively study the motion of water molecules in the brain, and with that get a glimpse of the brain tissue cellular architecture. For instance the picture below, on the left, represents a map of the water diffusion coefficient in the brain, i.e. the easiness with which water molecules are able to move on tissues comprised of neurons and glial cells. On the right is represented a map of water diffusion directionality, which can for instance translate the directionality of neuronal axon tracts.

  Intuitively, water diffusion coefficient values are low in regions of the brain where the tissue complexity is very high, as water molecules movement is more constrained due to the presence of cell membranes, organelles, large proteins, or even due to low cellular membrane permeability.

For particular neurological diseases large micro-structural changes are observed in the biological tissue. We can identify these areas because as the brain tissue degenerates its level of complexity decreases and water diffusion increases. This means that the MRI image will look different from what we expect to see in a ‘normal brain’ MRI image.

Since we cannot look at a living human brain at each stage of an evolving disease, we cannot know what happens at the micro-structural level at each stage, specifically to the intracellular volume fraction, cell radii or permeability. These could be some of the parameters varying over the disease course and hence affecting water diffusion. That’s where computers can help us!

If we simulate different kinds of cellular architectures, based on previous knowledge about biological tissue structure, and put some independent random walkers in there, the water molecules, we get the value of the water diffusion for each kind of architecture. Now, think backwards! For a certain value of water diffusion we can guess the kind of cellular architecture we may have.

Water diffusion in a free medium has a Gaussian behavior (free diffusion). In a complex medium like a biological tissue, diffusion is not free. Its behavior deviates from a Gaussian curve. Kurtosis of diffusion quantifies this deviation, and hence, the complexity of tissues.

In the present article I focus on the correlation between the varying volume fraction of the intracellular space and the values of diffusion coefficient and kurtosis of diffusion using Monte-Carlo simulations based on the methods used by C-Y Lee et al. [1].

As an example of how the simulation is performed consider a varying intracellular volume fraction (V) between 10% and 80% of the entire space in 14 steps. (When I say entire space I mean 0.4x0.4 mm2 which could be the resolution of our high-field MRI image, so the corresponding tissue area of a pixel). So 14 cellular architectures (or configurations) will be generated for the random walkers. As an example, here are 2 pictures of cellular architectures corresponding to the intracellular volume fractions 10% and 80%:

  Now, if I put 1 million random walkers in each cellular configuration and each walker performs approximately 1,000 time-steps, considering collisions with cell membranes this can be extremely computationally expensive. With my Intel i3-4010U processor this takes approximately 45 minutes! That’s way too much time to wait if I need to repeat the process with a different cellular configuration or with a different cell radii distribution. In fact, there are many important parameters to consider in this experiment. That’s why I need CrowdProcess.

  With a lesser number of walkers (for example 1,000) CrowdProcess allows me to run the same experiment 10,000 times (or much more) simultaneously! Which is the same as running the experiment once with 10 million walkers but in 18 seconds! These are the results for the example I just gave you:

  To get a visual idea of what’s going on 10,000 times simultaneously:

CrowdProcess allows me to do my research 150 times faster. I hope this way neurological diagnosis becomes faster in a near future.

One question we get asked very often is how fast a supercomputer made entirely of web browsers is. Given that GFLOPS are relatively meaningless to most people, we often compare to how fast something would run locally, or on AWS.

In one example, we ran a Monte Carlo Pi estimation on:

Intel Core i7 2600 Laptop

AWS's cc2.8xlarge compute optimized instances (88 ECUs)

CrowdProcess (at 8900 workers)

The laptop used one node.js process, the cc2.x8large instance used Hadoop, and CrowdProcess obviously used Web Workers in a diversity of browsers.

Results, in millions of tasks per second (each task was composed to 200M trials):

  With CrowdProcess at full power, the platform performed the equivalent number of calculations to 4750 cores of i7 processors, or 3350 cc2.8xlarge AWS instances (294.800 ECUs) .

Not surprisingly, CrowdProcess at full power easily beats the compute nodes, what is surprising however is how well each browser does. The laptop is running on a single thread, and each browser node is getting around 50% of that speedup.

A major caveat is in the random number generators, as the browsers lack uniformity in the way the produce random numbers, and certainly produce them differently from the Java implementation we ran on AWS. This can make the results biased towards the JavaScript run.

A fairer comparison should have all nodes running the same random number generator, and run more than one algorithm (these were pure Monte Carlo Runs). More comparisons to follow.

Naturally, there is always the case for GPUs, and we are aware that any Telsa could beat all of these numbers. It should be noted however that the vast majority of developers only know how to write code for CPUs. Besides the vast majority of languages being easier and simpler to write than OpenCL, Javascript is even simpler than tons of other languages (go ahead and try our Playground) which makes it an extremely popular language. And the best part, it's quite fast.

In a while, we intend to start looking into the potential to use WebCL on CrowdProcess, that will make it more competitive with GPGPU computing. For now, we are very happy to see how fast it is compared to all accessible alternatives. 

Ok, it is braggy. But we are just that proud. As of today, there is a REPL for launching distributed computing tasks over thousands of nodes, that even a 9 year old can understand. 

One of most fascinating things about working at CrowdProcess has been to get to know the users.

Amazing work, by physicists, statisticians, medical researchers and a plethora of developers and engineers running fascinating problems of distributed computing on web browsers.

As our users’ jobs fundamentally belong to them, we are often saddened not to be able to publish everything that goes through.

Which is why CrowdProcess is paying 1000$ for everyone who comes up with an idea for an application, develops it and releases it as open source code.

Be sure to read the complete announcement and rules before entering, but it’s more or less the following:

You come up with a cool idea for an application that solves a real world problem and is suitable for CrowdProcess (we can help);

You develop and release this application as a web app;

You get paid;

Of course you can submit multiple applications and make more money!

Need some inspiration ?

Here’s a nice hello world application that calculates the value of Pi.

Gentlemen, start your engines!

One of the fun things about working at CrowdProcess is meeting the amazing developers who work on bleeding edge computational challenges. One of them, Bruno Vieira, has been working in bringing bioinformatics into the world of Javascript.

Besides his work in helping us with genomics in the browser, he has also been working on bionode.io, a Node.js library for client and server side bioinformatics.

His reasoning is that having bioinformatics methods that can run on the browser could immediately benefit biological visualisation projects like BioJS or web genome browsers. It would also allow using those methods on browser based distributed computing grids like CrowdProcess. Thanks to Node.js, the same code could be run on the server to perform some tasks, which are currently performed with other bio libraries, such as Biopython or BioRuby. In this sense, one can think of bionode as an underscore.js for bioinformatics.

Over a year ago we had a vision. A world where scientists, researchers, and those working for the benefit of humanity could have access to supercomputing resources, for free. 

Since then, a lot has happened:

We connected millions of different devices, ran huge computations, and helped developers work on incredible projects, from catastrophe simulation to genomics.

We launched an enterprise version, and started working with companies on leveraging their own resources.

We were joined by some of the most awesome investors, who have become core members of our young team (more on this very soon).

"Computer programs that evolve in ways that resemble natural selection can solve complex problems even their creators do not fully understand." - John Henry Holland

A few months ago, one of our users took on the fascinating task of implementing a genetic algorithm for optimization of job scheduling on CrowdProcess.

In a rather crude nutshell, genetic algorithms act the same way as nature: they take a group of candidate solutions and allow them to mutate, reproduce, and crossover. They then keep the best results (as defined by a fitness function) from one generation to the next. For more on genetic algorithms, we recommend this book by David E. Goldberg

Genetic algorithms are, by their nature, prone to parallelism. Going back to the parallel (no pun intended) with nature, the more candidates and generations you have, the more likely one of them with reach a better optimum. In nature, that optimum is simply the ability to survive.

So how does this overlap with a distributed computing platform made of thousands of web browsers? Extremely well, it turns out. The solution is to make each web browser an independent population, run each simulation in isolation from all others, and return the best result from each one. These can then be compared locally, and the "best of the best" chosen as the optimum.

In the case of the current problem, the objective was to find the order of jobs that gave the fastest completion time of the whole production cycle. The problem is essential for production management, as proper planning of jobs can lead to significant savings in production costs.

Applications such as telecommunications routing, financial forecasting or fleet logistics use Genetic algorithms, and we hope that the CrowdProcess platform can bring them into more common use.

The source code is on this link, and we encourage developers to try it out, and to run Genetic algorithms on thousands of web browsers in parallel.

Our heartfelt Thank You to Jerzy Duda from AGH University of Science and Technology in Krakow, Poland for providing us an initial code of GA and the test problems.

We are very interested in supporting developers who would like to run GA algorithms on CrowdProcess for their own use cases, and potentially add functionality such as editable fitness functions, and control over parameters such as population size, crossover type, mutation type, etc.

CrowdProcess is a very particular distributed computing platform. It has browsers connecting and disconnecting all the time, and their number varies considerably during the day.

So how are job times affected by these variations? We decided to run a very basic experiment, which give a (non scientific) feel for this. If you want to know how we did it, read on. If you are more interested in the cool graphs, scroll down. 

Here is what we did:  We took a simple Run function, which uses a Monte Carlo simulation to calculate pi with 1000000000 points, and returns only the time it took to calculate. In node.js, each run takes about 12 and a half seconds.    

function Run() {    var inQuarterCircle = 0,    n = 1000000000,    i = n;    timer= Date.now();       while(i—) {          if (Math.pow(Math.random(),2 ) + Math.pow(Math.random(),2 ) < 1) {             inQuarterCircle++;       }    }    var pi=4*inQuarterCircle / n;    return (Date.now()-timer)/1000; }

Next, we made 4 json files, and called them (to be very original), small, medium, large and huge. Each one got a different number of empty objects, corresponding to the number of tasks to be run (our function does not take an input in this case)

small.json:          2,080 tasks medium.json:    10,000 tasks large.json:          20,000 tasks huge.json:          60,000 tasks

Excellent. Now it remained only to run them on the platform. During the different runs on the platform (across multiple days), the number of browsers varied  considerably, as follows:

So it was important to control for number of browsers, and each experiment was run multiple times. 

Each job we sent was returned with the times that each task took to execute. The interesting thing was to determine how different browsers took different times to run the same task. 

The average time on the local run, in node, on a Toshiba L50 (Intel® Core™ i5-3230M Dual Core) was 12.771 seconds. 

On the browsers, the distribution looked more like this:

This is an example from a small.json. Interestingly, a full 27,85% of browsers outperformed the local run (72.12% took longer). If you are wondering about the long tail, 3.3% crossed the 1 minute mark, and none past the two minute mark (because of a platform timeout at 2 minutes).

Interesting… now on to the experiments themselves! If a single task would take 12.771 seconds, and a small.json has 2080 objects, then the expected sequentially run would take a bit over 7 hours. To the platform! 

Not bad, the worse result was a 172x speedup, and the best was 288x. Time going down pleasantly linearly with the number of browsers.  Beyond, to the medium jobs! 10k tasks, expected time 35.7 hours. 

Again, beautifully linear. as expected, and speedups ranging between 240x and 517x. Not much of a challenge for a distributed computing platform though…

So up again, to 20k tasks. (expected time: 70.9 hours).

All linear, except for a massive outlier. The most likely answer is that the platform was being shared by multiple developers running different tasks at the same time. Speedups at a respectable minimum 238x and maximum 643x.

Finally, the run corresponding to the ‘huge’ file, with 60k tasks (“huge is a major overstatement, the platform has run jobs orders of magnitude higher, but it sounded like the natural thing after “large”).

Expected time for the “huge” file was 212 hours (a bit over a week sequentially).

Max speedup at 755x, and a minimum at 268x. One question springs to mind, what happens if we plot speedup vs number of browsers? 

Well, this happens: 

Interestingly, the job sizes have a clear impact on speedup, and not only on time of computation, even though the tasks often outweigh the number of nodes by more than an order of magnitude.

Which begs the question… What happens to speedup per browser, with the number of browsers?

Speedup per browser went up to almost 0,5 with a large number of tasks on a small number of nodes, but clearly decreases as the number of nodes increases. It seems comparatively always higher on larger jobs.

CrowdProcess began February with a very considerable improvement in the scheduler, and consequent improvements in platform performance. With this we have been able to reduce the processing time per job over 29%.

Now that we’ve told you the good news, let us explain how we came to this result, and how we compare results on a browser powered distributed computer:

As you might know the number of browsers connected to CrowdProcess’ platform is volatile by design. We therefore ran experiments when the number of browsers was fairly stable, ranging between 1435 and 1588.

We used a sample of  >300 jobs for each experiment: one using the old scheduler, and the other using the new one. Next, we calculated the accumulated average time per job, as the number of jobs increased.

We used a dummy Job, in which each task would wait 2 seconds before returning 1.  Each job had 1500 tasks. (more about jobs and tasks on CrowdProcess) We cleared the data of outliers, defined as results over 2 standard deviations from the mean.  We then plotted the accumulated average job time, which is what you can see on the graph.  

We are currently working with two experts in Bioinformatics: Prof. Jonas Almeida, a researcher at the University of Alabama at Birmingham (UAB) and Prof. Alexandre Francisco from INESC in Portugal. We developed with a browser powered version of a Microbiome Sequence Alignment tool.

"Microbiome Sequence Alignment?! What’s that? Please tell me, CrowdProcess, I want to know!", you say.

So, imagine you put the content of your stomach into a sequencing machine - a machine that translates genes into genetic code (ACGTs) - and you get a “book” of your stomach contents. A book written in a sentence using only four letters (ACGT). Gibberish.

You then get your dictionary of ACGT-ish - a database that matches known organisms with the respective sequences - and you read it to find out what microorganisms, bacteria and bugs are populating the contents of your stomach.

Now that you know what Microbiome Sequence Alignment is, you might be wondering what a distributed computing platform powered by web browsers has to do with it.

Here’s your answer: in the development of the work we are doing with Prof. Jonas Almeida and Prof. Alexandre Francisco, we’ve made a simple demo just to show what’s possible with CrowdProcess.

We made this because we believe that massively parallel computing obtained through web browsers can pave the way to a future where distributed computing is accessible to everyone and anyone. That and the ease of use of JavaScript can truly bring (computing) power to the people. Aditionally, by using CrowdProcess behind the firewall, we want to help hospitals and clinics bring genome sequencing to everybody.

We want to recycle the internet's wasted computing power to help solve humanity's toughest challenges, and that's only going to happen if we all do it together.

That means that every developer should be able to use CrowdProcess. How can we accomplish that ? Well, a simple API sure is a start, and even better, an extremely easy to use Node.js module:

Pretty simple right ? Especially if you're familiar with async's map. It does the same thing, except instead of using your computer, it runs in more than 2000 browsers at the same time.

Besides taking an array and calling back with the buffered results, it's also a handy Duplex stream:

I particularly love line 32.

It was never so simple and easy to run something in 2000+ nodes. And we'll keep trying to democratize access to computing power. It's now up to you to use it for the greater good.

To our users, our clients, our partners, our investors, our advisors, our startup fellows at Seedcamp and Startup Lisboa, to all our friends:

One of these days, Pedro Afonso, one of our awesome developers, said that we were "making the future happen sooner", by speeding up distributed computing applications such as healthcare diagnostic tools, so that "our friends could live longer and we could spend more time with them".

That quite sums up our mission: making the future happen sooner and giving everyone more time with their loved ones.

But our mission only has purpose if we all enjoy the time we have together.

So, with that in mind, enjoy the time you have with your loved ones, wherever you are. Merry Christmas,

The CrowdProcess Team

• • •

This post was written by Tiago dos Santos Carlos, Communications Strategist at CrowdProcess.

Follow @correcto & @CrowdProcess on twitter for more stuff like this and the occasional random tweet about everything and nothing in particular.

A few months ago we shared with you a vision: a world where a tiny bit of every browser's processing power could be brought together to achieve a greater good.

Flame graphs are a way to visualize hot-CPU code-paths. Here is how we managed to produce one.

First, you’ll need a system that supports DTrace. There are a couple of choices.

OS X and BSD support DTrace but they are missing support for ustack helpers, and without them DTrace can’t make sense of JavaScript stack frames.

DTrace on Linux doesn’t seem to be very popular, but you can try your luck on Ubuntu following a post by Jared Koumentis.

For now, seems that the safest choice is to use an IllumOS based OS:

SmartOS

Sponsored by Joyent.

There’s this guide by Stu Radnidge, but it involves using VMware Player, and in OS X that means buying VMware Fusion.

OpenIndiana

Features a Gnome 2.6 GUI and works on VirtualBox.

Anton Whalley has an excellent post to get you started.

OmniOS

Sponsored by OmniTI.

My first option was SmartOS since its sponsor is also the main sponsor of Node. I decided not to pay for VMware Fusion, so I tried my luck with VirtualBox, but to no avail.

OpenIndiana looks nice, but OmniOS is the only one for which I could find a Vagrant box and I don’t need a GUI. So I figured that OmniOS would be the easiest choice.

If you don’t have Vagrant (> 1.2.2), go get it.

First setup an instance of OmniOS with Vagrant:

# First install the box vagrant box add omnios http://omnios.omniti.com/media/OmniOS_r151006c-r1.box # Get a folder for the instance mkdir omnios cd omnios vagrant init vagrant up # Add ssh settings to your ~/.ssh/config vagrant ssh-config | sed -e 's/default/omnios/g' >> ~/.ssh/config # Log into the instance ssh omnios

Afterwards, install build tools and nodejs on the instance:

# Install build tools pkg install \ developer/gcc46 \ developer/object-file \ developer/linker \ developer/library/lint \ developer/build/gnu-make \ system/header \ system/library/math/header-math # Add to path ln -s /opt/gcc-4.6.3/bin/gcc /usr/bin/cc ln -s /opt/gcc-4.6.3/bin/g++ /usr/bin/g++ ln -s /usr/gnu/i386-pc-solaris2.11/bin/objdump /usr/bin/objdump # Install Git sudo pkg install pkg://omnios/developer/versioning/git # Install Node git clone https://github.com/joyent/node.git cd node git checkout -b v0.10.15-release origin/v0.10.15-release ./configure --with-dtrace --dest-cpu=x64 gmake CXXFLAGS+="-ffunction-sections -fdata-sections" sudo gmake install # Add to path sudo ln -s /usr/local/bin/node /usr/bin/node sudo ln -s /usr/local/bin/npm /usr/bin/npm

Now, get your Node program running, and make some use of DTrace:

# Tell dtrace what process to sample by specifiying a pid sudo dtrace -n 'profile-997/pid == 12345 && arg1/{@[jstack(150, 8000)] = count(); } tick-60s { exit(0); }' > stacks.out # or a name sudo dtrace -n 'profile-997/execname == "node" && arg1/{@[jstack(150, 8000)] = count(); } tick-60s { exit(0); }' > stacks.out # demangle the c++ symbols gc++filt < stacks.out > ~/demangled.out

To explain a bit about that dtrace command and how you can tune it a bit, some important information from the dtrace blog:

profile-97

samples on all CPUs at 997Hz. We use 997 instead of the more obvious 1000 to avoid oversampling periodic activity.

execname == "node"

the probe’s body only fires when the currently running program is “node” to avoid sampling other processes on the system. You could omit this and sample everything.

arg1

arg1 for profile probes is the current user-level instruction pointer, so testing against zero here ensures that we only fire the probe body for samples that interrupt userland code.

@[jstack(150, 8000)] = count();

the body of the probe populates an aggregation that counts the number of times each userland stacktrace has been seen. We use jstack to collect the userland stack trace. We gather 150 frames and give the ustack helper 8000 bytes of buffer for translating the stack frames.

tick-60s { exit(0); }

causes the DTrace invocation to exit after 60 seconds.

> out.stacks

saves the output to out.stacks rather than stdout.

Now on the host machine:

# Install stackvis npm install -g stackvis # Copy the demangled stack to the host scp omnios:demangled.out stacks.out # Generate the flamegraph stackvis dtrace flamegraph-svg < stacks.out > stacks.svg

Now open stacks.svg with your browser and stare at it!

If everything went well, you should have something like this:

Again, from the DTrace blog:

Each box represents a function in the stack (a “stack frame”).

The y-axis shows stack depth (number of frames on the stack). The top box shows the function that was on-CPU. Everything beneath that is ancestry. The function beneath a function is its parent, just like the stack traces shown earlier.

The x-axis spans the sample population. It does not show the passing of time from left to right, as most graphs do. The left to right ordering has no meaning (it’s sorted alphabetically).

The width of the box shows the total time it was on-CPU or part of an ancestry that was on-CPU (based on sample count). Wider box functions may be slower than narrow box functions, or, they may simply be called more often. The call count is not shown (or known via sampling).

The sample count can exceed elapsed time if multiple threads were running and sampled concurrently.

The colors aren’t significant, and are picked at random to be warm colors.

• • •

This post was written by Igor Soarez, Platform Developer at CrowdProcess.

Follow @igorsoarez & @CrowdProcess on twitter for more stuff like this and the occasional random tweet about everything and nothing in particular.