#wolfram research

... basically what we can do with a plus sign on a computer. [image: woman pressing a key on her laptop at her desk *bloop*]

Because neural nets are experiencing so much attention right now, they're going through a bit of evolution of their own. [image: network graphs with walking feet climbing onto land] New ways of constructing neural nets are coming out every day, and bringing with them...

... lots and lots of acronyms, with each new way of doing things trying to outperform the others on benchmark tasks. [image: Multiple acronyms in the background with a bar graph labeled "% classification error" on the y-axis and various acronyms labeled on the x-axis]

Despite all this variety, every neural net contains the same two common building blocks: There's a part that's linear, which is usually code for "things work out nicely." [image: linear graph illustration] Then there's a part that's nonlinear, which often means "lol good luck trying to prove anything." [image: nonlinear graph illustration depicting a dragon over a squiggly line]

The linear part is that synthesis step. In a neural net, there are a bunch of nodes (or neurons) that are connected by edges with different weights. [image: multiple equations being merged together with arrows being pointed at a blue dot that is generating the solution] Each node adds up the input from all its neighbors, multiplied by the weight of the edges connecting them, to get a number representing its state.

The nonlinear part is the decision step. Based on the number from the linear step, the node decides: Blips? No blips?? [image: confused blue dot with thought bubbles including blip or no blip]

Here's a cool thing: For as scary and annoying as nonlinearities can be, the ones that seem to work best for neural nets are all... pretty nice. [image: three graphs; one with negative values going to 0 and positive values going to 1, another being same as the last one, but smoother, and the last one having negative values going to 0 and positive values staying the same]

Which is kind of amazing. [image: "Step 1. Adding & Multiplying" with arrows pointing at a dot "Step 2. A Simple Nonlinearity" with a dot and a nonlinear graph] Neural nets have crushed the competition in tons of machine learning challenges, yet the basic rules for their nodes are simple and clean. It's the fact that they're combined into networks that give rise to the bogglingly good performance.

What do these networks look like? There are input neurons, that ingest inputs. Like the third pixel from the right and five down in an image. [image: cat picture with pixels that have arrows pointing at input neurons] There are output neurons, that give outputs. There might be ten of these if you're trying to classify the digits 0 through 9, or a LOT more if you're trying to identify the content of an image. [image: output neurons saying "cat", "shoe", and "bean"]

There are typically many more neurons in between the inputs and outputs, organized into layers. [image: network graph] They talk to their neighbors by alternating between simple linear steps and (still pretty simple) nonlinear steps.

Before training, the outputs from the output layer will be nonsense. [image: neural dots outputting scribbled musical notes] They won't match the data the same way a song played on a randomly tuned piano won't sound anything like what it's supposed to.

Training the network is like tuning that piano. [image: neural dots pointing to a musical note that gradually becomes more defined] Just like you'd tighten or loosen strings in a piano to make a song sound right, you increase or decrease the edge weights in a neural net to match its outputs to the data you're training on.

By tuning the edge weights like this during training, the network is primed to handle the same kind of inputs during testing. [image: network graph with a chat bubble saying "cat"]

There are a lot of flavors of neural net right now, and we don't really know which approaches are going to win in the long-run. [image: network graphs with legs running towards the finish line of a race] In fact, there's a lot we don't know about why neural nets work... at all...

But we're learning. And we know enough now to make informed choices about which tools to use for which problems. [image: woman in profile looking up]

We're learning that convolutional layers are good for exploiting the spatial structure of images. [image: picture of a dog being analyzed] We're learning that randomly tossing out parts of the data at each iteration can make the fit more robust. [image: picture of a cat followed by a duplicate image with an ear blacked out] We're learning that one of the simplest of the simple nonlinear activation functions, ReLu, seems to work best in deep learning. [image: nonlinear graph with max (x,0)]

If you're new to neural nets, how do you start forming insights like this of your own? [image: man thinking]

You get your hands on some data... [image: man with a box full of cards with numbers on them] ... and you start playing around. [image: hand dumps the box over]

You can tackle the first part with The Wolfram Data Repository, where lots of different datasets are already available and pre-processed for your use. You can grab a dataset and download it into your notebook with a line of code. [image: a screenshot of a sample dataset from the Wolfram Data Repository with handwritten digit dataset and pre-cleaned training & test sets]

As for playing around with it? The Wolfram Language's neural net framework makes that straightforward as well. [image: a screenshot of a pre-built neural net from the Wolfram Neural Net Repository] You can download a pre-built neural net from the Wolfram Neural Net Repository that's been trained to match the data, then you can look at its guts.

The Neural Net Repository gives you a lot of options. [image: hand reaching for one of three jars that has network graphs in them] Audio? Text? Images? No problem. You can pick a net that's a good fit for your problem right off the shelf with NetModel and tune it to match your data.

Or you can start from scratch: create your own networks using NetChain, train them, and see how performance changes as you knock layers out or add them in. [image: a screenshot of creating networks using NetChain with text highlighting how to create net, train it, and check its performance]

This kind of self-guided exploration can help you go from being familiar with the idea of a neural network to developing an intuition for them. [image: man on a computer with a lightbulb moment] And you can get up and running on all sorts of data types in minutes by tapping into the curated datasets the Wolfram Language provides.

But this is a lot more than a sandbox. The real power of the framework is how rapidly you can put its tools to work on your own datasets... [image: signal lines with network graph illustration]

... And how easily you can share your work with others. [image: woman handing a red box labeled "box contains USB with a .NB on it" over to a man]

Materials Science + Mechanical Engineering + Theoretical Computer Science to the Rescue

Scaling the process isn’t just a matter of building a more precise, more expensive printer (although that is necessary). It’s a matter of choosing the right type of sugar, understanding the physical behavior of the sugar as it is printed and of telling the printing robot how to move. The problem spans materials science, mechanical engineering and theoretical computer science; it contained more than enough material for a PhD thesis, and the cells aren’t even in there yet. The materials and mechanical engineering aspects are laid out in a recent publication in Additive Manufacturing; the planning algorithm is described in a recent manuscript still under review.

Instead of conventional sugars, this printer uses isomalt, the same low-calorie sugar substitute they use to make throat lozenges. Isomalt works better than conventional sugar, mostly because it doesn’t burn like sugar does. The printer melts the isomalt and pushes it out of a tiny nozzle under pressure. Like a pen, the nozzle is used to “write” thin isomalt filaments, but in 3D. Printing speed, temperature and pressure are critical to achieving precise filaments. Right now the diameters of the filaments can be anywhere from 50 to 500 micrometers; to give some context, a human hair is about 10 micrometers thick. However, the researchers say it’s entirely possible to go bigger or smaller.

At that point it might seem like you’re done. But when you want to print a network comprising thousands of filaments, you encounter an interesting problem. You need to choose an order in which to print them. The printing process is freeform; you can move the nozzle anywhere you want. That means if you’re not careful, you can hit your sugar construct with your nozzle and destroy it.

Avoid Collisions AND Don’t Melt

Collision avoidance is a pretty common problem in robotics, so that part isn’t too hard to deal with. However, there is an additional wrinkle that is very specific to this problem. It has to do with the fact that every time you go to an existing filament and draw a new branch, you melt the material at the joint. Imagine you’re building a bridge, but every time you weld a new beam on, all the existing welds around it melt. This makes the problem a lot harder.

Without this constraint, the problem of choosing a sequence is analogous to finding your way out of the maze on the right. There are dead ends, but you can see them. You won’t get lost in them; you’ll immediately turn around. But with this particular constraint, choosing a sequence is like finding your way out of the maze on the right. The maze is big; for long sequences, it becomes, for all practical purposes, infinite.

The best you can do in this case is make an educated guess at every fork. For example, if you had a compass that pointed towards the exit, you might take the path that most nearly coincides with the direction of the needle. In some cases, there is no feasible printing sequence; the maze has no exit. More strangely still, you can’t know, at least with our current understanding of computer science, if you should give up. You know the exit exists if, and only if, you actually find it.

In order to navigate the maze and generate a printing sequence, you need an efficient algorithm as well as a fair bit of computational power. This part has become much easier recently with high-level languages such as the Wolfram Language. Says author Greg Hurst, a developer at Wolfram Research, “This problem spans many disciplines. Computational geometry is needed for collision detection, NP-complete graph problems — like finding cliques — need to be solved, and sparse matrix solvers need to be invoked thousands of times throughout. With the Wolfram Language, I was able to hammer out fast code in a matter of months.”

Here’s a Wolfram Language command to visualize nozzle-beam collisions. The red beams must be drawn after the green one to avoid contact with the blue nozzle as it draws the green beam:

Here’s the full implementation with relevant information to monitor the progress:

What’s Next?

So the problems of turning a design into a set of printer instructions, and of having a printer that is sufficiently precise into execute them, are more or less solved. This doesn’t mean that 3D-printed organs are just around the corner. Just pouring a bunch of liver cells around a vascular mold doesn’t give you a functional liver. You need to make the cells to grow and organize the same way they do in your body.

Cells respond to the chemical and mechanical properties of their environment, and successfully recapitulating tissue requires tailoring these parameters as well. But printing vasculature is a very important step, because without it, everything dies. Now we can start tweaking the environment without worrying so much about the all-important blood vessels.

Today, we have sugar bunnies. Tomorrow, we might have miniature organs on a chip that we use to test new drugs. In several years, who knows. Maybe we’ll be able to replace a failing heart with a patient-specific replica. Maybe the processor in your computer will be replaced by a vascularized slab of neurons. We can’t yet say what the application will be, but we have the tools now to start entertaining these ideas. Stay tuned!