vladsbox

The total eclipse passed through the continental US today, with an estimated 7 million people travelling to view it. Now that it has passed, we can look at a historical indicator of where it was viewed from, by means of traffic. Spontaneous traffic jams seem to mysteriously correlate to the path of the moon.

There has been previous work on automatic colorization of images, a fairly impressive process that takes a black and white image and converts into a colour one. This uses DNN’s and their ability to guess at probable outcomes based on massive amounts of trained data. It’s pretty cool, but it can sometimes go wrong — water is blue because it usually is, so it’s a good guess. Peppers are… mostly orange?

Well, a paper and talk at Berkeley (now surfacing due to SIGGRAPH) highlights guided colorization; that is, a user supplies the DNN with some initial information on what colour an item should be, and the network fills in the rest. This provides impressive results, as expected, but more impressive is the YouTube video that shows the researchers’ software and its ability to on-the-fly change the colours based on user input.

This is an excellent demonstration of the better capabilities of DNNs that are available today; not necessarily self-serving, autonomous assistants, but very intelligent tools that can massively enhance productivity of the human operators.

Twitcher is the Killer App. From Futurama / 20th Century Fox

WordPress is an excellent tool for publishing long and short articles. The analysis pieces I write fit perfectly, have an easy editor, and can be made in a very efficient manner all through an online admin and mobile app. So that’s cool. But, sometimes, running an analysis just to recommend something is overkill. To that end, I have combined a wordpress plugin, some CSS, and a Chrome extension to make a Twitter-lite, capable of posting short snippets in the sidebar.

Motherboard recently ran an article discussing how lobbyists for telecom companies were arguing that the vulnerabilities in their networks due to the aging SS7 protocol weren’t that bad. Well, spoiler alert, they are that bad. The (overly) simplified statement is, anyone with access to the SS7 network can redirect any telephone number, practically anywhere in the world. This is the basis of number portability, i.e. why you can switch from AT&T to Verizon (or vice versa) and keep your telephone number.

The key here is “access to the SS7 network”, which is different from the telephone network itself. SS7 is a “control plane” protocol, so telephony users don’t get access, but telcos do. So it needs to be an inside job.

The internet itself suffers from a similar issue with the aging BGP standard. A decade ago, a misconfigured BGP router knocked YouTube offline, and similar events have happened in many occasions. Last year, it knocked Twitter offline. Once again a system that is not very robust, and can be manipulated by anyone connected, this system mainly works because all the actors are known, so most of the time errors are honest mistakes.

There are more of these trusting systems lying around, hidden in the background of society. They do need to be fixed, but it is more than any single company can be asked to fix, as they are global standards.

VentureBeat has a piece on the upcoming Nokia 8, HMD’s flagship entry. It lists the customary high-end specs expected; Snapdragon 835, 4gb RAM, 64GB flash, dual camera, and higher end version available for more moolah. The piece also cites an unnamed “Scandinavian Retailer”, the source of which appears to be an entry a few days ago on WinFuture.de, claiming said retailer had specified a €589 price for the new high-end headset. Otherwise, the handset seems to keep all the features of its smaller siblings, including the front face, apparent capacitive home button, and overall layout.

Visually, I can only identify three distinctions of this phone. The dual camera setup, the USB-C slot on the bottom, and a new backside that is reminiscent of the original iPhone, with its wide, smooth curve (hey, HMD, didn’t you hear Apple has a patent on rounded corners?). The rounding appears very sweeping, protruding across the side button and even shaving off some of the headphone jack. When compared to the already immaculately-looking Nokia 6, that visual reveal is relatively unimpressive, though at least HMD has earned the benefit of the doubt. When comparing the pricing though, if accurate, there’s going to be a tough sell to argue for why this flagship should be purchased.

JSpeex is a Java implementation of the Speex codec. By default, Java ships with either no encoding or the venerable G.711 encoders. However, if you want to store a bit of a larger data set, well, you’re out of luck. This is a problem for me, as I wanted to store a lot of spoken data in a little space. Luckily, the Speex codec is right for that, but it wasn’t as easy to get it into my project as I wanted.

Let’s get this clear first: jspeex is not an actively maintained project. The last code update I can find is from two years ago. So it is not too surprising I couldn’t get it off the shelf, and the documentation was a bit off. Nevertheless, it works just fine for my purposes, so I write this here in case anyone else needs it.

Adding the Dependency

jspeex is not hosted on Maven Central. So adding it as a maven repository, you have to add the jboss releases repository. For copy-paste purposes, that boils down to the following code:

<repository> <id>sonatype-snapshots</id> <name>sonatype-snapshots</name> <url>https://oss.sonatype.org/content/repositories/snapshots</url> </repository>

<dependency> <groupId>org.mobicents.external.jspeex</groupId> <artifactId>jspeex</artifactId> <version>0.9.7</version> </dependency>

Recording and Playback

jspeex integrates into the Java AudioCodec API as a provider; therefore no configuration is needed to get it running. However, due to how the audio format conversion works, it is not cleanly possible to record and playback. The AudioSystem DataLines appear to need PCM, and cannot accept Speex directly. Therefore, we need to jump through a small hoop to get recording and playback working.

// Create target Speex AudioFormat. In this case, mono sound. AudioFormat speexFormat = new AudioFormat(SpeexEncoding.SPEEX, sampleRate, -1, 1, -1, -1, false); // Convert the stream speexStream = AudioSystem.getAudioInputStream(speexFormat, micStream); // Can now write regularly AudioSystem.write(speexStream, SpeexFileFormatType.SPEEX, bos);

For recording, it’s a simple conversion needed. Create an AudioFormat with the same sample rate and record it. The inverse is needed for playback: converting the stream. This primitive player will handle regular audio files and SpeexEncoding format as well:

public static void play(InputStream is) throws Exception { // AudioSystem requires a buffered stream. BufferedInputStream bis = new BufferedInputStream(is); AudioInputStream audioStream = AudioSystem.getAudioInputStream(bis); // Speex requires a reformat into PCM. if (audioStream.getFormat().getEncoding() instanceof SpeexEncoding) { AudioFormat format = audioStream.getFormat(); AudioFormat pcmFormat = new AudioFormat(format.getSampleRate(), 16, format.getChannels(), true, false); audioStream = AudioSystem.getAudioInputStream(pcmFormat, audioStream); } // It works better if the playback is in 44.1khz for some reason. Let's do that. AudioFormat outputFormat = new AudioFormat(44100, 16, audioStream.getFormat().getChannels(), true, false); audioStream = AudioSystem.getAudioInputStream(outputFormat, audioStream); // Pure playback and cleanup past this point Info info = new DataLine.Info(SourceDataLine.class, outputFormat); SourceDataLine sourceLine = (SourceDataLine) AudioSystem.getLine(info); sourceLine.open(outputFormat); sourceLine.start(); int nBytesRead = 0; byte[] abData = new byte[4096]; while (nBytesRead != -1) { try { nBytesRead = audioStream.read(abData, 0, abData.length); } catch (IOException e) { e.printStackTrace(); } if (nBytesRead >= 0) { @SuppressWarnings("unused") int nBytesWritten = sourceLine.write(abData, 0, nBytesRead); } if (nBytesRead == 0) break; } sourceLine.drain(); sourceLine.stop(); sourceLine.close(); audioStream.close(); }

Recording while Streaming

This is an edge case that only truly applies to me perhaps. I needed to record the audio while streaming it to a remote location. Luckily for us, AudioInputStreams are very much regular input streams, so I simply needed to use the venerable PipedInput/OutputStreams to copy and save simultaneously.

int sampleRate = 16000; AudioFormat format = new AudioFormat(sampleRate, 16, 1, true, false); DataLine.Info info = new DataLine.Info(TargetDataLine.class, format); TargetDataLine line = (TargetDataLine) AudioSystem.getLine(info); line.open(format); line.start(); // Now, we want to duplicate this line data into two sources. PipedOutputStream pos = new PipedOutputStream(); PipedInputStream pis = new PipedInputStream(pos); AudioInputStream audioForSpeechRecogniser = new AudioInputStream(pis, line.getFormat(), AudioSystem.NOT_SPECIFIED); byte[] buf = new byte[4096]; int read = -1; while ((read = line.read(buf, 0, buf.length)) > -1) { if (abort.getAndSet(false)) { line.stop(); line.close(); pis.close(); } try { if (read > 0) { pos.write(buf, 0, read); bos.write(buf, 0, read); } } catch (IOException e) { break; } } pos.close(); audioForSpeechRecogniser.close();

As we can see, all I really needed to do to pipe my stream to two locations was to do a read loop that writes it to two locations. Simple.

WordPress is an awesome tool for blogging. It is extremely feature rich out of the box, is complemented by a swath of plugins for any functionality it doesn’t have, and if that still isn’t enough it’s possible to edit and tweak down to the smallest detail to make it absolutely perfect for what you have in mind. Unfortunately, one of those tweaks is speed.

After optimisation, my landing page cold-loads in 130ms, of which only 50ms is spent in PHP

I say unfortunately, as I believe speed — like security — should not be an afterthought. A lot of the items on this list are items that only need a manual configuration and they’re done, but the manual configuration is a step on the way. It is a journey well worth it, though: after running through all the hoops, the PHP generation only needs 50ms to load a page, with the rest spent in networking. This can further be improved with HTTP/2 that I will discuss later.

Also, it is worth noting that while I am writing up this piece, it is essentially a collection of tips, guides and tricks that I have found around the web. Sadly I do not have proper notes on that (it was a LOT of googling) so I cannot credit them properly, but I am putting it together as an omnibus piece as it is the full guide I wish I had when I started this journey. So, without further ado, let’s start optimising for a faster WordPress!

Step 1: PHP

By far the easiest way to a faster WordPress is by using PHP 7. Without any other optimisations, this downright halved the loading time of my test blog. My webhost, Dreamhost, defaults to PHP 5.6, which works but is not nearly as fast. This should become the default soon enough. I did not encounter issues with WordPress itself when using PHP 7, but some plugins were not happy, especially the memcached wordpress object cache.

Overall, the site is running fine and is happy, so I would absolutely recommend PHP 7 as a WordPress baseline.

Step 2: Opcache

This is the first of two types of caches that we will deal with when it comes to WordPress. This cache contains the compiled php files. This is effectively the result of PHP reading the WordPress source, and then storing the result in memory. If we did not use an Opcache, every single page loaded would have to read, parse and compile every single php file needed for the page, a very CPU and disk-intensive process.

As this is such a basic item on the list, there have been many, many implementations over the years. eAccelerator, APC, Zend cache, XCache have all come, been used, and been overshadowed for whatever reason. For me, opcache was pre-built by Dreamhost, but not included by default. This was easily fixed by including it in my PHP config (phprc for dreamhost)

zend_extension=/usr/local/php70/lib/php/extensions/no-debug-non-zts-20151012/opcache.so

I did not need any further config on my moderately-little used website. This provided an additional, significant, speed boost over the Dreamhost-default XCache. We’re not talking much, but every bit counts. Note that opcache will, per default, check the filesystem for changes after two seconds. That is, after two seconds, before it executes a cached script, it will check for any changes of it on the filesystem. This essentially means the full speed boost (not touching disk) is only for any requests within those two seconds (configurable, of course) but for most websites that is more than plenty (i.e. it takes the edge off the highest of volume loads).

Step 3: Object Cache

This is where it gets tricky. At this level of caching, the usual recommendation of quick searching is something like WP Super Cache. This is an extremely popular and highly compatible plugin that works on a fairly simple premise: once a page has been rendered, save it as a static file, and then serve the static file to the next visitor. It is very fast when it works, cannot be beat if direct file access is used, and broadly compatible. It is not surprising that it’s so widespread, and if your web server cannot run persistent processes, this is what you should use. Feel free to go to the next step.

If you can run processes, there are places you can go in addition to WP Super Cache. These speed up page generation for logged in users as well, such as yourself, dear admin. The go-to here is memcached, simply put a memory cache that runs as a separate process and can be read and written from many processes. WordPress will store stuff here — whole webpages, intermediate results, etc — and retrieve it from blazingly fast local RAM rather than disk or SQL. There’s just one teensy tiny problem: the most common memcached solutions for WordPress won’t work in PHP 7, nor will they work with the php memcached extension; they require the older, and not updated, memcache extension (note the lack of d).

Unfortunately, the only plugin I found on this area was WP-FFPC, which admittedly works very well, but comes with a warning label that development is “on hold”. Notwithstanding that, it does an excellent job of caching pages and objects in RAM. Had I hopped over permanently to nginx, it allows nginx to directly interface with memcached as well, which should theoretically knock the socks off WP Super Cache.

Step 4: Image Resizing

Now, this might sound off, as Jetpack already provides an image resizer for you. However, as any free CDN, I’ve found that sometimes it’ll slow down for reasons I cannot explain. If I’m going to do it, I should control it myself, is what I’m saying. The choice here ended up being the oddly named EWWW Image Optimizer, which not only automatically cleared jpeg and pngs for me, it also created webp copies of all images. For Chrome users, this is a great improvement.

The major speedup here was on mobile (which stood for more than 50% of my traffic, so it was well worth doing!). Especially on 3g, this more than halved the time it took to load a page on my site, with no work needed from me. On Android, this went even further.

Step 5: CSS Minification

Jetpack, WP SuperCache and even WP-FFPC happily cache items for you, but they don’t really take the time to actually minify them for you. Minification is a very useful step: it takes HTML, CSS and JavaScript files and removes stuff the browser doesn’t need: comments, inline documentation, spaces, tabs, newlines and other non-syntactical characters. Not only does this mean less data to transfer, but it also means the web page needs less CPU to render on the target browser. Minified source files will be parsed quicker, as the receiving browser no longer needs to process useless text.

That’s great! For this, I used Fast Velicoty Minify, a plugin that does exactly the minification. The results are static CSS and JS files that are served up by the web server, skipping a lot of php work.

Step 6: Compression & Caching

Wait, caching again? Yes, but on the client side. WordPress is able to handle gzipping and caching for anything we load dynamically, but we’ve been naughty boys and pushed a lot of content out of PHP. Images, CSS, JS and, if used, WP Super Cache files are all served up by the plain web server. It’s time to crack open the .htaccess file and add some stuff.

If you’ve done what I did, the htaccess file will already contain stuff from WordPress (permanent URIs) and EWWWIO. We now add the following block:

SetOutputFilter DEFLATE SetEnvIfNoCase Request_URI "\.(?:gif|jpe?g|png|webp|pdf)$" no-gzip <FilesMatch "\.(ico|pdf|jpg|jpeg|png|gif|js|css)$"> Header set Cache-Control "max-age=604800, public" </FilesMatch>

Essentially, we’re saying “DEFLATE” everything (a very, very simple compression). But don’t do it for images (there’s really no point to compress images). Oh, and if the file is an image, pdf, css or js file, add a cache header. This will allow the browser to keep a copy of that image/script so that it doesn’t have to download it all the time. On my blog, even minified CSS adds up to 377k data. Deflate takes that down to 104kb over the wire, but it’s still 100kb more than nothing, which is what the cache control allows us to keep it at.

Step 7: The part I didn’t manage

The sum of the above took my page load time down from 5 seconds to 1 second (cold) and less than 500ms (hot). That’s a lot. There is one, final, oomph that could have been done, however: nginx.

So far, everything I’ve discussed uses apache as the web server. It is a very capable web server, and highly compatible. Nginx, however, is faster. It’s also better at supporting HTTP/2, the newer, speedier version of HTTP. I managed to get the site going with nginx, but it was not stable. The biggest contention was between EWWW and nginx, as well as a few bugs in Dreamhost’s nginx management. I may yet get this done.

I measured a slight speed improvement with the plain nginx change, but not enough for me to continue down that path. For now.

The unfortunately beleaguered headset maker Facebook Oculus made the news at Bloomberg for a rumoured wireless headset that, according to the outlet, would define a “new class” of VR headsets that is neither a phone nor just a PC display. Unfortunately, while it may indeed serve to advance the VR space, it’s neither first nor a new class.

First, let’s consider whether it is new. It is not. Google has been preparing its standalone VR headset, “Daydream with WorldSense”, partnering with HTC to bring it “later this year“. It is likely that the Oculus response is predicated in part on this; the big question will be if Oculus will bring its ecosystem to Android or continue to run its own race. Somehow, I suspect they’re tempted to still try and build their own ecosystem.

Second, let’s consider whether this is a new “class” or not. Both the HTC “wireless” headset and the Oculus one likely have more in common with the Gear VR and Google Cardboard than anything else. They are basically phones that are permanently wedged in a VR headset with no other job to do. It’s not as if there isn’t precedent; the Rift DK2 was quite literally the screen from a Galaxy Note 3 — logo, camera, home button cutouts — wedged permanently in the DK2 headset.

To recap: Not a new class, not first, and probably with their own ecosystem. It remains to be seen if Facebook/Oculus can take the VR world by storm, but they are not making great indications of it so far.

Thanks to the Nokia 6, I am now in possession of a Google Assistant phone. This gave me the perfect foil against the upcoming Cortana and Siri speakers: if you have an assistant with a screen, how different are the answers to one without, even though it’s the “same” assistant? I expected that the audio part would remain relatively unchanged, that I’d get more displayed on the screen, and that search results would be a greater crutch than the Google Home speaker.

The first surprise was that my test harness didn’t work. Sure, the phone properly recognised the queries, but it played the response through the Google Home speaker. It seems that any notification that’s voice-only is directed there, whereas voice+visual notifications stay on the phone. It’s a bit confusing. The trigger for this appears to be the phone’s bluetooth connection. A question like “OK Google, what’s up?” will generally play on the Google Home speaker if Bluetooth is on. Rapidly turning Bluetooth off and asking the question again will lead to the answer being played on both devices. Curiously, the two simultaneously answers received were different on the phone and on the speaker. Odd.

Another oddity that formed was that, in general, for answers that were identical between the phone assistant and Home assistant, it would play in one tone (fairly loud), but for phone-unique answers, the Assistant voice would be slightly quieter and ever so different in tone.

The assistant also makes the expected accommodations for the existence of a screen; weather with icons, locations with maps and call buttons, and, of course, instead of not giving an answer it will default to Google search, which oftentimes has a correct answer in the first few results. I haven’t made graphs of this yet, but I recorded a lot more partial scores on the phone assistant.

One accommodation that, indeed, makes sense on the phone but wasn’t expected was that the phone assistant isn’t equally gung-ho about reading the entire contents of pages. For example, on the Stockholm travel question, it neglected to give the temperature information, but certainly displayed it.

From these two assistants, we also begin to see a bit of the internals of Assistant and a very impressive contextual recognition. In my original test run of the assistant, I asked about George Clooney, his films, his newest films, and asking it to describe “the first one”. The Home speaker gives up at that point, but the phone assistant does a Google search for Suburbicon. What we learn from this is that Google’s assistant is backed by speech and intent recognition sufficiently good to recognise the question, but not good enough to answer it.

The phone was also a lot more willing to read out answers from the web, especially when these answers really didn’t help any. For example:

Q: Are there any road closures in Yosemite?

A: From National Park Service: Viewpoints, Frazil Ice, Horsetail Fall, Glacier Point, Hetch Hetchy, Mariposa Grove and Tioga Road and Tuolumne Meadows. [Google is reading a list of winter road closures, without really saying that they are winter closures. The initial response that is spoken makes it sound like some sort of cataclysm has befallen Yosemite]

Q: How much can I put in my checked luggage?

A: Typically, each passenger can carry a 1 Quart clear plastic bag with original containers of less than 3.4oz. If you are bringing larger bottles of sunscreen/cologne you will have to put the larger bottles of liquids in your checked luggage that goes under the plane. [Interesting factoid, but not what I was looking for. To add insult to injury, this official-looking answer is actually Google reading out a random poster in a forum thread…]

Q: Can you read newspapers?

A: According to Amazon.com, you can subscribe to magazines or newspapers from the Kindle store on Fire and Kindle devices, supported Kindle reading apps, and the Amazon website. [Well, at least the EU can’t accuse Google of using its nascent voice assistant monopoly to only market its own products]

Q: What devices can you control?

A: Here’s a summary from reviewed.com Smart Home: Belkin WeMo insight switch, Belkin WeMo Light switch, Samsung SmartThings outlet (via SmartThings hub), iHome Smart Plug (via Wink Hub), and more [This looks kind of acceptable… until you actually follow reach the bottom of the list, and read the title of the page that Google assistant didn’t read. The title reads “Everything that works with Amazon Echo and Alexa”. Relax, Google, I was joking about the EU thing!]

Overall, it seems that both Google assistant endpoints use the same pool of answers, but the phone is a lot more lenient in accepting answers. This, in turn, leads to a lot funnier errors, a lot more partial answers, and even some confused ones (e.g. the command “Give results in metric units from now on” recommended that I download a third party app).

The accommodations for the phone were overall pretty nice, and the Google Home speaker integration was simultaneously surprising and confusing.

Heading image by Craig Sunter / Flickr

So we are veering into pet peeve territory, but the Next Web has a breathless article headlined This app uses artificial intelligence to turn design mockups into source code. Besides being misleading, the author confounds AI with the current implementation, i.e. deep neural networks, and their functions with an algorithm that is being improved. This is fundamentally flawed, for the same reasons as the AI assistants’ persistent inability to understand vague sentences: neural networks of today have no understanding of what they are doing.

Imagine, if you will, that it is 1995, and the world is just now being introduced to the magical machines called computers. A user is sitting at one of these devices, and has learned that they must turn it on such that the monitor has a shining light and the computer box has a shining light. This is going on still, but the monitor is black. They try pressing the keyboard. They try moving the mouse. They try pressing the buttons again, but it still will not cooperate. As a user, this is what they have learned, and this is what they expect, and when it doesn’t cooperate they do not know whether the VGA cable is not plugged in, the graphics card is not working, or someone has edited the X11R6 conf file and messed up.

The computer is a black box. The users have associated certain inputs with certain outputs and expect it to work. It is similar to training a parrot to press a lever for a cookie. They may not know why a cookie comes out when the lever is pressed, but they will do it because they want the cookie. A neural network is the same thing: it associates certain inputs with certain outputs, and, when presented with an input (or one that is similar enough), it produces an output that is similar to what it associated with that input. And so, the UI generator described above will produce the following outputs:

Note the subtle differences. “Ground Truth” refers to the actual UI, and “Generated” refers to the output of the software based on an image of the ground truth. There are a number of cues that show what’s going on here. Ignoring the text, which is a fixable error, note that the colour scheme of the output is wrong. While the software is correctly identifying text fields, headings, titles and boxes, the contents of those boxes aren’t carried over. To understand why, we have to, for a moment, think like a neural network. The rest of this post is written against the backdrop of the previous entry, Deep Learning: A superficial look. I will use definitions from there going forward.

Let’s zoom in on the issue with colours. Specifically, let’s look at GUI 6, a relatively complex set of boxes and buttons, featured on the demo video. The image generated looks reasonably OK, but what is it that the neural network is actually doing? Well, the output of the neural network is, according to the paper, a DSL: a domain-specific language that describes a UI in text form. The visual representation is generated from there. This distinction is important, as it shows the “hidden” layer of transposition.

The DSL output is what the neural network is trained to generate. When shown certain inputs, it has been trained to produce certain outputs, but it does not necessarily know what those outputs are. In GUI 6 this means, for example, that the DNN has been trained on recognising boxes; two boxes, four boxes, one box. It’s also trained to recognise buttons in those boxes, but this is a probability game. One reason the button in generated GUI 6 on the top left is green could simply be that the DNN has associated “top left box with button” to be described in text as “top left box with green button”. Quite likely, regardless of what colour button there was in the source image, the generated GUI would produce a green button.

In fact, going further, depending on how the DNN is trained, it is possible that it would never generate certain boxes without buttons (I am not saying this is the case here, I am illustrating). Much like the, albeit slightly offensive, joke in the Big Bang Theory about a woman learning English from TV commercials — and repeating a Geico commercial when all she wants to say is 15 minutes — the DNN does not know the meaning of its output. It only knows it is the best approximation. This is also why the text content of the boxes differs between input and output: the DNN simply posts as text content whatever was the most trained output.

In terms of input and output, these are not issues that cannot be fixed. The DNN shown in that training video had 1500 items to train on (with permutations on each that brought them up to just over 300,000 entries. Note that this is artificial entropy; they made 1,500 master items and then added or removed parts, changed colours, etc, to bring up the complexity. Compare with the ImageNet challenge that consisted of almost 500,000 images, each using different subjects, angles, colours, etc. Furthermore, the output of that challenge was to create single word outputs to classify an image, e.g. “cat”, “dog”. This should illustrate the difficulty of training a neural network to provide “correct” input and output.

However, it doesn’t scratch the deeper surface, which is that it’s not understanding. For a human, or even a computer program, it is possible to create an instruction that says “use the same color as the button”. It’s a copy/paste job, requiring very little thinking. For a neural network, however, it’s an almost herculean task. For example, if I trained a DNN on red and blue buttons, the moment I showed it a green button it’d give either a red one or a blue one, or maybe a purple one, but it wouldn’t even know what to do with the colour green, let alone copy it over. (This, by the way, is why current best-practices for DNNs is to have “comprehensive” training data)

So, back to the headline, while that 77% number is impressive, it comes with several caveats. First, it is from their own testing data, and I’m sure that if I put in any other sort of UI mockup in there it’d choke immediately. Second, the current methodology is going to make it very difficult to reach more than 90-95% accuracy, as it more or less requires them to train the DNN on practically every type of UI one would want to “create” with it.

Heading image by Turinboy / Flickr

IBM has wholly adopted the Watson branding ever since its debut in Jeopardy. The Watson umbrella now holds 19 APIs and counting at IBM. The terms are also surprisingly good, such that I used them for my assistant testing. They do have some quirks though.

The funniest one, to me, was a transcription that came along the Mary Poppins question, asking about its cast. Quoth Watson, verbatim:

the cast of Mary Poppins includes Julie Andrews Dick Van **** and 17 others .

Dick Van What? It had been smart enough to not censor the proper name “Dick”, so it either parsed a lowercase one at the end or something else popped in. Not to worry, Mr Fawlty, I learn. The profanity filter in US English is on by default. Turning it off gives us the unadulterated (or is that adult-rated?) response:

the cast of Mary Poppins includes Julie Andrews Dick Van Dyke and 17 others

So it got the name right. Which means the speech recogniser not only was trained properly but produced a proper name output. But the word filter coming after that apparently took issue with Dyke, presumably in reference to the slang usage of it, but having no knowledge or interest in that it’s being used as a proper noun.

Davey Alba at Wired muses that we are headed towards a “splinternet”: an internet that is different depending on the country of origin. There are two unfortunate realities to this. First, this is already true, in his own acknowledgement. Second, it is inevitable. What’s worrying isn’t that there are different internets; it’s that there’s a max-censored one.

Speech has always been regulated. Even in western democracies, we place limits on speech, oftentimes for good reasons (and sometimes for not-so-good ones). The Internet came about as a new technology and shattered those, but it was never really going to last. As each country places restrictions on their online presences, we should expect — and do see — different acceptance tolerances. The problem isn’t truly when it is within one country’s borders: it is when other countries are censored as well. The scarier one, to quote the article, is thus:

The trend of courts applying country-specific social media laws worldwide could radically change what is allowed to be on the internet, setting a troubling precedent. What happens to the global internet when countries with different cultures have sharply diverging definitions of what is acceptable online speech? What happens when one country’s idea of acceptable speech clashes with another’s idea of hate speech? Experts worry the biggest risk is that the whole internet will be forced to comport with the strictest legal limitations.

That is a more troubling aspect. If nations can censor the Internet, then there will be a winner-like scenario, where the technology is no longer a tool for communication, but a tool for domination. That should worry us.

A recent paper from Microsoft Research, Azure and John Hopkins University discusses the issues of troubles of grey failures. This is a type of cloud failure under some discussion, that I treat as follows:

A grey failure is a problem that is affecting one or more applications deployed on a platform, that is detectable by those applications, but that has not been detected by the platform itself.

This is not a new problem. From an application development point of view, development can simply opt to consider infrastructure volatile and simply jump ship – to a different data center, or different cloud provider even – and hope the new host doesn’t have the problem. This is good practice. An application should never assume the platform is stable; doing so, the application will inherit every problem provided by the platform. The issue, as it were, is for the infrastructure provider to discover the fault.

The Microsoft paper brings up several points showing the incidence of grey failures. The comparison they choose to use is when the application (which, for them, is the OS inside of the VM) encounters high load versus when the infrastructure detects a problem. One example I found particularly interesting was the case of a malfunctioning network card.

The image above, from the paper, depicts a VM during a “triggering event”. The VM is migrated from its original host to a new host, and this causes an issue. We will start from the bottom right graph, which is the cloud monitor’s view of the VM. Green dots are “healthy” and red dots are, according to the authors, restart actions initiated by the VM user. According to the paper, “the underlying issue is not detected until well after the time period depicted”.

In comparison, looking at the VM level metrics, the authors state that when the VM is migrated, the network usage takes a slight dip. It never manages to recover to its “proper” utilisation, and the user reboots the VM. After the reboot, the VM doesn’t get any traffic. The underlying cause isn’t described, but this is noted as a grey failure.

Here is where the challenge appears. What is to say that the issue of the VM is from the platform and not a failure in the application, and the application itself has chosen to take this node out of rotation for debugging. This is a common use case – blacklisting a VM to preserve the state so a human operator can see it.

The answer (and, probably, most used means of detecting grey failures) is anomaly detection. Tracking usual patterns of tenants and software running underneath, and reacting to when they start behaving in unexpected ways. The key element in making it successful is efficiency. If the error detection needs more compute than the processes it is monitoring, it is not profitable, so we will continue to see random errors occur in our massively complex systems.

Obligatory tortured container metaphor. Image by Håkan Dahlström on Flickr

I recently came across a very confident entry by Steven J. Vaughan-Nichols about the rise of Kubernetes, and how it is spearheading a push by cloud vendors to break into the market by way of containers. Interesting; I knew Kubernetes was highly used — indeed, I use it myself — but to have the astonishingly high 71% that use it? There must be more to that.

Step one, on any such claim, is to go for the source. The source, in this case, is a press release touting a study commissioned by CoreOS — purely coincidentally a vendor of docker runtime base OS, although I’m sure that certainly does not affect the study’s angle — that is hidden behind an annoying email-wall. While the press release handily touts the 71% Kubernetes usage number, this is, in fact, not represented in the study. The study states that “almost 75%” of respondents use Kubernetes, a number discrepancy that, while not big, shows that there’s more baking in this press release.

Let’s compare the press release, to start with. “Our mission at CoreOS is to break the cycle of vendor lock-in by making the best open source infrastructure management software available to our customers,” the press release quotes the CEO of CoreOS. Like a badly written comedy show, here’s a graph from their own study on why the surveyed companies are looking into container runtimes.

So out of 200 surveyed companies, 6-7 considered the biggest benefit avoiding vendor lock-in. I’ll avoid the issue of which vendor is being avoided here; there are multiple items confounding this question: whether it’s the technology vendor (e.g. Docker), platform vendor (e.g. Kubernetes, PaaS) or infrastructure vendor (Amazon, VMWare, Dell). I’m lumping together the companies here a bit, but to me the biggest statement of the above graph is that companies are looking for some form of boost, presumably based on where their biggest expense is, by using container runtimes.

Before I continue, I’d like to address a small point from the original post. The post references Amazon as ideally wanting to use Lambda instead of containers. Since that’s a bit of an apples-to-oranges comparison, I will assume EC2 Container Service instead. Amazon’s quite-sprawling cloud offering has many tentacles that aren’t always easy to follow, but to my mind ECS is a better match.

Amazon’s ECS is, effectively, a hosted docker runtime. It’s a like-for-like match to Google Container Engine (intuitively named GKE), except the latter runs a Kubernetes cluster on-demand, whereas ECS does proprietary AWS magic. That, together with the survey respondents, is where the crux lies. The survey was conducted across fairly large companies; the type that are likely to use many solutions because they have many branches. From the study:

We surveyed 201 enterprise IT decision-makers in April and May 2017. This was not a survey of developers; rather, we received responses from those in C-level and director-level positions, including CISO, CTO, CIO, director of IT, IT Ops and DevOps, and VPs and managers of IT. […] A significant slice of the survey respondents represented large enterprises. The largest groups of respondents reported annual revenue of $500m-1bn; $1b or more; and $100-500m. Some respondents did cite $25-100m or less in annual revenue, but together they made up less than a quarter of our respondents. It was a similar story in terms of number of employees; most said their organizations have 1,000 employees or more, and a large portion cited 2,500-5,000 employees. [Emphasis mine]

The study additionally went on to detail that 36% of their respondents would “move workload” depending on where it was cheaper, and that a majority either hosted their container runtimes on their own hardware or a “combination of hosted containers as a service (CaaS) and on-premises CaaS”. Teasing out this we should be able to understand the appeal of Kubernetes to this crowd: the ability to run, privately and for “free”, the same runtime as is available in Google Cloud, Azure and, yes, even in AWS (with some hoops) means they can count the cost savings and the agile bonuses and all that other devops goodness we hear so much about without having to make a decision. Great. Of course, there is a small caveat we also have to deal with before we proceed.

We see the aforementioned bias reflected here. As company size increases, there appears to be a trend towards Kubernetes. Of course, this is just my hypothesis and the sample size isn’t very large, but I find it likely the thought process of a large company would be “Google’s doing it, and supporting it, and we can also take it in-house.” (It also bears noting that the study, at least if they asked the questions in the order they did, was incredibly slanted. The first question reads “Which of the following best describes your hybrid or multi-cloud strategy?”)

So that’s it, really. While there is an undercurrent of “not-Amazon” going on, the rise of Kubernetes has less to do with trying to compete with Amazon’s dominance of VMs, and more to do with technology evolving beyond VMs and towards containers. With that, the companies needed a way to manage them, and, certainly they are using a mixture of technologies as per figure 6. The 71/75% headline figure was how many are using Kubernetes, but that doesn’t necessarily tell us whether they are moving towards or from using it (and it may befit the study to not do so).

In conclusion… Kubernetes is nice, and big companies like it because they can — or think they can — easily migrate loads onto public and private clouds as they want to. This approach (or bias in the survey) will automatically slant usage away from a solution that is cloud-exclusive (e.g. ECS) and towards one that can be run in-house. It is important to note, however, that the study does not mention how many companies are not hybrid, and even defines hybrid as having in-house testing environments while running production fully in a public cloud.

Heading image by joiseyshowaa / Flickr

Deep Learning is the latest one of our machine learning intelligence analytics methods — let’s say methods — that is sweeping all sorts of development. It’s supposed to solve the toughest of problems, and be our gateway to AI. All the tech companies (and a lot that aren’t) will eagerly tell that they use or are looking at Deep Learning applied to rocket science, coffee making, and everything in between. So, what is it?

The Short Version

The oversimplified version is, more or less, “Deep Learning is using large, complex Artificial Neural Networks, trained on massive amounts of data, to perform operations beyond classification”. This doesn’t sound too complicated, and, indeed, it is not. The main advantage we have today is, simply put, we have the computational power to, well, power these large (or by historical standards, enormous) neural networks.

The Long Version – Artificial Neural Networks

Diagram of an extremely simple ANN, from Wikimedia Commons

To see where we came from, and where we are today, let’s discuss Artificial Neural Networks, or ANNs. These are, effectively, a facsimile of our understanding of how brains work; a set of nodes, joined by links, that are able to excite each other in various ways. The neural network consists of three types of layers. An input layer will accept the data, with each input neuron accepting generally a real number between 0 and 1. Zero, but usually one or more hidden layers exist for the purposes of transformation, which then feed the data into a limited number of output layers. Each node in the output layer, once more, produces a real number between 0 and 1.

Before the ANN can be used, it must be trained. Training consists of using pairs of predetermined input and output data. This data is defined as correct, and the training data is, in some order (often randomised) fed to the neural network. The network will produce some output data based on the input, leading to an error calculation. The error calculation will then be used to try and correct the weights assigned to the connections between each node in the neural network; strengthening connections that help produce the correct answer, and weakening ones that don’t.

Over a period of time, the neural network should converge into one that has figured out the commonalities between the input data, and used that to properly match the output data. This is where large data sets come in handy: the more data the network is trained on, the higher the chance it learns to properly identify which feature(s) of the input data are important.

Feeding Artificial Neural Networks

Because the ANN’s input layer can only take one number for each neuron, one can not, for example, simply take an arbitrary image and place it into the ANN. This is often called preprocessing or encoding, and consists of normalising the input data such that each data point falls between the chosen range (e.g. [0,1]), and that the data is input in a uniform manner. For an image, this could mean converting it to black and white, and making sure it is a square. For audio, this could mean converting it to mono sound, and increasing the volume if it is too low.

This preprocessing step is necessary to make the data uniform. A neural network can only reasonably work if the data it sees is similar to the data it has been trained with. Training on, for example, black and white pictures allows the neural network to recognise a wide variety of colours simply by not being able to tell the difference between colours, but it would easily be able to distinguish luminance. When wanting to identify, for example, a ball, it used to seldom be relevant what colour the ball was. (With deep learning, this is changing, but we’ll touch on that later)

Training Artificial Neural Networks

We touched earlier that training a neural network is a repetitive process. But it is also a complex process. Properly training a neural network involves selecting a good and wide data set, configuring a number of algorithms on how the network should be trained, the error, loss functions, and how the synapses should be updated during the training phase, among others.

Furthermore, during the training, the ANN should be supervised. The best results for an ANN are usually the ones that use large parts of the neural network (i.e. teaches it to generalise). Allowing the ANN to diverge leads to the creation of mini neural networks within the larger one, with each small ANN learning to do one specific thing, but with fewer neurons, working well while training but giving worse real-world performance (“overfitting”).

nVidia’s sales benchmark for their Deep Learning server, MSRP $150,000

Even without these items, training a neural network is a time-intensive process. The training time increases with both the neural network size, and the size of the training data. It is, therefore, effectively without an upper bound; the upper bound being what the designer/trainer decides is a reasonable size ANN and data set. This is a process that can start at hours, but easily run into the “days” category, if not more, with larger amounts of data.

This has led to things such as Google’s Inception image recognition neural network, an open, pre-trained neural network for recognising images. Not only is it pre-trained, but they have additionally made it possible to retrain only parts of the network (specifically, only the final layer) so that it can recognise other image classes than the 1,000 they have pre-trained. They note that retraining only the last layer of the network can be done in “as little as 30 minutes”, though they neglect to say how large a training set is used for that.

One of the more illustrating sentences in Google’s discussion about retraining Inception harkens back to the difficulty of properly training ANNs (emphasis mine):

A final test accuracy evaluation is run on a set of images kept separate from the training and validation pictures. This test evaluation is the best estimate of how the trained model will perform on the classification task. You should see an accuracy value of between 90% and 95%, though the exact value will vary from run to run since there’s randomness in the training process.

So, in a nutshell, training ANNs is hard, choosing the right data is hard, setting up the right parameters is hard, and, even if one managed to get all those things right, training can take a long time and still not come out good enough.

So why are we talking ANNs? Besides, aren’t they old?

In a nutshell, yes. ANNs are a practically ancient concept in computer science, and have always been viewed as the gateway to AI, primarily on the assumption that if we can gather enough computational power to simulate a human brain, we should be able to run the same “software” as a human brain, but in a computer.

The major change that has happened recently is just that: the hardware has become available to run vast, large ANNs that would be previously unheard of. While the fact that ANNs can easily scale with CPU and memory has never been a mystery, twenty years ago, the default assumption would be no more than two hidden layers to an ANN. Today, two hidden layers would be an introduction to ANN. This led to a decline in ANNs in the 2000s, instead being supplanted by hand-fitted algorithms. These algorithms could run faster, and do things like run on a mobile phone, qualities that were valued higher than the flexibility of ANNs.

Enter, Deep Neural Networks, or DNNs. There is (as yet) no semblance to convergence on how deep a network needs to be to classify itself as DNN, but we are definitely above two hidden layers; hundreds of layers is not uncommon for a DNN. These slides from nVidia and MIT form a good overview of the area, and the richness of neural networks and techniques that have been developed for them.

I get it, DNNs are awesome. I’ll use them everywhere.

Because there’s no free lunch, DNNs are good, but they are still good at specific tasks. For example, an LSTM (Long Short-Term Memory) is a type of network that is favoured for speech recognition, as it is able to “remember” things that happened recently. This helps it recognise a sentence, as it can reference the earlier sounds. This helps to resolve ambiguities in speech. Meanwhile, for image recognition, the current leaders in the field are CNNs (Convolutional Neural Network), iterating over the image one part at a time. For performing logic RNNs (Recurrent Neural Networks) may be preferred.

This can lead to fairly complicated setups needed in order to perform more complex operations. For a demo at Google I/O 2017, the speaker showed a system where a video could be fed into the system, and the system could be asked questions about the video. For example, the system could be asked “what is the man doing”, and answer “packing”, based on the observation that a man in the video was putting a box in a car.

Seemingly a simple use case, it actually boils down to a fairly large number of distinct neural networks pictured to the right. We have a CNN to recognise the video frames, an LSTM to put them together, another LSTM to recognise what the user is asking, a weighted function to combine the output of the video with the output of the user question, and, finally, a classifier (that appears to be an RNN) to produce one of seven possible output keywords.

This is complicated.

So it’s hard. Is there any Deep Learning shortcut?

This is where things get interesting. Because of the rapid increase in available CPU, GPU and TPU power, it is now possible to, quite simply, train multiple DNNs in parallel and see which one comes up with the better result. This cross-breeding between evolutionary computing and neural networks (or, quite possibly, infinite number of monkeys) allows the automation of DNN training parameters, quite possibly making it easy for new ones to be created.

Coupled with releases such as Tensor2Tensor it is easier than ever to test out multiple neural networks, neural network combinations, and training a sequence of connected neural networks (and, at the same time, running up the cloud bill) than ever before.

This, if anything, is the current promise behind Deep Learning.

It’s an upside down map, you think. What’s so special about that? From imgur.

The drawing of maps to promote a political view or a social view is not new. And it is also not the topic of interest here. I include the upside down map above as it shows a different, and quite unusual feature. Look at Greenland. There is something odd about it. Besides the projection, it looks misshapen, somehow, and it looks like a very odd mountain. It should be the tallest mountain on Earth if this picture is correct.

Map by Fridolin Becker, from reliefshading.com

A subtly different effect happens on this map. It’s, in fact, of the Swiss alps. It is tempting to think that the white snow is on top — it is not, it is in valleys. Both this map, and the upside down world map, display height in an unusual way; not because of shadows and lighting, but because of where the lighting shines from.

Cartographic convention has, for more than a century, been to draw lighting on a map from the northwest. From a realism point of view, this is absurd: the sun will never shine from that direction, and thus the shadows on the map will be off. The shady north side of a slope — the side most likely to retain snow and moisture — will be sunny on the map. Seen through that light, the southern-lit map above from Becker is much more realistic. But it doesn’t feel that way.

While the cartographers of the 19th century only had trial and error to guide their way, in modern day we have had, well, a lot more time for trial and error. Perhaps the most famous artefact of this, perhaps inadvertent trial and error, is the crater optical illusion, often seen in pictures of outer space bodies; moons, planets, etc.

Two large craters on the left, Spitteler and Holberg, with rocks jutting out from the middle, from NASA JPL.

The image above is fascinating in large parts because it is almost the opposite of the usual topography lighting we are used to from maps. Instead of a northwestern light source, we have an eastern one (similar, but not quite the same angle as the upside-down earth map). However, unlike an image of zoomed-out Earth, the foreign landscape gives us no visual cues on what is a valley and what is a mountain; there are no rivers or vegetation. The black and white photo also eschews the usual colour differentiation for high and low lands.  So while the shading forms a similar effect to Becker’s map, the latter has an unassailable benefit of having rivers drawn in. With those, we can calibrate our map expectation, and see mountains and valleys for what they are.

Fundamentally though, after much trial and error, convention of map making has settled on a simple and easy to follow rule: the light will shine from the northwest. This gives us the familiar map of mountains and valleys we all know, and leads to the following awesome effect. Below, I have rotated the Spitteler and Holberg image. One of them has the light from “above” and the other is lit from “below”. The images are otherwise identical.

Notice that the left image by default looks like craters, while the right image looks by default like mountains. At least for me, it takes conscious effort to see the opposite patterns now that the lighting appears to be coming from the north. The reason here is, simply, psychology. It’s the thing that can make a single shade of gray look like two completely and undoubtedly different shades of gray: the Cornsweet Illusion.

And that, in a nutshell, is why we use lighting from the north: it is psychologically easier to recognise light from “above”, even though it would never shine that way. Interestingly enough, despite my searching for a southern hemisphere map that would show South from above, I only found either transposed maps (i.e. lit from the northwest, so bottom right of the map) or the user-made map I showed above with the sun from the southwest, i.e. top right of the map.

Now that we’ve cleared the table on map lighting and its peculiarities, let’s discuss Shadowlands. The series of topography-data-turned-art shows a striking, black and white view of Europe. It is familiar, but ever so slightly off — and I’m not just talking about the black and white. Indeed, the shadows are deep, and the effect is striking, but there is one more subtle aspect contributing to the unique effect…

Iceland: We can just about make it out, but it looks almost foreign. Image by Robbi Bishop-Taylor / Shadow lands.

By its placement in this entry, we should be facing a foregone conclusion: the lighting angle is off. Ever so slightly, instead of coming from north-northwest, it’s coming from west-northwest; the direction of the sunset. Coupled with the dramatic shading that puts emphasis on the changed angle, and we get this impressive, almost-foreign landscape.

There are, of course, entire books written on the subject. But for our word-of-the-day-like purposes, maps are lit from the north/northwest. They’re done so because we expect the light to come from “above”, and even though we logically understand a map to be a top-down view, we still think of “up” on the map as “pointing to the sun”. And anything that points to the sun has to be brighter on the top side than on the bottom side.

Photo by Sunny Ripert / Flickr

The Internet has been a great equalising force as a platform for the spread of opinion, and it can take many shapes or forms, growing unexpectedly or quietly ignored. For many years, this meant a spread of ideas, opinions, software, music, movies, books, drugs and many other wares that spanned the full spectrum of both useful and legal.

As of the rise of social media, however, we have seen the spread of insular groups, like the Obama is a Muslim train of thought, the Flat Earth Society, and global warming is a hoax, to name a few. These have in recent years poured beyond the previously-thought-harmless online trolling scene (I’m looking at you, 4chan), and into the massacre, defamation, and modern day ethnic cleansing resurgences. These are things we thought we, as a society, as humans, had finally agreed only led to needless violence and death around the last time we did the whole world war thing.

The cure, ostensibly, was information. Education was supposed to reduce racism and xenophobia. Globalisation was supposed to provide more benefits than the conflicts it may cause. And free speech was supposed to unite all the people. No longer could speech be used by the oppressive government to control the people. We would be freer, happier and better off.

It is, of course, against that backdrop that we reach today. Recently, AP noted that Facebook is deleting 66,000 posts a month, due to hate speech. Germany just passed the toughest laws to date, fining operators of online websites that promote hate speech. It has been effectively proven that this is a good way to influence foreign elections.

And it is, at this juncture, that we should start to wonder: where should free speech end? How much should the platforms be held accountable? And how does one deal with misinformation?

There is, unfortunately, no clear answer. A century ago, globalisation was in full swing, and conflict between the developed nations was practically unthinkable. I will not delve into the chances of war, but the assault on speech is roughly at the same situation. We are reaching a point where, indeed, online speech is taking troubling turns: inciting violence, promoting xenophobia, people openly stating that racial groups should be killed.

Furthermore, truth is subjective. I find the above thought appalling because it goes against my belief system; however, if it did not, I would find it perfectly acceptable. There is nothing objectively right or wrong in the picture, as there is no impartial observer. History is written by the victors, so now we find ourselves in a battle not of swords, but of words. Joseph Goebbels is credited with the quote “repeat a lie often enough and it becomes the truth”. So, thus, if we allow the lies to stand, they become truth.

Enter the internet. Thanks to the miracle of forums and private areas on the internet, it is possible to create websites that let those lies go unchallenged, lets them be echoed, and makes of them the truth. The others become liars, cowards or cogs in a great evil conspiracy against the people — how dare they not kill other people?

This is an issue. It is one we should take seriously. We should not be afraid to seek out dissenting opinion, and figure out the root of it. It is much harder to do than to simply ignore it; wave at it and hope it goes away. But freedom is not something we can ever take for granted; not because of an evil conspiracy, but because we as humans have yet to fully evolve beyond tribalism.

What I do know is that, while the German law, while the Facebook directives, while they may be well intentioned, they are misguided. Three decades in, if there is one thing we should have learned about the Internet is that information and opinion are never deleted. They just move somewhere that you cannot see. That is a bigger problem.