Paul Miller

Today is my first day as Senior Analyst at Forrester Research, doing cloudy stuff for the team focused on CIO Professionals.

Cloud of Data-specific things like this daily Tumblr post are going to take a rest.

But I’ll still be tweeting (of course), and I’m sure I’ll be blogging for Forrester soon enough.

I didn’t really get it, back in January 2014, when Intel, Amazon and others made a big noise about slapping the equivalent of an ‘Intel Inside’ sticker on your favourite cloud.

Now they’re at it again, with the birth of a new ‘Cloud for All’ initiative, a nudge for OpenStack, and the promise that Intel and Rackspace will - in six months - deliver two 1,000-node test clusters. Just wait for someone to comment that OpenStack/ Intel/ Rackspace must be hard/ slow/ lazy if it takes six months to walk down the racks pushing 1,000 ‘on’ buttons… 

I still don’t get it.

Test clusters are good. Intel, Rackspace, and anyone else who wants to work to optimise code, smooth wrinkles, fix problems, validate conformance and all the rest of it? Also good.

But the whole Cloud for All thing? Not so much. Lots of cloudy stuff runs on Intel. Lots doesn’t. Lots of cloudy stuff runs on OpenStack. Lots doesn’t. 

Intel was already working to make sure its chips and other gizmos suited the needs of their customers. Operators of clouds are one, increasingly important, class of customer. Intel was already collaborating with OpenStack, and with partners like Rackspace.

Giving existing collaboration a fancy name doesn’t make it more important, or better, or anything. It just looks a bit desperate.

In fact, if they weren’t already doing most of this stuff, heads should have rolled long ago. Doing what you should be doing isn’t the stuff of headlines and press releases. It’s the stuff of business as usual, behind the scenes, quietly, without fanfare. We only care when you’re not doing what you should be doing. When you’re doing what you should be doing, and you shout about it, we all go, “Well, d’oh.” 

As I mentioned in my post about last year’s announcement with Amazon, this all seems such an odd thing to shout loudly about. Cloud providers care deeply about increasing efficiency, decreasing energy consumption, and more. All of that goes straight to their bottom line. But their customers couldn’t care less, most of the time. Those customers don’t require an Intel chip, and they don’t require a partnership, and they don’t require squeezing a little more efficiency out of a physical server. They care about the virtual machines and the applications that are delivered to them. They care about how well they run. They care about how much they cost. 

Maybe.

It might, of course, be unwise for a single short response at the top of a set of search results to even try to ‘answer’ a question when we think, theorise or hypothesise towards an answer, rather than knowing it.

I’m not suggesting, for a moment, that Google’s current answer is worthy of serious consideration… but even currently accepted scientific hypotheses about big rocks from space, climate change, etc are rightly open to interpretation and debate. We weren’t there. We’re piecing together a lot of bits. The picture is very incomplete. We know (or even only think we know) an awful lot less than we don’t know about the end of the dinosaurs.

Unequivocal answers to clear questions are a perfect use for Google’s Knowledge Graph. The weather forecast, the capital of Mali, the population of Florida, the age of the UK’s Queen.

Complex scientific debates, where there are perfectly valid differences of opinion? They probably need to be represented in some other way.

Positive cloudy news from two of the big three, with Amazon Web Services (AWS) making some significant improvements to their DynamoDB service and Google signing on as a sponsor of the OpenStack cloud platform.

AWS has got this incremental improvement thing down to a fine art, to the extent that too many competitors have stopped paying enough attention. AWS launches services that are rather limited, and then tweaks and tweaks and tweaks until the pieces fall into place and something far stronger emerges. An awful lot of players in this space are frogs in water, and the AWS team just keep gradually turning up the heat.

Hmm.

Small sample, possibly not asking the right questions of the right people, etc, etc. 

But long-standing concerns and misconceptions remain concerns and misconceptions.

Worrying about whether or not (big) data can be used for good purposes is something of a thing, these days.

Three years ago, Roberto Zicari interviewed myself and others on the topic. Now he’s seeking support for the notion of ‘Data for Humanity’:

“We want to bring people from different disciplines and professions together, who share the motivation of using Data for the Common Good and for Human Wellbeing, in order to ensure that data serves humanity.”

DataKind matches data scientists and data tech with deserving causes.

And now Bloomberg is getting involved, with September’s Data for Good Exchange and a new Research in Residence Programme in partnership with one great good cause, UNICEF.

In technology, as in so much else, things go wrong all the time. Web sites go down, companies lose your data, and more. We all know this and - to greater or lesser degrees - broadly accept that it will eventually happen to us. The real trick, often, isn’t to prevent everything going wrong but to respond well when it does.

But what is a good response?

Implicitly, we think we know what it is. A good response, most of us believe, will be:

quick, admitting fault before being found out, and proactively keeping customers up to date throughout;

transparent, actually admitting you’ve messed up, explaining what happened, and discussing what’s being done to put it right;

proactive, contacting affected customers and offering to help, rather than waiting for the complaints to pour in;

above and beyond, doing far more than Ts&Cs require, offering significant additional service credits, freebies, and more.

And yet, so many businesses go out of their way to do the exact opposite. They’re secretive, their existing communications channels go dark, they wait for customers to complain and then pile form after form on them in the hope that they’ll go away. The most persistent complainers will, eventually, receive the measly ‘compensation’ to which the SLA or terms and conditions say they’re entitled. And everyone is left with a bad perspective of the company concerned.

Are the offenders hopelessly out of touch, overly beholden to stingy shareholders, stuck in patently absurd ways of thinking about business and customers, and doomed to fail?

Or are they playing a longer game, recognising that blips in service will be forgotten, and that keeping things quiet is a better way to manage a long-term customer relationship, a long-term market profile, and long-term stock growth?

“Azure Data Catalog is a fully managed service that stores, describes, indexes and provides information on how to access any registered data source. It closes the gap between those seeking information and those producing it… Any user, for instance an analyst, data scientist or data developer, can register, enrich, discover, understand and consume data sources. Every user is empowered to register the data sources that they use. Registration extracts the structural metadata from the data source and stores it in the cloud-based Catalog, while the data itself remains in the data source.”

$83 million for Digital Ocean, with its bare-bones spin on cloud-based infrastructure… and $24 million for Confluent, the company behind Apache Kafka.

“Focusing primarily on devices that consumers buy, as opposed to products that retailers or municipalities install, [Altimeter Group Analyst, Jessica] Groopman asked if companies should add something that looks like a nutrition label or perhaps some kind of other explanatory label (I’m picturing something like the washing machine care instructions on garments) to explain their data-sharing policies. Others have suggested some kind of Good Housekeeping seal of approval for privacy, perhaps administered by an organization such as Truste.”

It’s an interesting idea. Would/could it be nuanced enough to work, yet unambiguous and accessible enough to be quickly evaluated?

“…it has been determined that — in general — Members and staff in the U.S. House of Representatives, when conducting official business, now have a choice between using proprietary and closed technology and open source solutions that are restriction-free, reusable and frequently more cost-effective.”

That this is news, in 2015, seems bizarre.

Next step, getting the same permission for the US Senate.

Why is there even a debate to be had?

Open source is definitely not always best. But choice has to be a good thing. Surely?

Two stories in the past week, to remind us that big telecoms incumbents have the customer relationships, the brand recognition, the data centres, the network and the ambition to offer cloud services.

These are normally aimed at their existing customers, particularly big enterprises with existing co-lo relationships to these telcos. They’re also, normally, aimed at the telco’s home market.

And so, Deutsche Telekom “wants to create a European rival to big U.S. cloud providers.” The bullish headline is hopefully CBR’s rather Deutsche Telekom’s. Because unless the company has more realistic ambitions, it may as well give up now.

And around the world, in Japan, NTT is riding on the back of OpenStack to ‘[expand] its cloud system globally.’ The Japanese are taking a more realistic attitude to their competition, with Motoo Tanaka, senior vice president of cloud services, telling The Register that “it is not conceivable for us just to compete with Amazon simply on public cloud.”

Indeed.

There is plenty of scope for differentiated and valuable cloud services from companies like Deutsche Telekom and NTT. They can offer services that Amazon cannot. They can even compete - in some areas - with offerings that Amazon does have. But going head to head, with an undifferentiated public cloud offering? That would be a little silly.

This post is on a Google-backed site, and Google clearly has a stake in demonstrating the costs and risks of insisting that data be stored in-country.

But, accepting that the full picture is rather more nuanced, there’s still some interesting insight here.

Storing and processing data within your own borders may reduce competition, it may increase costs, and it may (possibly, perhaps) decrease security… but it comes with benefits, too.

And, presumably, the biggest risks apply most to the smallest countries. Germany, the UK, and other rich markets are big, rich and developed enough to (mostly) support competition, economies of scale, and a thriving security market.

Apache Spark is on a roll, but it’s not alone. 

Spark emerged from a Californian university to grab the attention of big data enthusiasts beginning to bang up against limitations in Apache Hadoop’s implementation of MapReduce. MapReduce is great at churning through batches of data, sitting on disk. It’s far less capable (for all sorts of reasons) handling streams of data, or data stored in memory. Spark is great at both of these use cases.

At around the same time, on the other side of the Atlantic, European teams were assembling the code for the software that would eventually become Apache Flink. Roberto Zicari has just published a good interview with one of the lead members of that effort; Volker Markl.

Both Spark and Flink are now top-level Apache projects. Both have moved beyond their regional origins, and enjoy contributions from developers around the world. And yet Spark continues to receive the bulk of the attention.