#column store

Netflix put together a tool, Raigad, to help automate and operate ElasticSearch:

collects and publishes metrics to a centralized monitoring system

manages ElasticSearch node discovery and tracking using Cassandra

auto-configuration

index management

Judging by the logo, I think Raigad refers to the Indian fort built in 1030.

01: The real power of the Cassandra asynchronous database driver. The database is the heart, but its drivers are the veins. Just imagine what a clot would mean. ★

(nb: if you decide to call me biased on this one, I challenge you to check the driver’s features and API and let me know what you’d like to see being done differently).

02: A (big) surprise for me to learn that Hortonworks and Databricks (Apache Spark) were not in a partnership. At least in the light of the work Hortonworks has already put into Spark. Now things are good again. ★

03: Big name added to Aerospike’s customer list: KAYAK. Deployed on 2 datacenters on real servers using SSDs, Aerospike replaced ehcache and memcache. ★

04: After reading about Kayak and Aerospike, I’ve searched for similar case studies and found this article from Nov. 2013, about Amadeus and Couchbase. In that case the replaced technologies were Memcached and MySQL. ★

05: Astute description of the current world of data: “In a world where IT budgets grow at 2% per year–at best–and data is growing at 60% per year, something has to give”. ★

06: Big data is (presented as) the Holy Grail for the sellers: Big Data analytics has the potential to allow sellers to directly influence buyers behavior based on their past behavior and habits. ★

I’d like to know what’s the best a company can do for me (the customer). Unfortunately, I don’t think there are that many companies where their customers’ and the company goals are aligned. For now, I’ll stick with my initial reaction: “man, this sort of future will really suck”.

07: Hadoop would be perfect if only 12 things would suck about it. Not to mention that most of these are actually related to Oozie, Amabari, Knox, etc. I wholeheartedly agree though that a project with poor documentation and with strange and unexplainable exceptions and error messages will leave a very sour taste; so sour that many users will start looking for alternatives as soon as possible. ★

08: Two numbers from Wikibon’s survey of the Hadoop market: 1) 25% of Hadoop users are paying for subscriptions; 2) 65% of Big Data practitioners1 have started migrating from Enterprise Datawarehouses to Hadoop and BigData. Very encouraging numbers. ★

Other analysts have different data and thoughts though.

09: This article about Cassandra replacing Oracle at iovation is the perfect exemplification of the benefits of Cassandra: an always-on database, with better scalability and performance, development agility, all at a price that makes sense and scales with your business. It’s also useful to see how the CTO of iovation has framed and prioritized the main issues he wanted addressed. ★

10: A pretty good and well paced “let me introduce you to HBase” including non-boring details about its roots, data model, installation and up to running some basic operations in the HBase shell. ★

11: LinkedIn did some restructuring of their data science teams; a team which can be considered one of the first data science teams around (at least DJ Patil who lead this team before 2011 is said to have come up with the data scientist name). Many of the original members of the team have already left (and have now leading roles in their new places), but if your company needs an experienced data scientist, this might be a lucky time. ★

12: If you like experimenting with data, Alexandre Passant has a blog post giving basic details about getting data from Deezer, uploading it to Google Cloud Storage and then using it from Google’s BigQuery. ★

Leaving aside the first step (getting the data), we have to agree that this process got a lot simpler over time and now it’s down to what ideas and algorithms you apply to the data.

Having data remains the big problem. That explains why companies fight and invest that much into different data platforms and try to get as many data points as possible. Get it now and you’ll figure out how to use it later. The opposite won’t work.

13: Darshan of MongoDirector looks at the current trends related to databases, starting with DevOps and finishing with the numerous Database-as-a-Service products, to suggest that the future of DBAs is being part of the dev team as a developer specialized on databases. This is already quite true for small companies. I’m not sure though about the enterprises that that revolve around governance and access rules and policies. ★

14: A ton of papers about visualization categorized and presented in a very nice visualization. Even if you don’t plan to read any of these papers, I strongly encourage you to take a look at this page to see how to build a helpful navigation and visualization of a swath of scientific papers. ★

15: If you haven’t read an article about the CAP theorem in a while, Robert Hodges wrote one in which he shares his thoughts about the possible differences between the theorem terms and real live, and then about its applicability in the real systems. It’s not one of those posts proposing there’s a way around the CAP theorem or the more radical the CAP theorem is wrong, buy our product. It’s rather one in the vein of Daniel Abadi’s Replace CAP with PACELC: Problems with CAP, and Yahoo’s little known NoSQL system. ★

16: One of the major Hadoop events, Hadoop World, through the eyes of one of the top analysts, Merv Adrian from Gartner: last year’s Hadoop 2.0 and YARN, SQL-on-HDFS noise, was replaced by Spark. And a lot of companies are jumping onboard. ★

17: Mark Needham advises to use specific relationships (as opposed to generic ones) in Neo4j. And he presents the data behind his advice. ★

18: A free mini-book, “The DBA’s Guide to NoSQL and Apache Cassandra” authored by a RDBMS DBA turned VP of Product2 at one of the leading NoSQL database providers, DataStax. ★

I’m not very sure how this is defined. ↩

He’s also my boss. And someone that I had and still have a lot to learn from. ↩

Robbie Strickland (Weather Channel’s software development manager):

“It used to be that Cassandra was something of a beast,” recalled Robbie Strickland, the Weather Channel’s software development manager. “It was like an old space shuttle cockpit with a million knobs.” In fact, prior to joining the Weather Channel, Strickland says using Cassandra required spending “most of the day in the Cassandra IRC channel talking to developers.”

Some databases evolve into more powerful and friendlier tools, while others are just “familiar giant hairballs growing bigger“.

Konstantin Boudnik (WANdisco):

The idea behind consensus-based replication is pretty simple: instead of trying to guarantee that all replicas of a node in the system are synced post-factum to an operation, such a system will coordinate the intent of an operation. If a consensus on the feasibility of an operation is reached, it will be applied by each node independently. If consensus is not reached, the operation simply won’t happen. That’s pretty much the whole philosophy.

Not enough details, but doesn’t this sound like Paxos applied earlier?

Adam Fowler1 looks at the potential confusion for Cloudera’s customers when talking about NoSQL databases:

As for Cloudera customers I’m not too sure. It may confuse people asking Cloudera about NoSQL. Below is a potential conversation that, as a sales engineer for NoSQL vendor MarkLogic, I can see easily happening:

This announcement struck me as being too publicized — it’s normal for companies with similar interests to partner, but a fair amount of care should be put into clearing all possible confusions and I don’t think this happened.

Just to summarize: Cloudera provides support for HBase and Accumulo. And it has a deal with MongoDB and Oracle. I assume in the sale process, Cloudera will go with: “we work with whatever you already have in place”. As for recommending a NoSQL solution for their customers, it will probably go as in Adam Fowler’s post. To which we could probably add Oracle too.

Adam Fowler works for MarkLogic. ↩

You’ll probably recognize the names behind OhmData and their improved HBase product C5. In their own HN words:

We say we can do failover in a couple of seconds. We want to make it subsecond, but we can’t do that reliably yet. In HBase this story is much more mixed.

We wanted to really reduce complexity, as a result, you can just apt-get install c5 on each node and you are done. It’s one daemon, one log file, and that’s it. No xmx nonsense, and almost no tuning or config files. I don’t know if you have dealt with hadoop before, but the complexity is high.

Finally we have a much more advanced wireformat. In fact it’s advanced by being simple (protobufs + http). As a result clients in languages other than java become very easy, without a thrift client.

Are we in a new stage of NoSQL databases: “X that doesn’t suck”?

Google using Cassandra to show the performance and cost efficiency of the Google Compute Engine:

sustain one million writes per second to Cassandra with a median latency of 10.3 ms and 95% completing under 23 ms

sustain a loss of 1/3 of the instances and volumes and still maintain the 1 million writes per second (though with higher latency)

scale up and down linearly so that the configuration described can be used to create a cost effective solution

go from nothing in existence to a fully configured and deployed instances hitting 1 million writes per second took just 70 minutes. A configured environment can achieve the same throughput in 20 minutes.

Make sure you check the charts and get to the conclusion part. The other conclusion I’d suggest is: based on the real benchmarks I’ve seen over the years, Cassandra is the only system that was proven to scale lineary and provide top performance1.

Before saying that I’m biased, make sure you are reading at least this story and Netflix’s post. ↩

Great post by Nick Dimiduk1 covering the whats, whys, and hows of caching data blocks in HBase, the mechanism through which HBase is optimizing random reads2:

There is a single BlockCache instance in a region server, which means all data from all regions hosted by that server share the same cache pool. The BlockCache is instantiated at region server startup and is retained for the entire lifetime of the process. Traditionally, HBase provided only a single BlockCache implementation: the LruBlockCache. The 0.92 release introduced the first alternative in HBASE-4027: the SlabCache. HBase 0.96 introduced another option via HBASE-7404, called the BucketCache.

Nick Dimiduk works at Hortonworks and is the co-author of HBase in Action. ↩

For optimizing recent edits, HBase has another mechanism, the MemStore. ↩