#memsql

MemSQL raises $30M Series D round for its real-time database

MemSQL, a company best known for the real-time capabilities of its eponymous in-memory database, today announced that it has raised a $ 30 million Series D round, bringing the company’s overall funding to $ 110 million. The round was led by GV (the firm you probably still refer to as Google Ventures) and Glynn Capital. Existing investors Accell, Caffeinated Capital, Data Collective and IA…

PostgresSql, MongoDB, and MemSQL.  I am going to push data, learn about data, become data (is that too weird?).  After months, if not years, of trying to settle on which databases to thrown my life into, I settled on those three.  And I don’t know that i can tell you exactly why.  But here is what I can tell you:

I was looking for three types of databases: an Open Source RDBMS, an Open Source NOSQL database, and an In Memory Database.  Open Source was important because of cost.  I am running a start up and of course, money is everything.  So free is always good.  As far as MemSQL, I am using the Community Edition for now, again, because of cost.

My reasons for using each, really are simple.  I chose Postgres over MySql because I have the opportunity of using Postgres at my current work; MongoDB because I have heard and read about it every where for years and seems extremely stable; and MemSQL because, well, they didn’t pester me like some of the other In Memory DB.  Real reasons, all of them, in my book.

But in all honesty, in this day and age where you can find so many options for everything, in order to choose the “correct” thing, it would require so much research.  And let’s face it, by the time that research is done, new versions are out with new features, new databases show up promising so much more than others, that’s you might as well start over again.  In the end, specially for start up like mine, a decision like this comes down to just a gut feeling.  So here I am.

In the next few weeks and months, I will share the project I am working on that requires these three databases; their installation and the struggles that goes with it; the loading of tons of datasets and the issues that will come from it; and of course, the building of a business out of these databases and the data they will contain.

Coinalytics Opts for MemSQL to Speed Up Blockchain Technologies

Now that payments companies and merchants are rapidly adopting digital currencies like Bitcoin, blockchain technology becomes even more important as a ledger for verifying and analyzing transactions.

MemSQL (www.memsql.com), the leader in real-time databases for transactions and analytics, has announced that Coinalytics, the real-time intelligence service for blockchain, has tapped MemSQL to more…

Great article, with new ideas and architecture organization. Some examples of interesting topics:

Social media streams have become increasingly relevant in managing customer relationships and are generating volumes of time-driven streaming data.  And finally, general commerce but especially ecommerce has begun to recognize the need for real-time response with advance analytics in the range of well under one second. 

...

All of these data streams have driven the adoption of stream processing and in-stream analytics, and guess what; Stream Processing relies on in-memory speed. 

And the most important idea is a vision. A desruptive future, FREE OF ETL JOBS:

As thousands of people gather in the Bay Area this week for Strata + Hadoop World, we wanted to share how data-driven decision making is in our company DNA.  Most recently, we built a real-time data pipeline to ingest data into MemSQL using Spark Streaming, as well as a highly scalable infrastructure that collects, stores and processes user engagement data in real-time, while solving challenges that would allow us to achieve:

Higher performance event logging

Reliable log transport and storage

Faster query execution on real-time data

Let’s take a deeper look.

Higher performance event logging

We developed a high performance logging agent called Singer, which is deployed on all of our application servers to collect event logs and ship them to a centralized repository. Applications write their event logs to local disk, from where Singer collects and parses these log files, and then sends them to a central log repository. Singer is built with at-least-once delivery semantics and works well with Kafka, which acts as our log transport system.

Reliable log transport and storage

Apache Kafka, a high throughput message bus, forms our log transport layer. We chose Kafka because it comes with many desirable features including support for high volume event streams, replicated durability and low latency at-least-once delivery. Once the event logs are delivered to Kafka, a variety of consumers from Storm, Spark and other custom built log readers process these events in real-time. One of these consumers is a log persistence service called Secor that reliably writes these events to Amazon S3. Secor was originally built with the purpose of saving logs produced by our monetization pipeline where 0-data loss was extremely critical. It reads the event logs from Kafka and writes them to S3, overcoming its weak eventual consistency model. After this point, our self-serve big data platform loads the data from S3 into many different Hadoop clusters for batch processing.

Spark + MemSQL Integration

While Kafka allows for consuming events at real-time, it’s not a great interface for a human to ask questions on the real-time data. We wanted to enable running SQL queries on the real-time events as they arrive. As MemSQL was built for this exact purpose, we built a real-time data pipeline to ingest data into MemSQL using Spark Streaming. The pipeline is in a prototyping phase as we continue to work with the MemSQL team on productionizing it.

Figure 1 shows the elements of the real-time analytics platform we’ve described so far. We’ve been running the Singer -> Kafka -> Secor -> S3 pipeline in production for a few months. Currently, we’re evaluating the Spark -> MemSQL integration by building a prototype where we feed the Pin engagement data into a Kafka topic. The data in Kafka is consumed by a Spark streaming job.

In this job, each Pin is filtered and then enriched by adding geolocation and Pin category information. The enriched data is then persisted to MemSQL using MemSQL’s spark connector and is made available for query serving. The goal of this prototype was to test if MemSQL could enable our analysts to use familiar SQL to explore the real-time data and derive interesting insights.  

While we continue to evaluate MemSQL, we’ll be showcasing a demo of it at Strata + Hadoop World 2015 along with the MemSQL team on Thursday, February 19 at the San Jose Convention Center. Come visit us at the MemSQL Booth 1015 for more details.

Demo built with Spark & MemSQL

Krishna Gade is an engineering manager on the Data team.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.

1.15.2013///NY Enterprise Tech Meetup @ Cooley

On Tuesday, January 15, 2013, I attended the monthly NY Enterprise technology meetup at Cooley. The featured demos were eZ Systems, a globally recognized commercial, open source content management software provider; Pursway, a big data algorithm developer that identifies social networks of customers and key influencers in that network; MemSQL, an in-house memory database for real-time analytics…

According to this, the next version of MemSQL will support JSON as a data type, similar to PostgreSQL JSON type, but with a slightly—really, just slightly—better syntax and function names:

By supporting the JSON datatype within a high performance database, MemSQL enables real-time analytics on data feeds of variable structure.

Use standard SQL over JSON data, including built-ins, GROUP BY, JOINs, and more.

Create JSON indexes online.

Client drivers require no changes to support JSON.

JSON properties are updatable.

Just in case you are wondering how this sits with me yelling about a premature return to SQL, keep in mind that:

MemSQL is SQL-based

They’re trying to extend SQL on top of JSON

So, I guess that’s OK. It won’t be easy though as they’ll have to address some interesting problems—null vs missing attributes, applying aggregation functions on heterogenous types, and so on. Or they might decide to go with “f### it, just normalize your JSON data first”.