elastic.io - Automate your data flows in the cloud

I was spending couple of evenings last week looking for alternative to Heroku that is hosted in Germany. Heroku is an absolutely awesome Platform as a Service (PaaS). We at elastic.io are using it (and paying for it) last two years and very happy with value it provides, however after the NSA scandal we are getting more and more questions about data protection and where our servers are physically located (note: we are a German company and Germany has the best data protection laws in the world).

While looking onto the PaaS alternatives I suddenly found many different open-source PaaS platforms like Cloud Foundry, Dokku, Deis, Flynn which is actually a strong indication of a simple fact - Platform as a Service is a commodity. Almost all of these products (may be except Cloud Foundry) are young open source projects, however based on my superficial view deliver similar set of features one would expect from PaaS.

This is actually good news for PaaS users - as typical PaaS feature-set is obvious now commoditization and open source products will lead to more players on the market, more competition and lower prices. At the same time it is a bad news for Heroku (or better to say Salesforce) and other players on PaaS market (like CloudBees, CloudControl, and many others) - commoditization, increased competition and lower margins will push PaaS players into different higher-margin areas.

According to Gartner report on aPaaS many of PaaS vendors them realized it and started to add new products/features to the PaaS platforms - like plug-ins/add-ons marketplaces, synchronization / data integration and  RAD tools.

In our last post we summarized what we need in our new content management system (CMS). After careful considerations and analysis we selected a new interesting CMS system. It is called prismic.io. And it's cool not only because it's name ends with .io (which is actually a top level domain for British Indian Ocean Territory) but also because it satisfy (almost) all of our important selection criterias.

Dear startups of the world, how do you do your web-sites? How do you maintain and keep information on your website up to date?

Our website so far is build on top of a Twitter Bootstrap Theme we bought from wrapbootstrap.com. We took the theme, adjusted the content, put it inside the simple node.js express application and pushed it to Heroku. That's it (how our blog works is a topic of another post).

With this approach we have maximum simplicity and control over how our web-site looks and how it works, however one drawback is - all people editing the website should speak Boostrapish, HTMLish and CSSish (LESSish). This wasn't an issue before as our team was small (do things that does not scale) but as we grow as a company we need to find a better solution - we need a proper Content Management System (CMS).

I have to say there are many CMS out there, the most prominent/well-known are Wordpress, Typo3, Joomla, etc. Before we could make a choice (to be honest we already have a winner) here is my line-up of the 5 must-have features we look for in our new Content Management Systems.

Feature 1 - No Pages, please.

Content is no longer a number of pages - the page structure, navigation, composition should be orthogonal to the ideal CMS structure. We need content pieces that can be placed everywhere, a page, an app, an e-mail. CMS should take care about the content and designers should take care about content composition -  pages/navigation/etc.

Feature 2 - Collaboration and Version Control

We use Git everywhere. Non-colaborative, non-versioned way of working is not acceptable. CMS system should support strict versioning and proper release management.

Feature 3 - Clean HTML

You saw it before - WYSIWYG editors that do not produce a proper HTML. I mean HTML which is not renders ok but also semantically correct. One of our preferred solutions right now is a Markdown, just like Github did it.

Feature 4 - Google SEO Friendly

Another important feature. Content is the king. We have to make sure Google can read it.

Feature 5 - Availability, Scalability and Speed

As you might already know that Amazon have found that every 100ms of latency cost them 1% in sales revenues. The CMS we want shall not only be available and scalable, it should also be very fast (or at least have a  good caching architecture).

If you are like me you already seen many e-mails and blog posts about OpenSSL Security Issue (aka. Heartbleed). We took all necessary steps to protect our servers.

It is important to note that none of our systems handling HTTPS/SSL connections directly. We are using Heroku PaaS which is in turn is hosted on Amazon AWS, SSL connections are handled by Heroku's SSL Endpoint which is nothing else but SSL Termination Proxy.

Immediately after Amazon and Heroku fixed the issues we updated our certificates, passwords and authentication tokens.

It's important to note that we are not storing data that we transport between SaaS applications, as well as we never store (unhashed) passwords.

Integration is not a new problem. There are many different ways to solve it, but most of the time it's the end-user's problem.

For example if you are using CRM X and you would like to push your leads to billing/invoicing/marketing system Y - it's usually your problem. You can expect Y and X to give your APIs with keys and secrets, however programming and maintaining integration between them is still very complicated and error-prone process that can only be understood by 1% of internet users.

We believe that there should be another way to achieve this. Integration is not an end-user's problem it is actually the the software vendor's problem. Software vendors should give a simple one-click integration solutions to their end-users that can be used by 99% of us.

That's why we created elastic.io Integration Widget.

It is just a piece of JavaScript code that can be placed on every web-site. Once triggered it will give you access to your end-user data - for example it will allow you to import your end-user contacts from Google Contacts, Xing and Linked-in. When using it in your application you will not need to pay attention to different APIs, data formats, OAuth authentication flows, etc. Your end-user will be presented with a beautiful UI that will guide them through the authentication flow. But that's not only it - we will also carefully extract, transform and normalize data so that you will have it at your disposal in ready-to-use form.

You can give it a try right now with this simple JSBin or on our website or have a look into the documentation.

Another great feature of our one-click integration widget is file export. The main goal of it is to export (and in future import) files into different cloud storages - you may think about it like a universal gateway to Dropbox, Google Drive and Box.com cloud storages. It works the same way as contact importer - it will guide your users through the OAuth authentication flow and selection of the target storage, once it is done you will get a user-specific WebHook URL - every file you will push there will be placed into the end-user selected storage system. It works as simple as an HTTP POST request and require only a one authentication header from your side (we will handle different authentication for different cloud storages on your behalf). Want to give it a try - check it on our website or go in documentation.

Today we would like to invite you meet Renat, Igor and Yvonne from our team at our booth at the world-biggest IT exhibition CeBIT in Hanover, Germany. 

Some facts:

3500 companies from 70 countries that will present their products

Around 300,000 visitors

1000 international speakers presenting from 20 different stages

All of it only in 5 days from 10th to 14th of March!

You can find our booth as part of code_n area at Hall 16, Stand D30. You will get a unique chance to see our new products in action and talk to elastic.io founders!

Ah, and by the way, if you need a ticket to visit us just tweet this:

I'm going to meet @elasticio at #CeBIT - Hall 16, Stand D30

It's been a long time we haven't posted on our blog. Now this time is over and we are opening a new page in elastic.io history. This is an important announcement we were working for last couple of month.

We would like to announce that we have closed a significant seed investment round with a group of private investors in Germany. 

We will use these resources to significantly speed up the product development, sales and marketing efforts. We will have more power to innovate and iterate quickly, and we already have two new products which will be released very soon.

CSV Writer component will transform incoming data to CSV rows. To demonstrate how it works we will take this flow:

WebHook component will get following configuration:

We will send three parameters - first,last and email. Now let's have a look into CSV Writer configuration:

Following fields can be configured:

File name - name of the file that will be created

Append to File - flag that will show if new file will be created every time or new data will be appended to the file

CSV Header - field similar to CSV Reader where you should specify target CSV format that you would like to create. You can copy-and-paste CSV header or just Drag-and-Drop sample CSV file.

For our sample we will take following simple CSV header

First Name,Last Name,E-Mail

After you enter the CSV Header sample don't forget to map incoming data. As you entering CSV Header you will notice a Mapper component before CSV Write will be automatically filled with metadata that reflect CSV file structure:

Last but not least specify the Dropbox target account and start the flow. Now we can test it with simple curl by sending an HTTP request to the WebHook:

> curl -d 'first=Homer&last=Simpson&[email protected]' https://in.elastic.io/hook/52553eb7cb7eac0200000006 {message: "thank you"}

After that you will find a following CSV file in your Dropbox:

Let me demonstrate how CSV Read works. You would need to feed CSV  Reader with the CSV file data. Source of it could be Dropbox, Google Drive, SFTP or any other kind of storage. We will start with this flow:

When configuring this flow you will notice new CSV Read configuration panel:

What you can do here:

In the CSV Header you can specify the structure you expect in your CSV file. You can copy and paste first Header line in the text area or (if you have a modern Browser) just drag and drop a sample CSV file into the area below.

Separator field will let you specify what column separator is used by your CSV file

Skip rows will let you specify how many rows of your CSV files are occupied by header (does not contain actual values)

After you place an expected CSV structure you will see something like this:

You can configure a parsing rules of each individual column in your CSV file. We can use following types:

String - no formatting, value will be taken as it is

Number - you can specify the decimal separator (comma or point)

Date - you can specify different date formats

Boolean - will be true if value is 'yes' or 'true'

For our sample I will take following CSV file:

First Name,"Last Name","Birthday"",",,E-Mail,Income Renat,Zubairov,04/04/1980,Russian,[email protected],1.5 Homer,Simpson,01/01/1970,American,[email protected],75000

And as a target for our WebHook I will use a great RequestBin tool, after starting my flow I will see following in my request bin:

This blog post will shortly describe how are building and delivering our product every day. When we started with this automation we tried many different tools our there. Our goal was to implement a automated best-of-breed build and deployment pipeline without installing and maintaining our-own development, continuous integration or version control servers.

We are happy to present one of our customers - SPHERE.IO a product of Commercetools GmbH.

Made by developers for developers SPHERE.IO is the most versatile cloud-based commerce platform. You can call it Software-as-a-Service or Platform-as-a-Service. We call it Cloud Commerce OS.

The idea of sphere.io is very simple yet amazing - use API to decouple user experience from the backend. E-Commerce companies are just like other IT/Internet businesses facing new mobile challenges. Tablets have significantly changed the shopping behavior. Smartphone users are demanding on-the-go experience. Another e-commerce challenge is that products are no longer simple like 'once-piece-one-price red t-shirt' but the digital goods or subscriptions. Answer for both of these challenges is a sphere.io Cloud Commerce OS. It's a flexible, api-based backend as a service product where front-end can be programmed easily on top of it, being it a native iOS or Android App, web-site, TV app or all of these.

In the last weeks we have been very busy developing our SDK that will allow API providers to integrate into elastic.io integration platform with ease. The progress looks quite good so far. Because we are eating our own dog food, we started implementing APIs using our SDK. Here is the list of new APIs available at elastic.io platform that have been implemented using our SDK:

Zendesk

Paymill

Sphere.io

Admineo

As you can see, all these APIs are implemented in a descriptive way. We use Swagger with a bunch of minor extensions for API description. Additionally every description may have programmatic extensions. The full documentation of the SDK will be announced in a separate blog post.

Apart from that we are happy to announce the SFTP app that allows you to read and write files on a remote SFTP server. Now you can scan a directory of a SFTP server for new files and trigger an integration flow for each new file. Also upload of a file to a SFTP server is a "piece of cake" now.

On July 25th we will give a talk Integration in the Cloud at RuhrJUG (Java User Group Essen). After a brief introduction into SaaS, PaaS and IaaS, we wil dive into technologies we are using at elastic.io. We will cover JavaScript and node.js. Then we will introduce the attendees into our SDK which allows any API provider to be part of our platform.

elastic.io White Label Showcase - Fill your integration appstore:  See how Salesking - one of elastic.io's customers - allows its end customers to backup invoices in Dropbox with the help of the elastic.io White Label product, i.e. embedding the integration usecase in the native website of salesking, so the end user can activate the integration with ONE CLICK. Write to [email protected] for more info on the white label product.

Around 95% of all my e-mails I first read on my iPhone. I could later open them on my desktop, but ~80% of my e-mails I never see on the Desktop.

elastic.io proudly received an award as one of the Top 3 Start-ups in Cloud Computing in Germany from Europe´s most important Cloud Computing Association Eurocloud at the Eurocloud Gala Dinner which followed the Eurocloud Conference 2013! A big Thank you to the organizers from the Eurocloud Team, in particular to Andreas Weiss and Dr. Bernhard Becker for this fantastic event. We were extremely happy for the opportunity to also contribute presentations about cloud integration in various slots during the conference itself!

Photo Source: http://www.flickr.com/photos/ecoev/sets/72157633593288784/