Revelcloud

8A few months ago, we blogged about some surprising metrics coming from Azure Disks.  Notably, we found instances where Standard Storage was faster than Premium Storage and instances where Premium Storage was dreadfully slow (as in 2MB/s).  A great guy from the Azure storage team reached out and helped me to understand better what was going on.  

Here’s my take on what’s going on.  Caveat: I’m not a storage expert, just a guy trying to make sense of this stuff.

First off - when measuring the disk throughput, it is going to be governed by the “transfer request size” and the amount of concurrency (”number of threads” in CrystalDiskMark or “Outstanding I/Os” in Iometer).  Azure disks (and likely all cloud disks) are more heavily dependent on request size & concurrency to achieve high throughput than would be a local drive.  We’ll see how that plays out below.8

Before we update the benchmarks, let’s update/explain some of our findings from the original post.

In the initial post, CrystalDiskMark showed a dreadful 2.207MB/s write speed for the Premium Disk using a 4k transfer request size and 1 thread (this is the bottom right cell of the test) whereas the Standard Disk showed 61.82.  The reason for this is that the Standard Disk was also the o/s disk and this disk has read/write caching enabled - so it was the write cache that “sped up” this test.  As of yet, you cannot enable the write cache on data disks and using the write cache does expose some risk of data lost.  So, the conclusion that the Standard Disk is faster in some cases than the Premium Disk is flawed - it’s the “Standard Disk with read/write caching” that may be faster.

Regardless of #1 above, we’re still left with the question: “Why does Premium Storage only deliver 2.207 MB/s write speed especially when local SSD is 40x faster (82.75MB/s) for the same test?”  The answer, turns out to be pretty straightforward.  The cloud drive is limited to 500 IOPS per thread - so with a 4K transfer size and 1 thread, you get about 2MB/s (4k * 500/s = 2MB/s).  If you change this to 2 threads, you get 4MB/s.  This is the most simple example of how concurrency affects cloud drives.  You could also get 4MB/s by doubling the transfer size from 4k to 8k.  

Now, let’s update some benchmarks using Iometer (the Azure storage expert showed me this tool which provides more explicit control over transfer size and concurrency.  

I’m going to use a 4k transfer size, as this is my best guess as to what is likely for ElasticSearch (I’m still very unclear on how transfer size is affected in “real life” between the application stack and the o/s).  This corresponds to the built in “4 KiB; 0% Read; 0% random” ‘Access Specification’ in Iometer - with 0% Read, I believe this means it’s a write-only test.  When I use the term “threads” below, it corresponds to the “# of Outstanding I/Os” in Iometer.

I’ll just provide the numbers for the Premium Disk (P30 disk on DS12 Azure VM - 4 cores, 28GB RAM) compared to local SSD (4x 800GB SSD in Raid 10 - 6 cores, 64GB RAM).  I realize the machine sizes are different and may skew the results (but hey, you’ve got to work with what you’ve got!).

The maximum IOPS for the DS12 (12,800) and the P30 disk (5,000) are taken from this article as are the maximum throughput for the DS12 (128MB/s) and the P30 (200MB/s).

Premium Storage - 4k, 1 Thread

Notes: the highlighted portions show the key metrics - IOPS (474) and Throughput (1.94MB/s).  This is exactly what we expect - a single thread is limited to 500 IOPS and with a transfer size of 4k, we get the expected throughput (4k * 500 IOPS = 2MB/s).

Premium Storage - 4k, 8 Threads

Notes: again, we get the expected result.  IOPS goes to ~4,000 (8 threads * 500 / thread = 4,000) and throughput scales accordingly.

Premium Storage - 4k, 16 Threads

Notes: here we get something interesting as we see that the IOPS is not 16 * 500 = 8,000 but rather 5,000 which is the limit for a P30 disk.  We can conclude here that: The maximum throughput on a single P30 disk with a 4k transfer size is 20MB/s.  This is not a limit of the VM, but rather the disk.  In theory, you could stripe 3 P30 disks on a DS12 and increase the max IOPS to 12,800 which is the limit for the VM.

Premium Storage - 32k, 16 Threads

For kicks, I ran this test which has a very high transfer size (IMO) and a very high degree of concurrency (16 threads).  But, this does provide the maximum throughput as advertised on the DS12 (128MB/s).

Notes: here you can see that while we haven’t reached the maximum IOPS for the P30 disk (5,000), we have reached the maximum throughput for the DS12 (128MB/s).

Local SSD - 4k, 1 Thread

Local SSD - 4K, 8 Threads

Local SSD - 4K, 16 Threads

Notes - Looks like we’re getting close to the max IOPS for this disk array.  Even when I move to 64 threads, it only drives IOPS to 26,000 (minor increase from the 24,685 in the 16 thread test).

Local SSD - 32k, 16 Threads

Notes - What?  1GB/s?  Now that’s absolutely blazing fast!

---------------------------------------------------------------

Conclusions:

This is a pretty complex scenario, dependent on #cores, Windows, ElasticSearch, Lucene, Java, document size etc.  I feel pretty good that they capture a “typical” ElasticSearch scenario, but obviously there are a lot of variables involved.  Regardless, for this test setup, bastardized though it may be, I feel OK in making this conclusion - and it matches my ‘empirical’ evidence in our applications having run on both DS12|Premium-Storage and physical machines with Local SSD.

SSD is significantly faster (5x to 30x) than Premium Storage, regardless of transfer size or concurrency.  For “typical” ElasticSearch scenarios (see below) SSD is 3x to 5x faster than Premium Storage

“Typical write” ElasticSearch scenario (4k transfer, 8 threads), SSD is 5x faster than Premium Storage (92MB/s vs 15MB/s).  By default, ElasticSearch uses #processers as the number of bulk operation threads - this means only 4 threads on the DS12; a good optimization if running ElasticSearch on a DS12 may be to increase many or of all the threadpools.

“Typical read” ElasticSearch scenario (4k transfer, 8 threads), SSD is 3x faster than Premium Storage (281 MB/s vs 67MB/s).  [These tests are not shown above.]

It wasn’t what we wanted, and it certainly wasn’t easy, but today, we can confirm that we’ve moved all of our apps from Windows Azure.  This post explains why, after 4+ years on Azure, we’re moving on.

Our setup

We run a pretty simple setup - a number of web apps that pull data from ElasticSearch and get it back to the client.   Our ElasticSearch data spans about 1TB - so more that a few records but not “big data” by any stretch.  From April 2011 to April 2015, we ran on “Cloud Services” (aka Platform as a Service).  In April 2015 we moved to “Virtual Machines” (aka Infrastructure as a Service).  At the end (August 2015) we were running a single Web App with Auto Scale, connected to two DS12 VMs.  Each VM had 4 cores, 28GB RAM and used the “P30 Premium Storage” disk for Elastic Search.

Issue #1: PSS turned out to be second class

We bet on Platform as a Service.  It sounded great (someone else manages the hardware and the o/s) but eventually it just fizzled out.  It received very little updates and the “Cloud Drive” system that allowed you to mount a drive was deprecated, and the SSD-based premium storage that we wanted for ElasticSearch wasn’t made available to PSS - this is the primary reason why we moved to VMs in April 2015.  

As an aside, the PSS replacement which is now called “Web Apps” is solid (tons of updates and innovation) but this is for the web tier only (i.e. not ElasticSearch).

Issue #2: Storage/Disk performance

Premium Storage is confusing - the performance is based both on the machine and the disk (see here for details).  For the D12s I was using, it maxes out at 150MB/s - in theory.  In practice, we found out later that Premium Storage may be slower for writes than Standard Storage and it’s considerably slower than local SSD.  We posted details and benchmarks from CrystalDiskMark here.

That any cloud storage is slower than SSD is expected and forgivable - but that Premium Storage ended up slower than Standard Storage is not.  What’s going on here?  To find this out after we moved our infrastructure to VMs, was a huge disappointment and it slowed down our data pipeline considerably.

Issue #3: Random VM Outages & Useless Support

Sometime after we moved to VMs in April 2015, we began to have these “Critical Kernel Power” outages on our VMs.  The VM would just instantly go dark, without any orderly shutdown.  The event viewer would record this little gem:

“The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.”

In April, we lost both VMs for about 90 minutes and the issue (or at least the same symptoms) recurs every 10 days or so on each VM.  Luckily, it’s only been 1 VM at a time so our services have stayed “up” but this is a low quality of service for 2015.  We provided more details in this post.

Issue #4: Bizarre & Erroneous Billing

When we moved to VMs, we also deployed our web apps on to Azure Web Apps - with everything running in the same data center.  The Azure Web Apps connect to the VMs using an “Azure VNet” - this is just how Azure does it.  

But I noticed on my bill in April that it was about $1k more than I had expected.  Why?  All the traffic between the Web App and the VMs (in the same data center) was being metered and billed!  WTF?  Bandwidth charges in the same data center?

Anyway, endless support and months later, I did receive a credit.  But in the interim, I was covering up to $100/day in, what turned out to be, erroneous charges (nearly $4k in total).  Oh, and for good measure, the month after Azure acknowledged the mistake and credited my account - you guessed it - they charged me again for the same VNet bandwidth.  I’ve still got to get that one sorted with them for the last couple of months.

This kind of thing just saps time and resources - a major headache.

Issue #5:  Cost

In the end, it was the culmination of issues #1-4 (and a bunch of smaller grievances) that sent us looking for alternatives.  But our first motivation wasn’t cost - not by a long shot.  In the end though, we moved to physical hardware with local SSD.  

Our old setup of S2 Web App + 2x D12 VM (w/ P30 Premium Storage Disk) was running about $1150/month (not including the erroneous VNet charges).

The new setup with 2 physical machines, Windows licence, 1TB local SSD in Raid 10, load balancing, and firewall runs about $750/month.  It’s a month-by-month charge (although there was a $130 per machine one-time setup).  And these physical machines have 50% more cores (6 compared to 4) and 225% more RAM (64GB compared to 28GB).

We acknowledge completely that comparing two physical machines to the Azure setup isn’t “apples to apples” - but we will point out, that the two environments service the exact same need.

We’re now 14 days in on the new platform, and a few days ago we moved our last service over.  No issues (to date), way more capacity, and our monthly bill has dropped $400 (about 35%).  

Sometime after we moved to VMs in April 2015, we began to have these “Critical Kernel Power” outages on our VMs.  The VM would just instantly go dark, without any orderly shutdown whatsoever.  The event viewer would record this little gem:

“The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.”

Normally, it would be just 1 VM in the Availability Set but occasionally it would be both of them simultaneously resulting in our services being down to the public.  Normally, they would reboot in under 5 minutes and, thankfully, our ElasticSearch cluster was never corrupted.  At least once, the outage lasted for about 2 hours (during which time, I was unsuccessful reaching Azure support).  I did receive a credit for about $150 though, but that nowhere near covers the damage that was done.

You can see the frequency of the problem below:

Initially, in April, the support technician did a root cause analysis and a public postmortem data was released:

2015-04-22 04:41:00 : “A software bug caused Storage backend servers to be not able to handle requests promptly.“

2015-04-22 20:33:00 : “ A software update error caused the Storage availability drop and services degradation. ”

Azure’s acknowledgement of the severity of the issue was very poor - nowhere in the public postmortem does it say “some VMs may have been offline for up to 90 minutes as a result of this bug”.  That’s just poor transparency.

And I wasn’t the only one: I commiserated with total strangers over at trackjs who were dealing the same thing (their explanation posted here). 

Anyway, after the April outage, the same problem kept happening - maybe every 10 days or so but with only 1 VM at a time.  I don’t know if it’s the same underlying issue or a different one, and really how much should I care?  When I had the energy, I’d open up yet another support ticket and generally the response was:

“We finished investigating the issue that caused your Virtual Machine to reboot, and identified that this is due to a recently identified platform bug. Our core platform engineers have already identified the fix for this issue. This fix will be deployed in an upcoming release. We do not have an ETA for the release at the moment. “

[Update September 1, 2015.  After some work with someone from the Azure storage team, some of the results below were better understood and we ran some more sophisticated tests.  If you’re deep into this, this article might provide further insight]

We run a pretty simple ElasticSearch setup on Windows Azure - and disk speed for ElasticSearch is paramount.  We found our import process to be pretty slow (especially compared to off-azure imports with local SSD) so we thought we’d take a quick measure of the disk speeds for our VMs.  

Below you’ll see the results for a disk using Azure Premium Storage, a disk using Azure Standard Storage, and the ‘transient’ disk that comes with the “D” series VMs.  All the Azure tests were done on the same VM - a “D12″ instance (4 cores, 28GB RAM).  We also tested on a physical machine (6 cores, 64GB RAM) with an SSD array in a RAID 10 configuration.

The standout observations:

- Random writes are slower on Azure Premium Storage than on Azure Standard Storage and at 2MB/s are awful for any write-intensive operation.

- Local SSD is significantly faster in almost every category [no surprise there], although random 4k reads were slow than Azure Storage (premium or standard) [somewhat surprising].

Today, The Address Project went live in Florida... With over 5 million address listed, it's one of the largest online sources of information for Florida addresses. Most addresses include land size (acres), the square footage of the main building, and property boundary info.  The entire site is also fully searchable.  

Congrats to Candid, a Revelcloud client, who was just selected into Extreme's advanced cohort track.  There aren't two more deserving people than Candid founders Nariman and Dominic.  

We're pretty excited to announce that our founder, Scott Howlett, has joined the advisory board for Candid.

Candid enables real-time exchanges between candid photographers and brands.  Scott's really excited about working with Candid founders Nariman Haghighi and Dominic Gignac as they share a common vision around the cloud and social, especially in terms of how the economic benefits can be distributed more broadly.  Candid sums it up like this:

We're pretty excited to announce that we're now working with The September, an online store that's taking the headache (and extra fees!) out of shopping for designer shoes.

Founded by Aynsley Wintrip Harris (formerly of Four Seasons) and Christine Carlton (formerly of Holt Renfrew), they're making themselves noticed in the Canadian retail space.  Find out more at www.theseptember.com (and why not peruse the shoes while you're there!).

Part of our mission at Revelcloud is to launch a number of startup businesses that explore the opportunities that sit at the intersection of social, local, mobile and advertising.  Today, we're excited to announce that www.allhx.ca is up and running.  

Well, you've found us here on the www.  We're glad you came.  

A lot goes into a name, or at least in our case it does, so let us take a minute to explain ours.  "The cloud" means many things to many people but to us it takes on two very important dimensions: the first is the huge number of people worldwide that are connected to the Internet and the second is the huge power of cloud-based computing.