Blog | StorageDNA

StorageDNA was at NAB 2017 and as expected the media industry has attracted a host of storage technologies and vendors to help them deal with the growing amounts of storage. The following is a quick round up of what we saw this year and highlights “new” technologies.

Storage highlights at NAB 2017 can be divided into the following key categories:

Object Storage

SSD/Flash Storage

Cloud Storage

Tape Storage

Optical Storage

Object Storage

A couple of NAB’s back, one would find one or two object storage vendors. This has grown to over five or six vendors at NAB 2017. As a reminder, object storage is a new class of hard drive based storage that attempts to overcome the limitations of RAID with larger hard drives. Object storage vendors on display were:

Cloudian

Scality

HGST Activescale

IBM Object Storage (Cleversafe)

Object Matrix

Quantum Lattus

SSD/Flash Storage

Flash and SSD storage has been steadily dropping in price to the point where SSD/Flash storage arrays are within reach for data hungry media workflows. SSD/Flash storage arrays offer tremendous bandwidth and IOPS. While media applications are bandwidth intensive, parallel 4K operations (ingest, transcode, conform) are IOPS intensive and thus stressing traditional hard drives based RAID storage. Key SSD/Flash based vendors at NAB 2017 were the following:

Pure Storage

Quantum Xcellis All-Flash

Archion

DDP

DDN Flashscale

IBM Flashsystem, Storewize

Cloud Storage

Cloud storage and compute was a highlight at NAB 2017. Most users are determining how much of their storage and applications can truly live in the cloud. While a number of IT based workloads have moved to the cloud, media workloads are largely limited by the data sizes they operate on. That said, numerous vendors are offering storage and connectors to the cloud.

Vendors who are offering cloud storage includes:

Microsoft

Backblaze

Google

Amazon

Numerous vendors are offering connectors to help manage or upload data to the cloud. They include:

Axle Video

CatDV

Archiware

Xendata

Quantum StorNext

DDN WOS + S3

Avere

Tape Storage

Tape storage continues to play a major role in media and entertainment. With LTO-8 on the horizon rumored at 12 TB raw capacity and close to 400 MB/s of throughput, LTO continues to be a viable and cost effective storage medium.

We were also delighted to provide a sneak peek at dnaLTFS, the world’s first direct access file system for tape. We demonstrated how Davinci Resolve could be used to directly access LTO tape without performing restores.

Optical Storage

I am growing pretty tired of the ongoing FUD thrown at tape technology by disk and cloud vendors that state the true cost of tape is not just the media, but also the cost of the hardware, management and maintenance of the technology.

Most articles follow the same format:

They start with the assumed need to store a petabyte or more worth of files, content etc. They then calculate the cost of a very large robotic tape library with multiple drives that is spec’d to hold the entire petabyte+ online. This is then compared to the cost of a PB or more with a disk-based or cloud-based solution. And eventually it is argued that tape is not as cost effective as either. Here is a great example to one such article: click here.

Ok, I’d like to respond with one simple question.

Q: Are users (inclusive of small to enterprise users) comfortable with storing LTO tapes on a shelf?

A: YES, they are.

Let me elaborate…

From an end user perspective, when making a decision of where to store cold and rarely used data, tape jumps out as a clear choice. It literally costs pennies per GB, allows you to easily make multiple insurance copies and has long-term reliability on a shelf, along with little to no power or management.

Now, the decision to pick a single LTO drive, a great mid-range robot (e.g. Quantum i3, i6) or a thousand plus slot robotic library (e.g. Spectra Logic T950, Spectra Logic TFinity Exascale) to access their tapes is a decision based on each organization’s access needs. Today there are many customers that have multi-PB’s of archived material on LTO tape, yet have only had to invest in a modest size robotic library – one that specifically meets their accessibility requirements.  In other words, not everyone needs a very large (often expensive) robotic library to effectively create and work with lots of content stored on LTO tapes, which is contrary to how these tape cost bashing articles often position this point. They falsely argue that everyone needs to house their entire LTO archive in this big, expensive hardware and have it available online at all times.

Now the comparison becomes more real, as hard drives on a shelf cannot be trusted over long periods of inactivity. Disks also need a hardware-based server enclosure, with power and RAID (or Erasure Coding) for data safety. All of these factors mean that disk does indeed need a lot more resources to keep your data safe  - while LTO doesn’t. And all of this costs more.

I rest my case. 

CONCLUSION

I have been an ardent follower of Backblaze and their rather open policy on sharing information. Their hard drive stats hold a wealth of information about long term hard drive usage which is unavailable from hard drive vendors (here is a link to the latest one: https://www.backblaze.com/blog/hard-drive-reliability-stats-q1-2016/). Recently, they also open sourced their Erasure Coding algorithm that they use for their new cloud storage offering – B2.

As I was reading more about B2, I stumbled upon a very useful page on their website (See below).

Courtesy Backblaze.com https://www.backblaze.com/b2/cloud-storage-providers.html

The above table compares the various storage prices from cloud storage vendors. As the ongoing debate the role of cloud storage should play in the overall storage hierarchy continues, the above offers a nice cost comparison between different cloud vendors.

The first thing that jumped out at me was not just the base cost of cloud storage, but the additional cost to download. Let’s take Backblaze for example. While you will pay roughly $60,000/year to store 1 PB of data in the cloud, you will pay an additional $60,000 to download it – should you need to. This means that you are essentially buying your own data back from cloud providers. While this is not uncommon at this stage for cloud storage providers, it is something that a lot of customers do not account for in their cost calculations.

That being said, cloud storage definitely makes sense if you plan to use cloud computation services (e.g. Amazon EC2) as well, since you’re already paying for access to your content as part of the fee structure for those services.

We are at a storage crossroads. The traditional workhorse of storage - hard drives - have hit their physical capacity limits. Manufacturers are finding it difficult to pack more bits on a platter thereby losing their cost, performance and capacity edge to both SSD and LTO based storage options. Across IT departments, SSD/Flash storage is quickly replacing hard drives thus becoming the norm for high performance scenarios. On the other hand, LTO is fiercely protecting its archival role via impressive bandwidth and capacity improvements, with LTO-7 at 6 TB capacity and 300 MB/s of performance. 

In the media vertical, we believe hard drives will continue to be the “go to” storage for many workflows. However, we expect the demands of high bandwidth workflows to move toward increased usage of SSDs. LTO storage will stand strong against hard drives’ attempt to displace it across archival and nearline storage scenarios.

SSD and Flash Media in 2017

Flash provides tremendous performance and workflow benefits especially in 4K environments. We expect to see more SSD and flash deployments across removable media as well as NAS/SAN deployments. While we expect the adoption of SSDs to increase, the cost of SSD will continue to be a road-block in media workflows due to the massive quantity of data being produced. For 2017, we expect increased adoption of SSDs but do not expect a complete replacement of hard drives.

Hard Disk Drives in 2017

Hard drives will continue to fill the role of the media storage “workhorse”. However, with the introduction of massive hard drives (e.g. 12 TB), customers are increasingly nervous of data loss in both standalone drives and RAID’ed configurations. To address this, customers will potentially look to LTO storage due to its increased reliability and lower costs. Some customers are considering object storage solutions due to its ability to overcome RAID rebuild issues. However, object storage (while more cost effective than a NAS) is still many factors more expensive than LTO-based storage.

LTO in 2017

As we look back upon 2016, the year has seen many worldly events occur (Rio Olympics, Brexit, Zika Virus, Cubs win World Series, LTO-7 hits the streets – among many others!), but its not until you actually sit down and contemplate/calculate how much content you’ve got sitting on all those LTO-5 tapes do you realize that you’ve got some very important work ahead.

As the LTO roadmap moves steadily forward, your timeframe for successfully moving that content onto faster, denser storage is shrinking. With each passing day without a plan, you’ve lost valuable time in the race to complete a job that ultimately never ends – data migration.  In case you have not heard, the LTO roadmap provides 2 generations of backwards compatibility (see chart here) – its something you’ve been told I’m sure. But now’s the time to “wake up and smell the coffee” as they say.  You need a plan, but why?

1. LTO 7 - Keep your content readily available

If you don’t migrate your content forward, the chances of it not being accessible get higher as time passes and that is a scary thing to think about in today’s age of content re-purposing.  Let’s face it, all hardware is prone to failure over time - and if/when the day arrives when your LTO-5 drive no longer works and there is limited ability to source a new one - you are SOL and all the content on those LTO-5 tapes will not be easily accessible.  Sure you could go buy yourself a “brand new” LTO 6 drive, but that will only get you so far and with LTO 8 on the horizon, why buy into an intermediate band-aid?

2. LTO 7 - Consolidate content

With LTO 7 firmly established as a cost, performance and capacity leader in today’s storage arena it makes perfect sense to make the move. Doing so will allow you to consolidate all that content from LTO 5 tapes onto roughly one fourth the number of LTO 7 tapes.  Imagine the shelf space you could save – ultimately making room for much more content all being written to brand new LTO 7 tapes. You can now hold a single project’s worth of content on a single tape – much easier for organizing as well as distribution.

3. LTO 7 – Save money by taking advantage of higher speeds and greater capacity

LTO 7 capacity per cartridge is 6TB, providing highly reliable storage at just $.02 per GB (read more here), while the increase in speed of LTO 7 drives delivers 300MB/sec throughput.  These numbers are hard to ignore – even before that cup of coffee.  Not only are you saving money on the actual storage, you will save valuable time (remember time=money) by being able to complete jobs in half the time, or maybe a better way to see it - double your productivity!

Seagate recently announced a 10 TB hard drive. That is huge capacity in a single device! At the same time, a number of NAS vendors have announced support for these drives, which means that 8 - 10 HDDs can be configured in a NAS scenario and you now have a whopping 80-100 TB of capacity.

Lacie 12Big Thunderbolt (120 TB max capacity)

A great example of this type of solution is Lacie, offering the 12 Big Thunderbolt with 10 TB drives (shown above). This provides you 120 TB of capacity at $13,999.

So, the question lots of people have started asking is “Do I really need LTO? With such vast amounts of capacity on Hard Drives, why do I need to keep my content on LTO tape?"

Well, the short answer is a resounding "YES! MORE THAN EVER!"

Here is why you need LTO:

Reason 1: A hard drive failure has higher costs than ever before

When you store your data on a large stand-alone hard drive (e.g. 10 TB HDD), a hard drive failure will cost you more in terms of lost hours of content. For example, a loss of a 10 TB drive results in a loss of 12.1* hours of Prores 4K, 120* hours of DNxHD-220, 1.68* hours of RED-5K-10bitRGB. Think about the time and money it took to shoot that content – or even worse if that content was fully edited into a final program and was lost!

This can be catastrophic and in order to be prepared in the event of a failure, users are currently making dual copies of their content in order to be safe, which is far more costly than having a secondary copy on LTO.

*Computed from AJA-Data Calc.

Reason 2: RAID technology is less secure

While a 10 TB hard drive failure in a RAID’ed configuration will not result in instantaneous data loss, it does put all of your content at greater risk.

Imagine the following:

You just experienced a hard drive failure in your RAID. Now, instead of a few hours, it is going to take you a few days to rebuild that 10 TB hard drive. During that long rebuild time, you run the risk of losing a second drive. A second drive failure will result in complete data loss across the entire RAID. So rather than 10 TB lost you will lose multiples of TBs. So, as you can see, even in RAID scenario, large hard drives are scary.

Reason 3: LTO is far more reliable and cost effective

Over the past several years, there has been a trend in the professional media marketplace which has resulted in LTO tape usage being relegated to purely Archival and Disaster Recovery duties – while shifting daily backup duties to inexpensive spinning disk solutions.  A big reason for this is that media entities are being inundated by more and bigger content and LTO along with traditional HSM architectures have been unable to keep pace with this deluge of content.

Some food for thought when considering spinning disk storage as your back up:

Consider that NAS vendors are making low cost systems by packing cheap, high-density disks in them. These drives are 8TB - 10TB, and are configured in various RAID profiles. Now consider what happens when a single 10TB hard drive fails. The RAID rebuild time will run for many hours, possibly days, because the drives are so large. As the RAID rebuild is occurring, you keep your fingers crossed and hope that another drive doesn’t have an issue. The RAID controller is busy rebuilding while you are vulnerable to a complete data loss scenario.

It takes only one of these incidents to realize that while cheap disk storage might seem like a great working copy, it cannot replace the safety of an LTO copy.

Now consider another point - LTO-7 can push 300 MBs on a single drive. If I used multiple LTO-7 drives, my restore time is many times faster than a RAID rebuild.

We believe LTO-7 (especially multiple LTO-7 drives) with a small fast online storage tier (e.g. SSD) is going to present a new alternative to HDD based systems. It enables a safer, faster, more scalable, and a more cost effective storage option.

IBC 2016 once again did not disappoint on the storage front. There was a large presence of storage vendors at the show. Here is what we saw and found interesting.

 Shared Storage (NAS, SAN) Democratization

NAS shared storage is seeing a massive democratization. What used to be an expensive option accessible only for larger facilities can now be afforded by smaller Post and individuals as well. This is largely rooted in the stabilization of open source network filesystem and storage software offerings. Both Linux and FreeBSD offer high performance and stable NFS/SMB implementations. Customized packages such as FreeNAS are also great alternatives. Vendors who are leading the charge on bringing shared storage to smaller customers include the following:

o   Avid with the Nexis Pro

o   Studio Network Solutions with Prodigy

o   Facilis

o   GTechnology Rack 12

o   Promax

SSD (Solid State Disk)

SSDs are taking the IT world by storm. In media, while a few vendors have begun to support SSD, there is not a massive push of SSDs into storage systems and workflows. This could be on account of the fact that the primary motivation for SSD adoption in IT are IOPS hungry applications (virtual machines, databases etc.). On the other hand, media applications tend to be bandwidth hungry. Paired with costs higher than HDDs, the adoption of SSDs seems to be slower in the media vertical.

That said, we believe SSDs will eventually replace HDDs as their costs drop. We are already seeing interest in SSDs in more demanding workflows, such as 4K+ content, animation rendering, etc.

A few vendors who had SSD offerings at IBC included:

o   Quantum

o   Facilis

o   DDP

Nearline and Object

Nearline storage is seeing growing relevance in media due to the massive amounts of data that is being created. As of recent, Object storage vendors are also making a push into media as a smarter form of Nearline storage.

So what is the difference between Nearline and Object storage?

Nearline storage is a denser, more cost effective version of primary NAS storage. They are Ethernet based, support NFS/CIFS file-systems, pack a larger number of Nearline hard drives in the same chassis and thus are often more cost effective.

Object storage, on the other hand, works differently even though the underlying hardware is more or less the same. They are designed primarily as a cloud storage platform (primary access method being HTTP e.g. S3), employ data striping (like Reed Solomon Encoding) rather than RAID, and therefore tend to be slower but more resilient than Nearline storage systems. It is worth noting that numerous vendors who claim object storage offerings only implement a subset of the above mentioned attributes.

Some of the notable vendors demonstrating at IBC were:

o   IBM Object Storage

o   Cleversafe

o   Data Direct Networks WOS

o   Object Matrix

o   Netapp

o   Isilon/EMC

o   Studio Network Solutions

Cloud

The cloud continues to play an interesting role in media. Cloud adoption is still slow given the hurdles of connectivity, cost and security. However, adoption is steadily increasing every year. especially for smaller amounts of data.

Cloud vendors with products on display at IBC 2016 were:

o   IBM Cloud

o   Amazon S3

o   Oracle

o   Microsoft

LTO

LTO continues to play a major and growing role in media storage. With LTO-7 capacities at a whopping 6TB raw and speeds of 300MB/s, it is dwarfing many primary NAS and SAN systems. Most archival vendors have rushed in to support LTO-7. Some of the vendors who were displaying LTO-7 products at IBC were:

o   Quantum

o   Spectra Logic

o   IBM Spectrum Storage

o   Oracle

o   StorageDNA

o   Archiware

o   SGL-UK

o   Xendata

Optical

Optical media from Sony (Sony Optical Disc Archive) presents an alternative, long term storage medium. However, the most unique aspect of optical media is its ability to also serve as a direct access medium like a Hard Drive. Sony’s Optical Disc Archive offers the possibility of serving as part deep archive and part Nearline. We especially see Sony Optical Disc Archive Generation 2, which has been announced at 3.3 TB raw capacity and up to 2 GB/s of read throughput, as a viable storage medium for media. Vendors who were displaying integrations with this technology at IBC included:

o   Sony

o   Object Matrix

o   StorageDNA

Tape long held the top spot for archiving until over time it was replaced by disk for daily backups in the world of IT. Today, tape becomes relevant again with LTO-7.

Why Disk Replaced Tape for Backups

Disk gradually started replacing tape for backups about 10-15 years ago. This happened due to a combination of reasons.

LTO was slow: At that time, LTO was at a generation where tape performance was very slow. However, disks in RAID configuration were providing a strong combination of random IO and bandwidth.

Distributed Backup: Distributed backup was popular where organizations were attempting to backup thousands of laptops, desktops, servers and these backups were required to complete in 24-hour periods. LTO based systems, which were serial access struggled to keep up with a massive number of parallel backup streams. Once again, disk arrays (being random access) were able to cope with this far better.

LTO was problematic: LTO was at its infancy at that time. This resulted in environments where backups often failed due to tape or robotic failures. This gave LTO a “bad rap”. Disk, on the other hand, faired far better than LTO with fewer moving parts.

Why Tape is Becoming Relevant for Backup

In our last blog post we highlighted how LTO tape with LTO generation 7 has reinvented itself. These factors along with a change in how data is generated and stored makes tape highly relevant for a number of backup scenarios. Key factors driving LTO relevance in backup are:

Centralized Primary Storage: Most organizations have migrated away from local storage to centralized NAS and SAN storage. Especially in media, users now have a choice of SAN and NAS storage platforms at price points starting as low as $10,000 USD. The distributed backup model which is problematic for tape is now being replaced by centralized storage, which is a better fit for LTO tape (esp. LTO-7 with 300 MB/s) as data can be backed up in a single large stream.

Big Data: The world is creating a lot more data. This data is largely unstructured and while organizations need to keep it around, it does not inherently justify being stored on expensive disk based systems. While vendors argue the benefit of disk based systems (e.g. object storage) for big data archival, it is still 4x-5x more expensive than LTO-based library systems. With LTO-7’s performance, it is more conceivable to store this big data on tape (due to cost efficiencies) and quickly retrieve data when needed.

Faster, Denser LTO tapes: What was considered a slow storage medium (e.g. LTO-2) has matured into a high performance storage platform with LTO-7. LTO-7 drives can keep up and are often faster than primary SAN and NAS systems. Additionally, LTO-7 now packs a density of 6TB/cartridge allowing densities of 500 TB in a 5U rack space (e.g. DNAevolution Infinity 80 slot automation library). These improvements allow for LTO-7 to keep pace with nightly backups of multi-TB primary NAS and SAN environments.

Conclusion

More data often means adding more disk to the primary storage environment, which means adding more tape or secondary disk storage to the backup environment.  It seems like storing and protecting business-critical data should be easier in this day and age.

Tape is a tried and true solution to a new, emerging challenge.  While there are several misconceptions that it is slow and unreliable, LTO technology is reinventing tape usage with new features that you might not know about, as featured in a recent ESG Lab Review Report*. 

3 NEW THINGS YOU NEED TO KNOW ABOUT TAPE

1. Tape is really fast.

When you compare the raw speed data of tape with the raw speed of disk, tape is actually much faster.  Often, the disk transfer times that you are most familiar with do not factor in the complicated backup processes that are associated with disks.  Tape has a relatively simple process for writing data and because of that, reduces transfer time.

LTO-7 Ultrium high-capacity magnetic tape storage technology is capable of storing 15TB of data on a single cartridge with up to 750MB/s transfer rate[1].  Each generation of LTO performs significantly better than the last in terms of performance and capacity.  Even earlier versions (LTO-3) are faster than a typical 7,200 RPM disk drive (130 MB/s).

2. Tape is quite versatile.

When you realize that LTO is mapped out with defined capacity and capabilities all the way to LTO-10, it is obvious that tape is not a lost solution.  

New improvements are being made on a continual basis.  For example, the introduction of the Linear Tape File System (LTFS) in LTO-5 used partitioning for easy data retrieval by dividing the tape into separate writable areas.  LTFS works with the LTO and media technology to make the device look like a typical storage device to the underlying operating system.  Once it is setup, files can be simply copied to and from the device just like any disk drive.  With expanded functionality, organizations are using LTO for both tertiary and even primary storage.

As always, tape is an inexpensive, portable storage solution that plays nicely with other storage solutions like disk and cloud storage.

3. Tape is very reliable.

Reliability in tape-based data storage solutions is all about accessing data when you need it. LTO media was designed to be loaded into a device, written to, removed, moved, stored, recalled and reloaded into a device whenever data is needed.

LTO media reliability, introduced in the LTO-3 format design and continuously improved since, includes some innovative features:

Data is spread across multiple channels to protect against single-head element or media defects

Error detection/correction code (ECC) is applied at two levels. 

Level one- ECC rewrites the data to another location further down the original track.

Level two- ECC rewrites the data across multiple tracks and allows for data recovery in case there is a bad track.

Throughout, read-while-write verification is used to check the validity of each data set and is automatically rewritten if an error is detected.

As a result of this process, it would take 130 tape drives writing data continually for one year to encounter an error that could not be fixed by ECC. That means you are more likely to get hit by lighting, get killed by a shark or win a multi-million dollar lottery, than encounter an uncorrectable error in your enterprise tape environment.  Pretty good odds!

All of this sounds good, right?  Check back next month to learn about LTO in modern use at the ESG Lab.  These case studies will show you how tape technology is improving and expanding our data storage capabilities and why you should reconsider how you think about tape.

*Assumes 2.5:1 data compression

Hard disk drives (HDD) have long served as the backbone for Media workflows. From USB Thunderbolt HDDs in the field to being part of high performance SANs, HDDs have served as the go-to storage medium for media workflows.

However, this is about to change.

SSDs Taking the Place of HDDs in Primary SAN/NAS

Image: SSDs are replacing HDDs in modern high performance NAS and SAN

This change is caused by a multitude of reasons. For example, one of the primary reasons is that Solid State Disks (SSDs) have become far less expensive. While SSDs have always offered tremendous performance benefits over HDDs, they have traditionally not been cost effective, especially given the data volumes in Media pipelines. Now with recent cost reductions, SSDs are a viable alternative in primary shared storage NAS and SAN arrays. It is extremely plausible that SSDs will completely replace HDDs in Media NAS/SAN storage array in the next few years.

HDDs Areal Density Hitting Limits

Image: Traditional (LMR) and Current (PMR) recording techniques (source link)

While SSDs have always been faster than HDDs, HDD maintained a cost v/s performance edge. This edge was maintained by constantly improving HDD recording technology - Perpendicular Magnetic Recording (PMR). In recent news, its’s been cited that PMR recording technology is reaching its limits (Read here: http://www.computerweekly.com/Is-hard-disk-drive-areal-density-fatally-limited). The result is that we will not see HDD keep up with emerging storage alternatives - namely SSDs.

Can HDDs Replace LTO for Archiving?

It has become a popular belief that even though HDDs are losing ground to SSDs in primary storage, they can be used for Nearline and replace technologies such as LTO for archiving. This is fueled by the fact that HDD are random access versus linear access technologies such as LTO.

This opinion is largely fueled by the usage of HDDs for very large-scale storage needs by companies such as Google, Facebook, Amazon etc. It is often argued that if these data giants chose to adopt HDDs over technologies such as LTO and Optical for their long-term data storage needs, then it is the obvious direction forward.

This reasoning is too simplistic in comparison for the media user. These large organizations are providing cloud-based storage at peta-scale volumes.  They have added requirements like “instant access” for users who might need to recall a piece of media at any time, but these storage systems come at the expense of architectures that make sense only at very large data volumes and that often require very experienced IT resources, as these system are designed to run on commodity hardware.

By contrast, media companies create massive amounts of data locally and in remote locations with little IT expertise. This media is often copied to a field storage alternative (e.g. HDD, LTO), which in turn becomes the de-facto archive. Hence, in media workflows, it is difficult to imagine a time where all data will be stored in the cloud or a private cloud powered by a software driven, HDD based archive. A number of technology, workflow and mindset hurdles will need to be traversed before this can be achieved.

It is more conceivable in media that HDDs will be used as Nearline for offload, ingest and transcode purposes – while LTO and other optical media will be used for archiving.

Strong LTO and Optical Roadmaps

Image: Courtesy of the LTO consortium

Another challenge faced by HDDs is the strong roadmap presented by existing archive technologies including LTO and Optical (e.g. Sony Optical Disc Archive). Both of these technologies present a considerable threat to HDDs in both archive and Nearline workflows.

LTO technology in its latest avatar (LTO-7) is offering a 6 TB capacity with 300 MB/s of throughput at less than 2c/GB. Additionally, LTO has a strong roadmap up-to LTO-10, which is predicted at 48 TB and 1100 MB/s. This makes LTO-7 a strong contender for workflows beyond archiving, including offload and ingest.

Image: Courtesy Sony, US

Sony has introduced Gen 2 of its Optical Disc Archive technology with a capacity of 3.3 TB and read performance of 2 Gb/s. Optical media is also random access enabling faster responses.

Both LTO and Sony Optical Disk Archive technology are not only securing its position in archiving but also challenging HDD in Nearline and field workflows.

Conclusion

If I were to summarize the state of storage in Media from this year’s NAB show, it would be "Faster, Cheaper, Denser"

We are all aware that the usage patterns of the media industry vary vastly from that of IT. While the hottest storage trend in IT is SSD replacing traditional HDD storage arrays, the media verticals’ needs are different. While IT has an IOPS problem (due to virtual machine based workloads), media has a streaming bandwidth and capacity challenges. Read the difference between IOPS and BW here. 

Media companies are faced with overwhelming streams of data that need to be offloaded, edited, archived and re-purposed. Customers are in turn looking for storage solutions that will help them manage these vast quantities of data. 

Here are some of the key storage highlights from NAB:

Lower Cost Shared Storage 

Image: GTechnology GRack-12 

At one time, a shared storage solution was an expensive proposition that only a few organizations could afford. However, open source technologies like NFS, SMB, ZFS, BTRFS etc. have reached production grade capability. This has resulted in a slew of new shared storage offerings from the likes of Studio Network Solutions and Promax. To combat these, Avid has also released the Avid ISIS 1000. One of the most interesting announcements at NAB was that from GTechnology with the GRack 12 NAS. A NAS offering from GTechnology based on NFS/SMB and the open source BTRFS file system is a clear sign that shared storage is becoming a commodity offering. This is good news for numerous small and mid-sized productions that want the added efficiency of a shared storage solution at a highly cost effective price. 

Denser, Faster Field Storage

Image: Lacie 12Big

High resolution, HDR, high FPS camera recording with multi-camera setups (Virtual Reality, Reality TV) is now typical for even the smallest productions. For example, VR setups (e.g. GoPro Omni) records 6 camera streams simultaneously. Most productions today are generating a phenomenal amount of data. Offloading this data in the field is becoming a major pain point. Vendors such as Lacie and GTechnology have created faster and smarter field storage to address the problem of fast offloads. Lacie offers the 12Big Thunderbolt with 96 TB of capacity and 2600 MB/s of performance. GTechnology also offers the GSpeed ShuttleXL with up to 1200 MB/s of performance 48TB of performance. These solutions provide the bandwidth needed to offload multiple high-resolution camera streams in the field or in post.

Software Defined Storage

Image: Avid Nexis Software Defined Storage Platform 

Software-defined storage (SDS) is a relatively new technology that has emerged from the world of IT. SDS can be considered the major reason behind why the big storage vendors (e.g. HP, IBM, EMC) are losing revenue year after year (more here). This is because most IT workloads now run on Virtual Machines (VM). VM workloads are essentially comprised of multiple operating systems running on the same machine. This can be disastrous for traditional storage arrays with legacy controller architectures. Instead, VMs work better with localized SSD storage with local interconnects like PCIE. SDS builds on this concept by allowing VMs to use local SSD storage while clustering multiple commodity servers together to present a global file system.

However, we are seeing SDS based terminology now being applied to offerings in the media vertical. Avid released Avid Nexis this NAB 2016 and called it the "world’s first software-defined storage platform that enables true storage virtualization for any media application" (Quoted from http://www.avid.com/products/avid-nexis). However, I have to be honest that I am not really sure how the term "software-defined" applies to the Avid Nexis, given that it is built on closed hardware and a traditional controller-based architecture. On the other hand, Isilon also announced the Isilon SD Edge which is more of a true "software-defined" platform (https://www.emc.com/en-us/storage/isilon/isilonsd/edge.htm). 

Object Storage

Object storage from the likes of Quantum, Cleversafe, DDN were all present at NAB 2016. While Object Storage is not new to the media vertical, it is finding a niche. With the growing amounts of camera content, distributed/cloud workflows, and re-purposing requirements, Object Storage is starting to replace some Nearline NAS based deployments.

Smarter, Faster Long Term Storage 

Image: LTO-7 tape being 6 TB capacity @ 300 MB/s performance

 Image: Sony Optical Disc Archive Gen 2 @ 3.3 TB and 2Gb/s Read with 1 Gb/s Write 

Long-term storage is also making amazing leaps. Announced shortly before NAB, LTO-7 now delivers over 6TB capacity and a whopping 300 MB/s - that is 2.4 Gb/s of throughput from a single piece of media, and it is delivering this performance at $137/cartridge (*pricing from https://www.tapeandmedia.com/lto-7-tape-media-tapes.asp). It is surprising to see that tape technology (not HDDs, not SSDs) offers the best combination of cost, reliability and performance. With the amount of 4K, high frame rate and VR based workflows, LTO-7 will play a major role in both field, post and archival workflows in the near future.

Sony also announced Generation 2 of their Optical Disc Archive technology. Optical Disc Archive is positioned as a long-term archive technology (similar to LTO), but is based on optical media rather than tape media. Since it based on optical media, it has random access advantages. With Gen 2 delivering 2 Gb/s read performance and 1 Gb/s write performance and 3.3TB raw capacity, Sony Optical Disc Archive has the advantage of serving as a long-term archive medium as well as a directly accessible, hard drive like storage platform.

Conclusion

So I have seen quite a few posts about tape versus cloud, with many debating the pros and cons of both. But like everything that is considered black or white, the answer is gray.

The best way for me to explain is to look at my own personal data/content management situation.  

Like everyone else these days, I am an amateur photographer and videographer. I utilize Dropbox for my MyDocuments folder because it is easy, efficient and I don't spend an arm and a leg on the monthly fee and when I want something back, it’s reasonably fast to download a document.

BUT, once I tried to add my photos and videos to Dropbox....

Well, the situation deteriorated pretty quickly. The upload time was painfully slow, the cost per month was more than I was willing to pay, and downloading content took much longer than expected.  So my solution is to keep my videos and photos on a removable HDD (yes, I should use something safer!).

The logic is similar to why businesses choose cloud versus an onsite storage like tape.

At first glance, cloud storage provides:

Upload speeds are able to keep up with the daily quantity of data generated

Cloud storage costs are acceptable

Wait times for files to download are acceptable

There is a general trust in the cloud for keeping files secured

As a result, the cloud does work for a lot of businesses - especially small/medium businesses that generate word docs, excel sheets, and smaller databases.

Let’s look at a vertical where the cloud might not make sense; businesses dealing with rich media.  Even if a customer got over the hurdle of "security" (which is a very, very big hurdle), the following are the reasons cloud would not work:

It would take days to upload content

Cost is substantially more than onsite storage when it comes to larger data management

Time is an issue when data needs to be retrieved in a short timeframe.

Clearly, the cloud is not a one size solution when it comes to large amounts of content management

Conclusion

Which is better: cloud storage or tape?

The answer comes down to the use case and more specifically, individual business needs around:

Upload times

Access times

Cost per month

Security

In a world of trends, there are still several classics that have not gone out of style.  It is easy to think of these things in terms of what you wear or the music you listen to, but what about in the IT world?

Magnetic tape is the oldest storage medium still in use.  It dates back to 1928 when it was used for sound recording and has stood up to recent competitors like HDDs and SSDs.  Benefits of performance, durability, speed and low procurement cost are always current.

PERFORMANCE: Tape has a much longer useful life and is less prone to the risks of modern disk drives.  It can be read securely for 30 years while the average HDD lasts about 5 years.

DURABILITY: Data on magnetic tapes rarely experiences physical problems like a broken or damaged HDD or software failures and interrupted software updates that can happen with cloud computing, ultimately leading to a loss of saved data.

SPEED: Extracting data from a magnetic tape can be four times faster than reading an HDD. 

LOW COST: As more storage capacity is needed, acquiring several new tapes is cheaper than buying equivalent HDDs storage space. 

Keeping a product classic is about maintaining its core while modernizing its use.  

Case in point, businesses are taking a hard look at backup and archive systems and how each should be used differently to achieve specific goals.

Backup systems are intended for a hardware or system failure.  They protect against data loss and can a get an environment running again quickly and on short notice.  The focus is on speed and on data integrity

Archives are designed to store data for the long term without changing them.  Speed is less important while durability is critical.

Many companies have used tape as their main backup system without a plan to archive data that may need to be retrieved in its original format for legal or audit requirements. As a result, backup tapes end up being stored for more than 20 years in company facilities or off-site storage providers without a realistic way to use or reinstate that data.  Meanwhile, archive data is not appropriately managed.

Things are changing with modern archive solutions improving and becoming more cost-efficient. Magnetic tape-based archives can be integrated with document management systems (DMS) to define the lifecycle of a document.  For example, all associated documents and emails from a project can be archived and their retention periods can be defined to fit the project timeline.  The most important data of the project can be stored on an HDD-based DMS for store-term accessibility and use.  This approach to data management appropriately achieves dual business purposes by combining features.

In the previous write-ups, we took a deeper look at NAS and SAN storage systems. In this final article, we will take a look at Object Storage systems and how they fit into M&E workflows.

Object Storage – Refresher

As described in the first part of this write-up, object storage is a new breed of storage systems that can be best described by the following key attributes:

Designed for remote/cloud access.

Presents an HTTP based store and access interface.

Normally, does not provide traditional file-system hierarchies.

Enables multi-site storage capability for objects.

Object Storage – Where it fits

Organizations needing a storage solution that can enable globally distributed teams to work on the same content

Given that object storage was envisioned with an HTTP-based interface and is powered by globally replicating data algorithms, it is ideally suited for organizations that want to enable teams that are geographically distributed. However, any organization adopting an object storage system must account for interface challenges where applications that traditionally wrote to file-systems must now be able to write to an object interface.

Organizations requiring a private cloud

Often times it may be necessary to set up a private cloud for content distribution either within or external to an organization. Object storage serves this need extremely well, due to its ability to serve data over HTTP and its ability to globally load balance and distribute content.

Object Storage – Myths

Object storage is more resilient than non-object storage systems

Object storage vendors often utilize algorithms (e.g. Erasure Coding) instead of traditional RAID-based techniques. This is done to overcome the long re-build times of larger drives during which another drive failure can result in permanent loss of data. While erasure coding is considered the way forward, it is not unique to object storage systems. Erasure coding is used by numerous modern storage systems that are not necessarily classified as object storage (e.g. Nutanix).

Object storage is a good replacement for traditional NAS and SAN systems in a local environment

As described earlier, object storage was designed with an HTTP interface for remote access and globally distributed users. However, its key features are not particularly advantageous in a local shared environment. In a local environment, NAS and SAN solutions offer a better combination of performance and open access.

Object storage can be your new archive

An archive solution must meet certain key characteristics. The two most important ones are:

(1) the ability to externalize – so the media can be safely kept on a shelf for a long period of time

(2) the media should be self-describing, while allowing for content retrieval in an open and easy manner. Object storage systems are built on hard drives with complex data-spreading algorithms. It is a fairly common understanding that hard drives are not designed to sit on a shelf for 30+ years. Secondly, it is not possible to retrieve data from a single hard drive that has been extracted from an object storage solution.        

Conclusion

Although object storage is now a key piece of broader offerings of many storage vendors, it is best suited for enabling a cloud based or private hosted storage platform. Object storage is the primary contender when an organization is looking to build a globally accessible storage solution. On the other hand, it is also important to keep in mind that in most M&E use cases, traditional NAS or SAN storage may fit the need better than deploying an object storage solution.

In the previous write-up, we analyzed the beginnings of Object Storage. In this second part, we will compare object storage to current NAS and SAN based systems.

What is a NAS or SAN?

Lets first start by clarifying what we mean by a NAS or SAN.

NAS (Network Attached Storage): NAS systems are largely defined by their ability to export a shared, network file-system interface via NFS or CIFS. Since the early popularization of NAS systems from Netapp, network based file-systems are defined by their ease of access for multiple client systems to a common pool of storage. NAS systems have evolved as clustered NAS systems (e.g. Isilon) to combat the single server bottleneck. Clustered NAS systems utilize concepts such as load balancing, data spreading etc. to achieve scalability and bandwidth. It is worth noting that clustered NAS systems employ many of the same techniques highlighted in object storage systems. However, file-system access via NFS/CIFS is the major distinguishing factor of a NAS storage system compared to object storage and SAN systems.

SAN (Storage Area Network): A SAN refers to a shared storage fabric versus a direct attached storage connection e.g. SCSI. The early SAN fabric (e.g. FC) allows multiple computers to share a block storage device via a shared storage protocol. To manage this shared connection and preventing multiple clients from over-writing each other’s file-system, the idea of a SAN file-system FS was created.

In a SAN file-system, the client is loaded with a specialized file-system software that is responsible for presenting the local file-system and translating file-system requests for data over the shared storage fabric. As described, this shared fabric (or SAN) can be over FC, SAS or iSCSI. The major difference between a SAN file-system and NAS is the use of a NFS/CIFS based file-system protocol versus a proprietary file-system client speaking a block protocol.

It must be noted that this line between NAS and SAN systems is getting more blurred. Take for example; the Avid ISIS shared storage system. While Avid ISIS behaves like a SAN in some ways (since it has a proprietary file-system client), it performs its storage operations over Ethernet. This makes it a hybrid. On the other hand Quantum’s Stornext storage system is a classic example of a SAN. However, it can now share its file-system over an NFS/CIFS protocol – thus making it a NAS as well. Hence it is worth noting that while the difference between a NAS and SAN was clear in the early days, that line continues to blur.

 Beyond NAS and SAN storage

NAS and SAN based storage systems were architected to address key storage access and performance challenges for specific needs e.g. ease of content sharing, improved bandwidth etc.

NAS systems were designed to enable easy and rapid sharing of content across an organization via simple “Connect to server” functionality built into Windows and Mac clients.

SAN systems on the other hand were designed to overcome bandwidth performance bottlenecks of NAS system. Since NAS systems were built on Ethernet, it often times became a bottleneck especially in media centric workflows that demanded high performance streaming, play and editing. To combat this, SAN systems (built on FC) were designed to deliver predictable throughput.

NAS and SAN systems evolved to address key performance and sharing requirements largely around local shared access and streaming performance. However teams have become more globally distributed and the need for a storage that can scale beyond a local network is emerging. Object storage was created to fill this gap.

Given the proliferation of global teams, object storage presents a strong case as a storage system that can unlock remote or cloud-hosted workflows. Beyond remote pipelines, object storage also presents a distributed storage that is better suited for disaster tolerance.

Conclusion

We evaluated the origins of NAS, SAN and Object storage. In the next article, we will evaluate key areas where object storage can play a role in the media and entertainment eco-system. Additionally, we evaluate where object storage may not offer a considerable advantage over currently available NAS and SAN systems.

DISCLAIMER: StorageDNA is not an object storage vendor and does not promote any object storage system.

Recently, we have noticed some buzz around object storage. There are many discussions about object storage and our customers have enquired about the same. They want to know where it fits in their workflows, its key advantages and disadvantages. In this multi-part write-up, we hope to share and discuss object storage and its place in our storage hungry media world. We will cover the following topics:

Object Storage: What is it and its beginnings? 

Object Storage versus current NAS and SANs

Where Object Storage fits media workflows and archival

What is Object Storage? 

Object storage is a newer breed of storage systems that is positioned as a scalable storage tier designed for globally distributed and cloud based workflows.

Now lets understand what that means.

To understand object storage better, lets start with one of the earliest object storage systems - Amazon’s S3. It was envisioned and designed to serve as a cloud-hosted, scalable, bit bucket for users that want to offload large amounts of unstructured data (documents, images, videos etc.) to the cloud.

To enable this, storage providers such as Amazon had to support remote store and access operations for their users. Since LAN/local network based file sharing protocols (e.g. NFS/CIFS/Samba) were never designed for remote client-server communication, it would be inefficient to access cloud storage via NFS/CIFS based protocols. As a result, object storage was designed to present an HTTP based interface (e.g. Amazon S3 or Swift). This is primarily because HTTP was designed for remote web access and thus more appropriate for remote store and get operations. This interface is one of the key differentiating factors of object storage compared to NAS and other local file-system based storage systems.

Object storage was also envisioned as a storage target for applications rather than end users. As a result, object storage systems present a far simpler form of data organization. Unlike a file-system that supports hierarchies, object storage simply provides buckets and objects. The onus of maintaining hierarchies falls upon the application. The lack of folder structures is not viewed as a major drawback as most applications can manage their own hierarchy based on their specific needs.

Finally, since most object storage systems are cloud hosted, there is a major focus on multi-site disaster recovery. Most object storage systems move away from concepts like RAID and instead provide the ability to store an object (or file) in multiple sites and locations. This provides a greater degree of redundancy and reliability than traditional RAID based systems. This ability to globally distribute data also aligns well to remote collaboration based workflows, where data is geo-located closer to the point of access.

In our opinion, the above features serve as the most distinguishing factors of object storage. To summarize:

Object storage has been designed for remote/cloud access.

Object storage presents an HTTP based store and access interface.

Object storage does not provide traditional file-system hierarchies. 

Object storage provides multi-site storage capability for objects. 

Conclusion

It can be seen that the origins of object storage were as cloud-based, scalable storage systems. However, since the launch of the early object storage systems (e.g. Amazon S3), object storage systems have morphed to also support NFS/CIFS (largely because this widens the market for object storage vendors). This has positioned object storage for not just cloud use but also local access.