Thursday, March 10, 2011

Does Linear Tape’s (LTO/LTFS) improved Accessibility, Throughput, Capacity and Greenness Challenge Disk as the Optimal Storage for Rich Digital Media?

In April of last year the LTO consortium (IBM, HP, and Quantum) introduced the latest version of LTO. At the time I was myopically focused on data tiering technologies which caused me to miss what was probably the most significant advance in tape technology for many years. Not appropriate for a StorageTek alumni!

Although impressive, the significance of this announcement was not the capacity and throughput improvements but a tape file system that enables a disk like search functionality (random-like) that dramatically improves data-on-tape access times, simplifies on-tape file updates and includes a drag and drop functionality that simplifies on-tape data management. When this functionality is combined with a 3TB cartridge capacity, a 240MB/s transfer rate (2:1 compression rate), a media reliability that exceed that of disk and the natural greenness of a removable media architecture, it is not surprising that the latest generation of LTO tape is being viewed as a legitimate storage option for unstructured big data including rich digital media.

The perennial negatives of tape have been its slow sequential search characteristics that bloated data access times and the cumbersome necessity to scan the whole tape simply to know what it contains. LTFS (Linear Tape file System) is an innovative tape file system with the notable characteristics of being open source and application independent. The file system partitions the tape into two distinct, individually addressable, unequal segments with the smaller quick read segment containing descriptive metadata that enables the quick search capability (random-like) of the data contained in the second and much larger segment. LTFS minimizes the traditional search performance negatives associated with tape and while the performance will never approximate that of disk, LTO-5 tape with LTFS will deliver a data access response time that should satisfy the demands of many if not most applications in the big data world.

The key factors that separates LTFS from past attempts to solve these data-on-tape accessibility and management issues are as follows;

1. It is not a proprietary solution; it is open sourced with interoperable versions available from multiple vendors. This flexibility will simplify and enhance data mobility.

2. LTO-5 cartridge is self contained and potentially self-describing, hosting both the file system and the data. This separation of the file system from the application addresses the concern “after 30 years will the application still be available to access the data?” A reasonable question but with the separation of the file system from the application and the file system being open source accessibility over time to the data on the cartridge will be greatly increased.

With a native capacity of 1.5TB, a transfer rate of 140MB/sec (3TB and 240MB/s with a 2:1 compression), a reliability (BER) that exceeds spinning disk media and an energy efficiency profile that can only be generated by removable media, LTO-5/LTFS based tape solutions have a functional profile that uniquely positions them as long term repositories for data intense, rich media applications.

Bottom-line: The assault on tape by disk based solutions that integrate data de-duplication lose their impact in non-dedupe data environments such as commercial video, surveillance video, digital photography, PDF’s, images, seismic/scientific etc. These data types contain little if any duplicate data other than where files have been duplicated. In fact using a de-duplication solution in these data environments is likely to have the unintended and undesirable consequence of driving storage costs in the wrong direction, up.

Tape has a $/GB, kW/hr/GB, sustained GB/sec advantage over tier 3 disk and now with its random-like accessibility and drag and drop manageability, tape has an answer to disk based solutions, at least within use case boundaries defined by rich media.

With the introduction of LTO-5 and LTFS, tape has an opportunity to regain its position as a legitimate participant in the enterprise storage hierarchy. However, the challenge for the LTO consortium is to build a comprehensive ecosystem of back-up, archival and rich media management applications compatible with LTFS. Not a show stopper but a significant bump in the road leading to a broad based adoption of this technology.


Anonymous said...

We may be hopping over the bump sooner than some expect. Companies like Cache-A, StorageDNA, For-A, SGL and several others already announced/delivered solutions with LTO-5/LTFS tape support.

Because LTFS IS a file system, files may be copied to/from tape simply by scripts. Any application can read/write files directly to tape. If API-integration is desired then think of LTO/LTFS as if you write files to disk. Use the same POSIX interface, with fopen(), fseek(), fread(), fclose(), etc. No need for any SCSI or special tape-commands. In fact, your application would not know if it writes to tape or to disk (other than some occasiional long seek times). Hence it is relatively easy to convert an existing non-LTFS tape archive system to work LTFS tapes.

BillM said...

Anonymous .. many thanks. It is indeed encouraging to see the ecosystem growing. Looking forward to seeing some of the more famliar industry names lining up in support.

storage.architect said...

Bill to your point about the growing ecosystem around LTFS, I thought I'd point you to

Oracle now claims they are supporting LTFS with a single software stack for HP LTO-5, IBM LTO-5 and the T10K-C.

Anonymous said...

I've been following the wave around LTFS for some time. In fact, my company has evaluated and investigated just about every solution that's currently available.

We found that most of these work "ok", however, with the exception of one vendor, none were enterprise-ready or had very good performance. The upside was that most LTFS solutions we looked at were compatible with each other. You might not always be able to write back to an LTFS formated tape, however, we could read the data in almost every case.

The only solution we found that could provide the enterprise-ready requirements was Crossroads StrongBox. Our research showed that they are by far, the most advanced LTFS management solution out there. Also, it's fully multi-tenant and provides exceptionally good performance ... especially in read and file access.

For a company that is not that well-known, it's hard to believe how far advanced they are over IBM, HP, and others.

Hope this helps ...

Jeff Parker said...

Backup tapes are the most famous in storage data like LTO Tapes that gives writable and re-writable technology and the popularity of the tapes are going well day by day with more secure data transfer and data storage analysis.