A new optical disk technology offers a fundamental new capability -- which is storing offline archives in a format with a shelf life of many decades (or even centuries). The key is in the pits: unlike commonly available dye-based CD-R and DVD-R media, the Millenniata writer actually laser etches physical pits into the writable layer of its "M-Disc" DVD-ROMs. Because the pits are physical structures, like the pits on pressed media, they have the same kind of shelf-life -- but in a way that is economical for low-copy archives. The niche here is for digital archives of "time capsule" data: family photographs, historical records, original manuscripts, video footage and masters, and so on. Perhaps more remarkably, the drives and disks, are affordable enough for the target applications and available commercially right now.

One of the deepest embarrassments of the digital age has been the lack of any really permanent way to store bits. The great promise of digital storage has been the idea of "perfect copies," and most people have assumed that would entail permanence as well.

One of the deepest embarrassments of the digital age has been the lack of any really permanent way to store bits

However, the truth has been much more bleak: most of the commercially-available mechanisms for storing bits are depressingly subject to literal "bit rot" when sitting on the shelf. Magnetic media were known to be flawed in this respect even when they were introduced. Experience with magnetic video and audio tapes had already shown that they would fade over time. Digital storage staved off the inevitable by making it possible to re-copy the data without loss before it faded out, but that still left us with the burden of periodically re-copying data that we want to preserve.

Understanding how and why the Millenniata disks have finally solved this problem requires a little background history in optical disk technology -- so please bear with me.

Optical disks, Part 1: The Promise

Optical media offered a way around the impermanence of magnetic media. Early on, as the entertainment industry began promoting Compact Discs (CDs) and later, DVDs, a major selling point was the security of this kind of permanent storage.

DVD movies (like mass-produced audio CDs, CD-ROMs, and DVD-ROMs), it is true, do not fade out the way that even commercial VHS video tapes will inevitably do. Shelf life for these products is certainly on the order of decades, and probably centuries if they are stored appropriately.

DVD movies do not fade out the way that even commercial VHS video tapes will inevitably do

These disks are so good at preserving data, because the data image is a physical deformation of a reflective layer. Also, that later is embedded behind 0.6 (DVD) to 1.2 (CD) millimeters of polycarbonate resin (Blu-Ray uses only 0.1mm). This is an extremely tough, scratch-resistant plastic -- it's the same stuff "bullet proof" windows are made from. And even if the substrate does get scratched or pitted, it can be filled or re-polished to allow reading of the data layer.

There's one huge catch, though: these disks must be manufactured in a production-line "pressing" operation. Creating the die, and other set-up costs, is large -- so it's only economical to do this for runs of 1,000 disks or more. In practice, this costs close to $1,000 per run (about $1/disk) from typical DVD replication services. [1]

Well, that's great if you want a thousand copies, and it is true that more copies means a greater chance that the data will be preserved. But a thousand? Even if you had data worth that investment, it's not exactly environmentally sound!

Optical disks, Part 2: The Disappointment

What was missing was the ability to write disks one-at-a-time with unique data. Naturally, the industry responded to this, and gave us the modern dye based media: CD-R, DVD-R, and variations thereof.

What was missing was the ability to write disks one-at-a-time with unique data

These are great. I use them all the time myself, and I'm sure many of you do as well. They're also quite cheap -- CD-R disks are often cheaper than CD-ROM disks, even for runs of more than a thousand disks. So I'm not trying to put you off of them.

But while such media largely got a free ride on the coat tails of their pressed cousins, they are fairy gold: they are a fundamentally different technology, and as such, they do not share the great data permanence advantages of pressed media.

There are no physical pits on dye-based disks, not before you write data onto them, and not afterward either. The laser in your CD or DVD "burner" is not designed to do that. Instead, it essentially makes "scorch marks" in the photoactive dye-layer embedded in the disk. This is a chemical change in the dye -- the exact change depends on the type of dye used, and there are several different kinds (which is one reason for the different colors evident on different brands of DVD-R and CD-R media -- the other is the composition of the mirror-like reflecting layer underneath the dye).

There are no physical pits on dye-based disks, not before you write data onto them, and not afterward either

The problem of course, is that chemical dyes naturally fade over time, due to a wide variety of environmental factors. Storage conditions matter, and some high quality disks may last a few decades, but many will become unreadable in as little as ten years -- much like magnetic media. [2]

Optical disks, Part 3: Redemption

What Millenniata's technology does is to redeem the promise of optical media. Eschewing the flawed dye-based approach, their drive uses a laser etching approach to physically burn pits into the writing layer of their special "M-Disc" once-writable DVDs.

These pits are physically a little different from the pits in pressed media, but they serve the same purpose, and they are physical, structural changes in the writing layer -- the media is burned and distorted by the heat from the laser, causing permanent pits to be etched into the media.[3]

From a technological standpoint, that's it really. The rest is detail, but the detail is good:

• The disks store 4.7 GB of data, just like a DVD-ROM or DVD-R disk
• The drive presents to software much like other DVD burners
• The disks are readable on standard DVD-ROM drives

This means that the M-Disc technology is highly compatible with the existing optical disk technology that you are familiar with. The disks, once written, can be treated much like any other DVD-ROM you use. They obviously should be stored with as much care as you can manage, but are proof against almost anything a standard pressed DVD would be proof against -- which covers a lot of things that will destroy dye-based DVD-R disks. This was demonstrated by an environmental stress study conducted by the US Navy. [4]

The disks, once written, can be treated much like any other DVD-ROM you use

Millenniata specifically mentions GNU/Linux support for their writer, although from what I gather from the site, this is simply because it looks like any other USB-interface DVD burner from the point of view of the software. The drive itself takes care of the details involved in etching the data.

Of course, the disks and writer are a fundamentally different technology. So the disks can only be written by the Millenniata writer, and the writer cannot write on conventional DVD-Rs. It's a specialized system.

It's also very expensive. The drives and disk are available from multiple sources, but the cheapest price I found (at Trove Archive Solutions) put the drives at about $300, with disks ranging from $20 individually to about $8 in a 25-count bulk pack.[5] This compares to about $17 for a burner capable of writing all dye-based DVD and CD media which themselves cost less than $0.20 apiece. Clearly, at this price, the Millenniata drive does not replace the dye-based media niche. Even if you have one, you're going to want to have a separate drive for dye-based media anyway.

I can think of a lot of people who would find this archival drive well worth the cost

However, I can think of a lot of people who would find this archival drive well worth the cost: university research departments with research data to archive, libraries, and even individuals like myself who simply have a lot of data they want to keep permanently as heirlooms. It's not a lot of data -- which is good, because the 4.7GB capacity of a single-layer DVD-R already seems puny by today's hard drive standards. But it's data that is serious need of safer storage -- and that's what the Millenniata system can provide.

Estimates for the durability of the M-Discs are well over a hundred years. Probably they would last for thousands of years if stored well, just like pressed disks. The more likely problem here is with DVD reader drives vanishing into obsolescence. On the other hand, it is still possible to buy a player for 78 RPM records, more than a half-century after they have ceased being released commercially, and the DVD standard has been similarly very successful. M-Disc data will be part of a large volume of legacy DVD-ROM data.

Are optical disks a good idea?

Even if the M-Disc format meets all of its technical promise, there are some who feel that optical disk media are a fundamentally bad idea to begin with. They argue that the disks are too expensive, do not hold sufficient data, and also that permanence is a false hope in any case. Some have suggested that optical disk media is simply dead for these reasons. [6]

For many digital archivists and computer engineers alike, a new conventional wisdom has started to come into play: which is to simply give up on permanent media!

Instead, this approach treats all digital storage media as fundamentally impermanent, placing emphasis instead on the process of repeated and redundant copying of valuable digital data onto new media -- always keeping the data available on "live spinning disks."

Why worry about offline storage when the amount of online storage is increasing so rapidly that you will always have plenty so that you can reserve some space to store old archives?

Exponentially increasing hard drive storage has contributed to this viewpoint. Why worry about offline storage when the amount of online storage is increasing so rapidly that you will always have plenty so that you can reserve some space to store old archives?

A little conservativism on my part

Well, it's not a bad argument. To some extent, re-copying will be needed, no matter what media are involved. I do have a few criticisms, though:

• This requires a constant effort at preservation
• Forgotten re-copying operations can result in permanent loss of archival data
• "Live spinning disks" are in constant risk of failure -- the risk is small over short time periods, but inevitable over the course of years or decades
• Since re-copying is sometimes imperfect, there is a chance of large datasets being corrupted -- "mutated" -- during the process without the error being noticed until much later (long after it becomes impossible to recover from it)
• Only materials that are continuously recognized as valuable will be preserved

This whole concept is a little disturbing to me. For centuries, important archival materials have been stored in media that can literally last for centuries -- such as printed books. Films and photographs have been successfully stored for as much as a hundred years. We don't know for sure how they'll fare for longer than that simply because they haven't been around that long. Converting to digital seems like a good way to make them last even longer.

But if there is no really durable form of digital storage, then society will be continually forgetting archived data. We'll only have the resources to save those data we continually think are worth preservation -- and the historical record suggests we won't always get that choice right. There'd be no more discoveries from old film vaults; no more "Dead Sea Scrolls" dug out from a cave in the desert after millennia of storage; and no more photos discovered in the back of an old family album.

There are many examples of discoveries in ancient archives of information that seemed of little importance at the time. There are also plenty of examples of political and cultural revolutions in which old data was deliberately destroyed -- only to be recovered (sometimes centuries later) by scholars examining ancient archives. The "live with it" approach of recopying digital media simply doesn't address these problems.

But if there is no really durable form of digital storage, then society will be continually forgetting archived data

We tend not to think about such grand scales of time in computer science. The history of this field has been written in years -- if not months. Certainly not in centuries. But if computer technology is to be entrusted with preserving information for the ages, then we must adjust our perspective.

It's true, of course, that such needs do not apply to the vast majority of data. An awful lot of the data we keep (hard drive backups, computer operating systems, and so on) is "living data" -- constantly subject to change and improvement. As such, its value as a static digital representation is small. Very few people really need a copy of Linus Torvalds's very first release of Linux or of Stallman's first release of Emacs. We want the latest version instead.

Archives of continuing or potential historical value represent a far smaller volume of data. But that's a good thing, because really permanent storage capacity probably never will keep up with temporary storage solutions. And I'll keep archiving my computer files on dye-based CD-R or DVD-R disks and using a storage hard drive for backup protection.

For me, though, it's clear that this niche, though small, is very important culturally and personally. I actually do experience some anxiety about all those photos of mine which are stored on hard disk or dye-based optical media. I actually lost a considerable number of them when one of my hard drives crashed. Since then, I've made sure to keep two copies of the data on two separate hard drives on two separate computers. And I also keep dye-based media backups. But this is a precarious system. I would far rather have at least one copy on media with a longer shelf-life. And although it's going to take me awhile to save up my pennies to buy one, I think the Millenniata M-Disc is likely to be that option for me.

References

[1] Based on quotes from various DVD replicators, for example: Nationwide Disc or Discmakers

[2] There is a lot of disagreement on how long dye-based media will last once burned. The CDR FAQ has more information

3(http://www.millenniata.com/products/products/) and other press materials available from the Millenniata website.

4(http://www.millenniata.com/orphan/china_lake_summary/) by Ivan Svrcek, conducted at the Ivan Svrcek at the Life Cycle and Environmental Engineering Branch, Naval Air Warfare Center, Weapons Division in China Lake, California. November 10, 2009

[5] Trove Archive Solutions was the cheapest source for Millenniata disks and writers that I was able to find as of Feb 1, 2011. The featured product is a kit with a drive and 25 M-Discs, although these can also be purchased separately

6(http://www.pcworld.com/article/218110/the_optical_drive_appendix_of_the_modern_pc.html) by Jason Cross, PC World. Jan 28, 2011