What should you do to preserve records with a long life cycle?

Unseen dangers lurk. Opportunistic bacteria, crocodiles and sleeper cells all lie in wait for the right moment to wreak havoc on the unaware. Some hide, some take cover and some wear camouflage until the moment of revelation. To paraphrase Thomas Pynchon, paranoia is valid when the threat is real.

Add to the list of dangers: digital records with long life cycles. If a digital file is a record, it is important, by definition. Keeping an (important) digital record viable for a long time is problematic.

How long is long? Definitions abound, but consider that common removable digital storage media—like DVDs and Digital Linear Tape—lose integrity in fewer than 10 years, according to many sources. Similarly, leading software producers typically end support for versions of operating systems and application software after 10 years. Ten years is a useful (and some would say generous) life expectancy for the original forms of digital records.

Escalating variables

By way of contrast, parts of physical records remain after 3,000 years. The Dead Sea Scrolls go back a couple millenniums. Translating ancient texts challenges linguists, but the physical eye and sunlight are the only necessities. Similarly, microforms with a projected longevity of 500 years store large quantities of records in a small space. Their technology is only a bit more complex: In addition to a light source, the reader needs a magnifying lens.

The complexity multiplies geometrically for digital records. The list of issues includes computers, peripherals, protocols, operating systems, device drivers, formats, application software, storage media and more. That complexity raises the number of variables and, hence, the size of the array of risks to digital records. As the time of storage grows, so do the number of risks.

Long-term storage of digital records is not a universal issue. In this writer’s experience, the life cycle of a majority of records—especially in the private sector—has been less than 10 years. Selected records of business transactions, regulatory reports, accounting, inventories and compliance documents can and should be disposed of in fewer than 10 years.

The remaining minority of records—those with 10-year-plus life cycles—may be more significant than their ephemeral counterparts, on average. Those include records of laws, regulations and history; records of property, real and intellectual; and health/medical records. Concerning the last, diagnosing physicians improve their perspectives with lifelong records of their patients. The health history of parents and grandparents helps identify familial trends and likelihoods. Also, medical researchers covet families’ multigenerational records for longitudinal studies.

A real-world example: For a developer of implantable heart defibrillators, it is redundant and too costly to start each new model from scratch. Instead, each model builds on the previous one. That daisy chain reaches back to pacemaker research in 1972, and the laboratory notebooks that led to Defib Number One must remain viable as long as the line continues. There is no foreseeable end to the need for those records.

The HITECH problem

The U.S. Congress passed the Health Information Technology for Economic and Clinical Health (HITECH) Act as part of the American Recovery and Reinvestment Act of 2009. Touting that legislation, President Obama mentioned improved healthcare, but gave primary emphasis to the need to control costs.

As clarified by the Centers for Medicare and Medicaid Services on July 13, 2010, healthcare facilities that want to qualify for large payments in 2011 and 2012 will have to keep 80 percent of their patient records digitally. Also, 40 percent of prescriptions have to be digital. After two years of transition through government incentives, mandatory regulations will launch in 2013.

Between the U.S. government’s carrot and stick, and a feeding frenzy among opportunistic vendors, it is clear that American medical records soon will be digital for the most part. The economic stimulus will occur.

Unobserved in both the mainstream press and the computer trade press is the viability of long-term storage of medical records. In the great rush to digitize, few address the maintenance of records. Amidst the heady exuberance of progress, danger lurks.

Obstacles

Consider the words of futurist and optical character recognition (OCR) inventor Ray Kurzweil: “Information lasts only so long as someone cares about it. The conclusion I’ve come to … after several decades of careful consideration, is that no set of hardware and software standards existing today, nor any likely to come along, will provide any reasonable level of confidence that the stored information will still be accessible (without unreasonable levels of effort) decades from now.”

The complexity necessary for long-term preservation of digital records contributes to the potential for failure. Consider the vulnerable points, colloquially known as a chain of weak links:

• Removable media—There is no way to assess the longevity of removable media, including tape and laser-written disks (CD, DVD, Blu-ray, WORM, etc.). There are too many variables. The standard way of testing media is “accelerated aging,” but only the future will prove its accuracy. There are no standards for quality, and it is difficult to impossible to consistently identify the materials and manufacturing processes that produced a shipment of disks. Controlled temperature and humidity affect the stability of a disk’s substrate and dyes, but storage environment is only half a disk’s life cycle. Even for a disk made of high-quality materials, who can verify the conditions and duration under which the disk was transported and warehoused before use?
• Hardware obsolescence is obvious and visible to all who have reached adulthood. Floppy disk drives (5.25-in. or 3.5-in.) are difficult to find, to say nothing of players for ZIP, Jazz and other proprietary format media. In 1991, Sony advertised a 12-in. optical disk guaranteed to last 99 years. While the verity of that claim will be revealed in 2090, not even museums are likely to have a compatible disk player.
• As operating systems evolve, they have limited compatibility with their ancestors. Can a PC running Windows 7 read files created under CP/M? Can today’s IBM Power System OS consistently extract files written on a s/38 from 1979 or a system/3 from 1969?
• Application software becomes obsolete as well. Consider the boneyard of former market leaders: VisiCalc, WordStar, Wang word processing and countless others. Microsoft stopped supporting Office 97 about 10 years after its release.
• Output drivers—Retrieving old records also depends on being able to display and/or print content. Even with legacy CPUs, OSes and application software, the period monitors, printers and the software that drives them may be unavailable or inoperable.
• Encryption and password protection—Long-lived records, including health records, often require security and privacy protection. Maintenance of passwords and decryption capabilities loom as additional impediments to the retrieval of old records.

Options for long-term preservationThere are at least four viable options for keeping digital records available over 25 to 100 years.

The most obvious is media migration. Here, records transfer from a storage medium approaching obsolescence to newer technology. For example, records written in MS Word 97 and stored on single-density CDs transfer to Word 2010 files written on Blu-ray disks. The portage to newer technologies may be repeated several times for records with long or indefinite life cycles. A drawback is that each migration alters the metadata accompanying individual records.