Archiving: How To Save Our Digital Assets

Historically, film companies stored finished movies on film and kept them in a secure, temperature-controlled warehouse. This was fine in the analogue age, but today, in the digital age, all that information still needs to be stored securely, yet remain easily accessible - says Ray Quattromini, MD, Fortuna Power Systems...

The issue with storing digital data is that we now create more information with better effects, greater clarity and definition than ever before - and this consumes vast amounts of disk space. Where we put the finished movie/advert/corporate video and the extra footage, once the editing is finished, is a challenge that grows daily.

LTO (Linear Tape Open) came along about 15 years ago and marked the first time a whole movie could be stored on tape securely. Today, we have LTO-6 storing 2.5TBs native. The issue with storing data on LTO is you will always be playing catch-up.
- LTO can only read the previous two generations of data
- You will require the application that wrote the tape to be able to read it back, unless it was written in a non-proprietary format
- It can be slow to bring back, if required urgently

LTFS (Linear Tape File System) – Based on LTO, this could solve many issues associated with using an LTO archive solution. Designed by IBM and now adopted as a standard, LTFS (from LTO-5 onwards) generates tapes that appear as a big USB key to the operating system. In order to read an LTFS generated tape, all you need is an LTO-5 or LTO-6 tape drive attached to a Windows, Linux or Apple system. There are some very clever LTFS archiving solutions whereby the data is initially written to NAS, and attached to this is a tape library which can store 100s of TBs of data. When you move to LTO “x”, you just attach your new library and the system automatically migrates the data. But as it is doing so, it checks the existing tape’s data integrity and ensures your archive is 100% intact.

Hard Disk – Capacities today are 4TB, and to store 500TBs in 8U costs approximately £50,000. The systems have MAID (Massive Array Idle Disks) which means they power down when not required. Access is instant and they support all RAID levels. I know of many companies using USB-attached drives to store digital data, and these are not enterprise-grade drives, or even designed for this purpose. Disks do cost more than tape, but in order to read a tape you need a drive.

Cloud – This is the new kid on the block, but comes at a price. While it is great to have your archive available 24/7 and with global access, should you really archive old footage in the Cloud? Shouldn’t you use the Cloud to access the job you are working on now and allow access?

Cloud companies make money in the following ways:
- Every GB you store based on an agreed amount you are going to upload
- A monthly charge to store your data
- An additional charge if you want it in two locations
- A charge to download your data
In addition to this, there are many legal aspects that are not considered:
- How easy is it to get your data back if the company goes bust?
- Who has the legal right to the data if a dispute arises?
- What SLA is in place?
- How can you check that your data is not de-duplicated?
- How can you check that your data is not interleaved with other customers’ on tape?

Blu-ray – Unlike the other technologies shown, Blu-ray, while poor in capacity at 100GB, is designed to have a 50 year archive life. With Blu-ray you do not need any special software to read a BD disk if it was created in UDF 2.0, and it can easily be duplicated if required. Ideally, Blu-ray should be used for your digital master, taking more than one copy and storing your TBs on one of the above technologies.

Summary
The answer to storing many TBs of data is a big challenge. You want to avoid proprietary vendor lock whereby your data resides on systems, as you have no way of getting information back if the company goes bust. You also need to archive it on reliable storage that is going to last. As far as I am concerned, USB-attached storage is not an archive. Ideally, you should try and do the following:
- Make a minimum of two copies
- Store the data on two different media types i.e. LTO and hard disk
- Periodically check your data is still intact and hasn’t degraded
- Avoid data interleaving and de-duplication
- Every 5 years assess the market to look at the alternatives by carrying out a technology refresh.
- Keep the data in its original format
- Do not rely on incumbent suppliers for advice

By Ray Quattromini, MD Fortuna Power Systems Ltd.
Ray has spent over 30 years in the data storage industry working for manufacturers, distributors, resellers and now runs his own data storage business. He speaks daily with companies on how best to solve data storage requirements.

Read the article in the online edition of Regional Film & Video here.