Hedge, the popular media back up and verification company has launched Canister, an LTO backup app for macOS. Canister has been in development for a year and a half and it has been designed to make it easier to work with tape, without having all of the usual headaches associated with LTO.
Tapes are not hard drives
External hard drives are very user-friendly and they are designed to store material quickly and efficiently. Unfortunately, they aren’t designed to store material for long periods of time. Hard drives will eventually fail on you, it’s just a matter of when, and not if.
LTO, on the other hand, is rock solid tech, but if you’re not very computer savvy it can be overwhelming. Companies like mLogic do great work making LTO hardware easy to use, but with software, life is not that simple. That’s why Hedge decided to build Canister. Canister uses a super simple interface, duplicate detection, xxHASH64 validation, and creation of MHLs and Transfer Logs. Canister has a lot of new technology onboard to try and make the LTO experience a lot easier.
Pitfalls
LTO has seen rapid development, with IBM and Fujifilm putting in a lot of R&D to ramp up speed and capacity. Nonetheless, LTO does have its pitfalls. Working with LTO so far hasn’t been straightforward: you need a tape drive, a SAS to Thunderbolt adapter (an HBA), LTO tapes, and you needed to install several frameworks on your Mac. and have a firm knowledge of the do’s and don’ts of LTO tapes to make sure you don’t damage your tapes. You also need a piece of software to manage your backups.
So what does Canister have to make everything much easier?
No more Shining your shoes
Just like your cassette deck had a play/record head, so does an LTO drive. Each time the head touches down on or lifts up from, the tape, a little wear is introduced. On an audio cassette, this causes noise levels to rise. With LTO, something more important is at stake: data integrity.
To prevent wearing out a tape, reading and writing ideally happen in long, continuous runs. That’s why a LTO drive has an internal buffer of a few hundred MBs, and it will write that buffer in a burst to the tape. If the buffer isn’t filled completely, the tape overshoots and has to lift the head, rewind, drop down, and do the next (partial) burst. Imagine you have a source that can’t keep up with the speed LTO desires. A lot of unwanted winding forth and back will be the result. It’s called shoeshining.
Canister eliminates shoeshining entirely by actively managing the supply and demand between tape and storage. It even works if your storage is a slow network drive.
Drag ’n’ drop
Because inside a hard drive is a stack of round discs, it’s fast to jump between different sectors of data. This makes it possible to browse through a drive’s contents with no apparent delay. A tape, however, needs to rewind and fast forward between each sector. It’s linear. As a cartridge can hold about 1 KM of tape, zipping forth and back can take quite a while.
To be able to know what is stored on a tape, LTO uses a software counterpart called LTFS — Linear Tape File System. LTFS splits a tape into two partitions, one with the index and one with the actual file data. Having an index makes it possible to browse a tape in Finder. You could even drag files to the tape folder on the Mac, and they would be written to tape.
Unfortunately, doing this will cause the tape to do wear out very fast. Finder will access the data partition even when it only needs to retrieve file information stored in the index partition. It’s a bit weird that this is the major selling point for LTFS touted by the likes of IBM — it is actually the worst advice possible. You do not want to browse a tape with Finder.
It makes far more sense to browse a tape without accessing it at all. Canister achieves this by temporarily recreating the index on your computer after mounting a tape. This allows you to cherrypick what you need to retrieve or create folder structures on a tape before archiving onto it. Only when you’re ready for it, by starting a transfer, Canister accesses and updates the tape.
Drivers
Speaking of LTFS, it’s not just a matter of installing the driver and walking away. Unfortunately, just like most technology, a new generation of LTO is released every few years or so. With each generation speed and/or capacity are improved. The first LTO generation to support LTFS was LTO-5, and in 2017 LTO-8 was launched.
Since LTO-7, things have become simple: there is one company left that builds LTO drives — IBM. But not everyone uses LTO-7 or higher. Older LTO drives are still supported, and these older ones are still being put to good use on a daily basis.
For those older generations, LTO life is complicated. There used to be three companies building drives: HP, Quantum and of course IBM. To make an LTO drive work, you need to install drivers or frameworks on your Mac. The LTFS framework itself relies on another framework called ICU, and it utilizes FUSE to create a virtual file system — like Codex does.
Now, all three vendors have their own version of LTFS and require different versions of ICU to work. One cannot simply install all of them, as the last installed LTFS version overrules earlier ones, but the first ICU framework installed overrules later installs. This is very confusing, to say the least. There is also a multitude of companies making various drivers as well, just to further complicate things.
Canister comes with a simple installer which allows you to install the set of drivers you need. If you are not sure which driver you need, you just open Canister, and it will tell you which drivers are missing. Hedge also made it easy to swap drivers.
Illegal characters
Another easy trap to fall into is LTO’s character set. As LTO has to work across a wide span of operating systems, file systems, and of course languages, it has a strict set of illegal characters. One way to get around that is to rename files and keep track of what has been renamed into what with a database.
That means that to correctly retrieve media without relinking issues in your NLE, you require a database. Now fast forward to ten years from now, and your LTO software might be way beyond its end of life. With a potential shelf life of 30 years for your tapes, who knows what computing looks like by then? Can you imagine having to support a database from the early 80s, today?
So, preferably no database. Besides that, working around illegal characters is done all the time without needing a database, and you even know how: a space in a URL becomes %24. That’s called URL encoding, and it’s very robust. Each character has a unique code, and there’s zero overhead when encoding or decoding.
Canister utilizes URL encoding for all illegal characters, renaming files on the way to tape, and reversing the process when retrieving from tape. Without the need for a database, there is simply less to worry about — and you’ll know for sure it will work forever.
To index or not to index
When writing data to a hard disk, its partition table, the index of what goes where is updated on the fly. Again, tape behaves very differently. LTFS divides a tape into two partitions: one for data, and one for the index. After writing data to tape, the index has to be updated. There are three options: after every file has been written, after every x seconds, or when ejecting (“unmount”).
Updating the index for each file is a terrible idea when using frame-based codecs or backing up complete projects with tiny files inside. Indexing every so X minutes is much better, but will interrupt large transfers leading to shoeshining. Worse, the last bit of data written to tape is potentially not indexed if you shut down the tape machine too soon. Index on unmount indexes when, well, unmounting.
The ideal situation would be a mix of all three depending on file size, time or GBs transferred since the last index, and to always finish up when ejecting. Alas, only one can be chosen. It’s a severe limitation of how LTFS was designed , hopefully this will be taken care of in a future version of the framework maintained by IBM.
Creating the index on unmount is the only foolproof mechanism to make sure what has been written to tape is safe and sound. That’s why Canister ejects each tape when a transfer to tape is done. Luckily, mechanically reingesting the tape is also supported by LTO drives, so you don’t have to go and insert a tape when remounting.
How much does LTO cost?
LTO drives ain’t cheap. For example, a mTape LTO-8 Tape Drive with Thunderbolt 2 costs $4,999 USD. The LTO tapes are, however, very affordable compared to hard drives. A 12TB Quantum LTO-8 Ultrium Data Cartridge costs $225 USD.
Price and availability
Canister looks like a much easier way to make your LTO life far easier. For a limited time, Canister is available for $149 USD. You can download it at hedge.video/canister where you can also download a free 10 day trial.
