For those who may not be familiar with LTFS, you can read a writeup about it at Wikipedia.org. The import thing to note is that LTFS requires LTO-5 or LTO-6 tape drives. LTFS cannot work on LTO-4 or earlier tape models. Also, converting an existing OSVault TAR-based tape system to LTFS is not an easy task and is not outlined in this posting.
Our implementation of LTFS support in OSVault allows for files to be stored in the XFS file system on spinning disk, and to have those files stored in that file system automatically be copied to a tape in the LTFS format. When the XFS file system is becoming full, the data portion of older files is truncated from the XFS file system for those files that have been copied to LTFS tape. If a truncated file is opened later, OSVault uses the DMAPI interface to realize this and will automatically restore the file to spinning disk from the tape cartridge before the first I/O on the file is allowed to proceed.
There are five major steps in getting LTFS support into OSVault:
- The LTFS file system support had to be installed in the LINUX system that OSVault runs under
- The AUTOFS implemenation for SCSI medium changers (tape libraries) had to be developed that supported LTFS mounts and unmount
- The LIN_TAPE device driver had to be ported to the DMAPI kernel
- The Migration (copying) program had to be modified to support LTFS
- A LTFS_INV program to label all media in a robotic tape library and to update the database with that information
LTFS File System
Getting the LTFS file system was the easiest part of the implementation, since we used IBM's LTFS packages. Currently we are using LTFS 1.3.0 and that software is installed using an IBM-provided RPM package for CentOS 5.8.Using the standard IBM package gives us several advantages beyond labor savings in the development process. The biggest is that this baseline is the first implemenation of LTFS and has the most test time in its implementation (since 2010). On interchangeability of our tapes to other LTFS system, we can be reasonably sure that any difficulties are due to the other product, since we are using an industry-standard implementation and have not made any modifications for our product.
AUTOFS Implementation
This was probably the most difficult part of the LTFS development. We have been using AUTOFS 4.1.2 historically for mounting and unmounting DVD and Blu-Ray media in OSVault. Tape (TAR) support in OSVault did not use AUTOFS. However, due to the nature of LTFS, it was determined that the best way to support LTFS in OSVault was to use AUTOFS (and the automount daemon) to mount and dismount tape media.The AUTOFS allows the LINUX system to show every tape in the tape library as a separate folder in the /archive mount point. So, if tapes in the library have barcodes of OS0901L5, OS0902L5, OS0903L5, OS0904L5, and OS0905L5, then the folders /archive/OS0901L5, /archive/OS0902L5, /archive/OS0903L5, /archive/OS0904L5, and /archive/OS0905L5 are available to write to and read from.
In the above example, even though there are 5 tapes in the library, you can randomly access folders even if you have as few as one LTO-5 drive (although we recommend at least 3 drives in a production environment). The AUTOFS implementation will remove/unmount media to put requested media into a drive without the user needing to worry about that.
The AUTOFS changes included the creation of a new automounter module, mount_ltfs and lookup_ltfs. Those two modules deal with the robotic operations of moving media into and out of tape drives, finding the correct media in the robotic library, verifying that the tape in the drive has the same internal label as the barcode on the outside, and figuring out which tape drive to use for various operations.
A custom change to AUTOFS was to add in code to do an LTFS-specific set of operations when the tape drive mount times-out, as from inactivity. In most AUTOFS modules, there is only a generic umount done on the folder when it times out from inactivity. In our LTFS support, it is necessary to issue an unload to the tape drive, to take the media off the recording heads and return the tape into the cartridge (a generally accepted practice when not using a tape drive).
One of the advantages of using AUTOFS for LTFS access is that it is possible for networked users to directly read from the tape media, bypassing the /cache file system in OSVault.
LIN_TAPE Driver Port to DMAPI Kernel
We use the DMAPI interface to control the automatic restore of file data from removable media back to the XFS file system on spinning disk. The DMAPI code is not in the mainline LINUX kernel, and OSStorage has had to maintain their own baseline of kernel patches. We started with the 2.6.24-rc3 baseline that had DMAPI included in the XFS file system many years ago, and have been adding patches to that kernel to keep up with new devices and other kernel changes over the years.Adding the LIN_TAPE driver (IBM's standard open-source driver for LTO tape drives) to this kernel was challenging. LIN_TAPE had many conditionals in the source code based on the 2.6.24 release, which our kernel reported as supporting. But since we had a Release Candidate, some of the SCSI interfaces had changed and our LIN_TAPE port did not match up correctly. So the LINUX SCSI Interface code in LIN_TAPE had to be modified to match our kernel's implementation.
Migration to LTFS
OSVault transparently restores files from LTFS to the XFS file system when requested. But prior to that restore, the file had to be copied to tape media. We decided to modify the migration code that supports writing to BD-RE media (Blu-Ray rewritable media) and apply that the LTFS. Not many changes were required and this migration code has been extensively used in our lab and at customer sites since as early as 2003. The major changes required adding and "ltfs" format to the MySQL database and changing the database calls to support LTFS media entries. Since BD-RE media only goes to 100GBytes, a lot of subsequent testing was required to make sure that this migration program would continue to work up to 2.5TBytes of storage per media.LTFS_INV Program
A tape inventory program had to be developed that could query a SCSI Medium Changer (Robotic tape library) and create a database of tape media found in that tape library. That "ltfs_inv" program has been tested on Qualstar and SpectraLogic brands of tape libraries, but should work on any tape library that supports the SCSI standard for Medium Changers and has LTO-5 or LTO-6 tape drives.The ltfs_inv program loads all tapes in the library into tape drives, checks if it already has an LTFS file system on it, and if not it will format the tape with the LTFS file system. The internal volume label of the LTFS file system is also set to match the barcode on the outside of the tape cartridge and that internal volume label is checked each time OSVault mounts that tape.
