Firmware Dependencies in 100GB Archival Blu-Ray Media

In testing a new library with 2 Pioneer Blu-Ray drives, we noticed the drives had different firmware versions.  In using the drives, we also noticed that some new Mitsubishi 100GB Archival media, that the media will not format in the drive with firmware version 1.01, but will format in the drive with firmware version 1.13.  What makes this more complicated is that some Mitsubishi media does format in a drive with either firmware version.

We don't have enough test media to be 100% confident, but it appears that the media with ID "VERBAT/IMk" will not work with older firmware releases in the Pioneer drive, but that media with IDs "TDKBLD/RNC" and "VAMKM/IMk" will work in drives with either firmware versions.

The following is from a CLI test on a piece of new, blank media placed into two drives with earlier and newer firmware.

[root@vault5 mclib]# dvd+rw-mediainfo /dev/scd1
INQUIRY:                [PIONEER ][BD-RW BDR-PR1NZA][1.13]
GET [CURRENT] CONFIGURATION:
 Mounted Media:         41h, BD-R SRM
 Media ID:              VERBAT/IMk
 Current Write Speed:   2.0x4495=8990KB/s
 Write Speed #0:        4.0x4495=17980KB/s
 Write Speed #1:        2.0x4495=8990KB/s
 Speed Descriptor#0:    00/48878591 R@4.0x4495=17980KB/s W@4.0x4495=17980KB/s
 Speed Descriptor#1:    00/48878591 R@2.0x4495=8990KB/s W@2.0x4495=8990KB/s
:-[ READ BD SPARE INFORMATION failed with SK=5h/MEDIUM NOT FORMATTED]: Wrong medium type
READ DISC INFORMATION:
 Disc status:           blank
 Number of Sessions:    1
 State of Last Session: empty
 "Next" Track:          1
 Number of Tracks:      1
READ FORMAT CAPACITIES:
 unformatted: 48878592*2048=100103356416
 00h(c000): 47305728*2048=96882130944
 32h(0): 47305728*2048=96882130944
 32h(0): 30790656*2048=63059263488
 32h(0): 48854016*2048=100053024768
READ TRACK INFORMATION[#1]:
 Track State:           invisible incremental
 Track Start Address:   0*2KB
 Next Writable Address: 0*2KB
 Free Blocks:           48878592*2KB
 Track Size:            48878592*2KB
READ CAPACITY:          0*2048=0
[root@vault5 mclib]# mm 32769 32768 0 0 library
Opening library
[root@vault5 mclib]# dvd+rw-mediainfo /dev/scd0
INQUIRY:                [PIONEER ][BD-RW BDR-PR1NZA][1.01]
GET [CURRENT] CONFIGURATION:
:-( no media mounted, exiting...
[root@vault5 mclib]#

True Disaster Recovery in a Cloud Storage Environment

Recent news articles have discussed the Chapter 11 bankruptcy of a large cloud storage provider. This cloud storage provider marketed itself as completely safe, with up to 5 physical locations for customer data, and a large amount of venture financing to insure continuity.  That bankruptcy left several large customers with only 2-4 weeks to retrieve data that took months or years to store.  Speculation is that there could be upwards of 20 petabytes of data that need to be retrieved from those systems, and now that data access is in the hands of a bankruptcy court.

Obviously, in a cloud environment, as this case shows, disaster is not limited to fire or flood, but also includes insolvency of your provider.  Even if the data is read out from the cloud storage to another storage device at 1gigabit per second, it could take years before all data can be safely transferred.  But in the meantime, access to your valuable data is constantly in jeopardy and out of your control.

This is one of the reasons that Open Source Storage uses OSVault and tape-based archival for our storage systems.  The other reasons are cost and reliability.  With dedicated tape volumes per customer, and having those volumes owned by our customers, we can move a complete set of data from our site to another site in less than one day.  We use open formats to store data, so that moving to other vendors does not require complex and costly manual processes.  And our costs of storage are still 75% less than other cloud storage providers.

Tape-based archives give us some distinct, valuable advantages.  We can make second copies and move those second copies offsite for very little incremental cost (about $10 per terabyte capital cost and less than $.05 per terabyte storage cost per year).  Accidental or deliberate deletion of data is difficult or impossible since we keep maps of all data stored on tape in multiple locations and do not recycle tapes so that even deleted data can be recovered.  And we can grow our systems from small to large without expensive or time consuming hardware replacement or upgrades.

As an exercise, we implemented a 500TByte disaster recovery to a separate location.  The total cost of the additional hardware was less than $20,000 (although we had already purchased the hardware as part of our disaster planning).  The total time required to get access to all files in the 500TByte file system was less than one day.

So, when planning your cloud-based or in-house storage environment, please consider the following issues in your disaster recovery planning:

• Is my data stored in two separate locations, to safeguard against flood, fire and other disasters?
• Is my data safe from individual actions, either mistakes or deliberate?
• Can I get my data physically relocated without costly or time consuming effort?
• Do I have the resources and trained personnel to implement a disaster recovery?
• Is access to my data completely within my control during any disaster?

Recovering From A Failed Tape Cartridge

If you have been storing files on a tape cartridge and it starts to exhibit serious I/O errors, you need to act quickly to make sure you don't lose any files.  Luckily, tape cartridges rarely fail, and they usually fail when you first start writing data to them.

You will need to use the Command Line Interface to run most of these steps:

First Step - Run "unmigrate -s /cache -v LABEL -r",   where LABEL is the barcode label of the tape that is getting errors.  This will make sure that all files on the tape that are still on disk are marked as not migrated, so that they will be written to another tape at a later time.  The "-r"  means to "don't recall" the files, so that the tape will not be loaded to get files back.

Second Step - After running the "unmigrate", you should go the the web gui, click on the tape label in question, then select "Display Current Tape Contents".  The popup window that appears will show all the current files that are still on the suspect tape.  Hopefully, this display shows no files.  If it shows no files, you are done and should eject that tape from your tape library and destroy it.

Third Step - If you are at this step, then you have files on the tape that are not fully on disk storage and you need to recover those.  If you have been making second copies ("copy2" tapes), then you can just switch to the second copy for those files and restore them.  If you do not have a second copy, so to the "Try to Recover From Bad Tape" step below.

To switch to the second copy, run "switchmig -s /cache -v LABEL".  This program will make all files that have their primary copy on the tape with barcode LABEL now have their primary copy where ever the second copy was stored.  After "switchmig" completes, run "check_db" to make sure that the database has the new locations.

Fourth Step -  Now that you can recover the files from the second copy, you can elect to just leave things that way, or you can unmigrate all those files that were originally on the bad tape, so that there are two new copies on other tapes.  If you wish to just use the previous second copy as the primary copy now, just stop at this point and the next second copy migration will make another copy of the file.

Fifth Step - How to Unmigrate the files from second copy media back to disk. This step is more involved and requires you to use a LinkFile created in /var/tmp.   If you look at a directory listing of /var/tmp, you will see the linkfiles.  Select the linkfile from yesterday,

Try to Recover From Bad Tape Step
The procedure to follow at this point depends on whether the bad tape is a TAR tape or an LTFS tape.  For LTFS tapes, you should follow the LTFS recovery processes detailed by IBM.  For TAR format tapes, you should run "unmigrate -s /cache -w".   This will create a restore process that will recover all files in order, until a serious I/O error occurs.

Using the OSVault GUI To Recycle Tapes

In release 5.0.38 of the OSVault GUI, a button was added to automate the un-migration of files from a tape volume, as shown in the example below.  In conjunction with the Media Usage report, you can use this screen to move all files off a tape that is not very full so that the tape can be removed from the library or recycled.

When you go to the Media screen, and click on a tape volume, the above window will appear.  You should then click the "Display Current Tape Contents" to see what files are on the tape and get a count of files on the tape.  If that tape is mostly empty, you can then click the "Unmigrate Volume" button, which will start the process to make sure that all files on the tape are restored back to disk and marked as not-resident on any tape volume.

When the Unmigrate window opens, you must then confirm that you want to unmigrate that tape by again clicking on the button with the tape volume displayed.  That will then start the process of finding all files in /cache that are copied (or only resident) on the selected tape volume.  Once all files are identified, a work queue is built of those files and the restore of files only on tape is started back to disk storage.   This process will take some time to complete.  The window will display the progress.

When the unmigrate is complete, you should repeat the above process to insure that all files are marked as not present on the select tape volume.

Technical Details on LTFS Implementation with OSVault

For the past two years, we have been working on getting LTFS support into OSVault.  That work is complete and this posting is an attempt to outline the technical details in that development effort.

For those who may not be familiar with LTFS,  you can read a writeup about it at Wikipedia.org.  The import thing to note is that LTFS requires LTO-5 or LTO-6 tape drives.  LTFS cannot work on LTO-4 or earlier tape models.  Also, converting an existing OSVault TAR-based tape system to LTFS is not an easy task and is not outlined in this posting.

Our implementation of LTFS support in OSVault allows for files to be stored in the XFS file system on spinning disk, and to have those files stored in that file system automatically be copied to a tape in the LTFS format.  When the XFS file system is becoming full, the data portion of older files is truncated from the XFS file system for those files that have been copied to LTFS tape.  If a truncated file is opened later, OSVault uses the DMAPI interface to realize this and will automatically restore the file to spinning disk from the tape cartridge before the first I/O on the file is allowed to proceed.

There are five major steps in getting LTFS support into OSVault:

1. The LTFS file system support had to be installed in the LINUX system that OSVault runs under
2. The AUTOFS implemenation for SCSI medium changers (tape libraries) had to be developed that supported LTFS mounts and unmount
3. The LIN_TAPE device driver had to be ported to the DMAPI kernel
4. The Migration (copying) program had to be modified to support LTFS
5. A LTFS_INV program to label all media in a robotic tape library and to update the database with that information

LTFS File System

Getting the LTFS file system was the easiest part of the implementation, since we used IBM's LTFS packages. Currently we are using LTFS 1.3.0 and that software is installed using an IBM-provided RPM package for CentOS 5.8.
Using the standard IBM package gives us several advantages beyond labor savings in the development process.  The biggest is that this baseline is the first implemenation of LTFS and has the most test time in its implementation (since 2010).  On interchangeability of our tapes to other LTFS system, we can be reasonably sure that any difficulties are due to the other product, since we are using an industry-standard implementation and have not made any modifications for our product.

AUTOFS Implementation

This was probably the most difficult part of the LTFS development.  We have been using AUTOFS 4.1.2 historically for mounting and unmounting DVD and Blu-Ray media in OSVault.  Tape (TAR) support in OSVault did not use AUTOFS.  However, due to the nature of LTFS, it was determined that the best way to support LTFS in OSVault was to use AUTOFS (and the automount daemon) to mount and dismount tape media.
The AUTOFS allows the LINUX system to show every tape in the tape library as a separate folder in the /archive mount point.  So, if tapes in the library have barcodes of OS0901L5, OS0902L5, OS0903L5, OS0904L5, and OS0905L5, then the folders /archive/OS0901L5, /archive/OS0902L5, /archive/OS0903L5, /archive/OS0904L5, and /archive/OS0905L5 are available to write to and read from.
In the above example, even though there are 5 tapes in the library, you can randomly access folders even if you have as few as one LTO-5 drive (although we recommend at least 3 drives in a production environment).  The AUTOFS implementation will remove/unmount media to put requested media into a drive without the user needing to worry about that.
The AUTOFS changes included the creation of a new automounter module, mount_ltfs and lookup_ltfs.  Those two modules deal with the robotic operations of moving media into and out of tape drives, finding the correct media in the robotic library, verifying that the tape in the drive has the same internal label as the barcode on the outside, and figuring out which tape drive to use for various operations.
A custom change to AUTOFS was to add in code to do an LTFS-specific set of operations when the tape drive mount times-out, as from inactivity.  In most AUTOFS modules, there is only a generic umount done on the folder when it times out from inactivity.  In our LTFS support, it is necessary to issue an unload to the tape drive, to take the media off the recording heads and return the tape into the cartridge (a generally accepted practice when not using a tape drive).
One of the advantages of using AUTOFS for LTFS access is that it is possible for networked users to directly read from the tape media, bypassing the /cache file system in OSVault.

LIN_TAPE Driver Port to DMAPI Kernel

We use the DMAPI interface to control the automatic restore of file data from removable media back to the XFS file system on spinning disk.  The DMAPI code is not in the mainline LINUX kernel, and OSStorage has had to maintain their own baseline of kernel patches.  We started with the 2.6.24-rc3 baseline that had DMAPI included in the XFS file system many years ago, and have been adding patches to that kernel to keep up with new devices and other kernel changes over the years.
Adding the LIN_TAPE driver (IBM's standard open-source driver for LTO tape drives) to this kernel was challenging.  LIN_TAPE had many conditionals in the source code based on the 2.6.24 release, which our kernel reported as supporting.  But since we had a Release Candidate, some of the SCSI interfaces had changed and our LIN_TAPE port did not match up correctly.  So the LINUX SCSI Interface code in LIN_TAPE had to be modified to match our kernel's implementation.

Migration to LTFS

OSVault transparently restores files from LTFS to the XFS file system when requested.  But prior to that restore, the file had to be copied to tape media.  We decided to modify the migration code that supports writing to BD-RE media (Blu-Ray rewritable media) and apply that the LTFS.  Not many changes were required and this migration code has been extensively used in our lab and at customer sites since as early as 2003.  The major changes required adding and "ltfs" format to the MySQL database and changing the database calls to support LTFS media entries.  Since BD-RE media only goes to 100GBytes, a lot of subsequent testing was required to make sure that this migration program would continue to work up to 2.5TBytes of storage per media.

LTFS_INV Program

A tape inventory program had to be developed that could query a SCSI Medium Changer (Robotic tape library) and create a database of tape media found in that tape library.  That "ltfs_inv" program has been tested on Qualstar and SpectraLogic brands of tape libraries, but should work on any tape library that supports the SCSI standard for Medium Changers and has LTO-5 or LTO-6 tape drives.
The ltfs_inv program loads all tapes in the library into tape drives, checks if it already has an LTFS file system on it, and if not it will format the tape with the LTFS file system. The internal volume label of the LTFS file system is also set to match the barcode on the outside of the tape cartridge and that internal volume label is checked each time OSVault mounts that tape.

Handling Second Copy Tape Media in OSVault

OSVault has the ability to make two copies of every file, and to make sure that each file is on separate physical media.   There are some good reasons to do this:

• Tape is not infallible, although we have not had an LTO-4 or LTO-5 tape fail on us
• A second copy of files can be taken offsite to guard against catastrophes
• Sometimes humans make mistakes and erase media that is still needed

To Make Second Copies

The first step in setting up second copies is to create a Volume Group called "copy2"
and placing some of the tape media in that Volume Group.  Note that you can put
a tape in this copy2 Volume Group even if it has been used already.

You can then add a second copy tape migration task to OSVault from the GUI by
creating a new tape migration and putting a "2" in the "Extra Copy" field.

How when the task scheduler runs a copy2 tape migration, every file that has already been copied to tape as a primary copy will be copied again to a tape in the copy2 Volume Group.  A separate migration report will be generated if you wish to see the results of that tape migration task.

What To Do With Second Copy Media

Generally, once a tape in the copy2 Volume Group is full (or almost full), you should EXPORT that tape and stored it somewhere else.  There is little reason to leave the tape in the library, since OSVault will not automatically use that tape.  

How To Restore A File From the Second Copy Media

We are working on a GUI button to help with this, but until then a CLI procedure is available to restore the second copy.  This CLI procedure swaps the two extended attributes that show where a file is stored on tape, changing the values of "mig_filename" to the second copy location, and putting the previous primary location in "copy2".  Here's an example:

First make sure the file has a second copy, either by viewing the File Location Report or with the following CLI:

      attr -g copy2 /cache/testfile

Then, you can make the next retrieval of the file go to the copy2 media with:

    PRIMARY=`attr -g mig_filename /cache/testfile`
    COPY2=`attr -g copy2 /cache/testfile`
    attr -s mig_filename -V $COPY2  /cache/testfile
    attr -s copy2 -V $PRIMARY /cache/testfile

Avoiding the Network Stage SCP Hang

NOTE:  This fix is for InfiniDisc/OSVault versions prior to 4.1.12

If you notice that a stage of a file back from network storage is not completing, this might be due to an SSL/SSH protocol issue where the network socket does not close correctly.  The problem has been traced to a standard I/O issue in the DMAPI daemon.     The SSH/SCP program that is invoked is run from an init.d script, rather than a user terminal session.  That causes certain network problems in the programs.

The fix for this is relatively simple, just ensure that the DMAPI daemon is started with its standard I/O redirected to /dev/null.  To do this, edit the /etc/init.d/dmapid file and change the dmapid start line as follows:

Change

/usr/local/bin/dmapid /cache  &

to

  /usr/local/bin/dmapid /cache >/dev/null </dev/null 2>&1 
&

Manually Migrating Files from InfiniDisc to Vault

In some cases, you may wish to move a lot of files stored on an InfiniDisc, via an external hard drive, rather than over a network connection to an OSVault server.   There are several reasons for doing this, including initial setup of an InfiniDisc/OSVault configuration or trying to reduce a large backlog on the InfiniDisc, or just to keep from swamping an Internet connection.

To manually move(migrate) files, you follow a four step process.  First you have to select and copy files to the external hard drive attached to the InfiniDisc system.  This is probably best done in  Command Line Interface (CLI) rather than via a GUI.  Then the second step is to ship the drives to the OSVault.  The third step is to copy off the files from the external drive to the OSVault cache; and the fourth step is to confirm that the files are properly copied and to mark them migrated on the InfiniDisc.

Step One
This assumes that you have hooked up an external drive and mounted it as /tmp/mnt on the InfiniDisc system.  Once this is done, you can run the "find" command on the InfiniDisc to move files to the external drive.  The "find" command used is something like this:
          find /cache/ -type f +mtime +30  -exec /root/test_migrated {} \; -exec cp {} /tmp/mnt/ \; -exec attr -s nomigrate -V 1 {} \;
This single command will search the /cache directory on your InfiniDisc for regular files ( -type f ) older than 30 days (+mtime +30), will see if they have been migrated (-exec /root/test_migrated {} \;) and if they are NOT migrated, will copy them to the external hard drive (-exec cp {} /tmp/mnt \;) and then mark the file as non-migratable (-exec attr -s nomigrate -V 1 {} \;).  Setting the "nomigrate" extended attribute on the file will insure that the InfiniDisc does not try to migrate the file while the external hard drive is in-transit.

Step Two
Unmount the external hard drive, disconnect it, and move it to the OSVault server and plug it in there.  It then gets mounted a /tmp/mnt on the OSVault system.

Step Three
You now need to copy the files from the external hard drive to the OSVault /cache for management.  There are many ways to do this, but a favorite for us is to use RSYNC, so that if anything happens to the copy operation, it can be restarted where it left off, rather than starting from scratch again.  The RSYNC command would be:
           rsync -avz --ignore-existing /tmp/mnt /cache
The "--ignore-existing" flag keeps the system from copying any files that may have already migrated to the OSVault system, or allows you to kill this command and restart it anytime.

Step Four
You now need to tell the InfiniDisc system to check that all the files marked "nomigrate" in step one are successfully copied to the OSVault and to mark them "copied".  You do this with the "net_verify" CLI command.  Net_+verify will search a directory and sub-directories, will find any files marked "nomigrate" and will run a SSH check to the OSVault to make sure that the file is truly copied to the OSVault.  The syntax of the net_verify command is:

                    net_verify -n -s /cache  -f

The "-n" flag tells net_verify to look for "nomigrate" files, and the "-f" flag tells net_verify to mark the files as "copied" if they are confirmed to be on the remote Vault.

The /root/test_migrated script may not be on your InfiniDisc box.  If you, you can create it with a text editor in the /root directory and mark it executable (chmod 0755 /root/test_migrated).  The contents of test_migrated are:

attr -q -g migrated "$1">/dev/null

if [ $? -eq 0 ]

  then

exit 1

  else

exit 0

fi

Labeling Individual Tape Media in OSVault

When adding new media to a tape library, you can always just put the media into unused storage slots using the importmedia CLI command, and then run "check_inventory -l /dev/sg# -f" to find that new media and internally label it.

However, sometimes you need to just put one piece of media in a library, or you just want to check the label written on a tape with the barcode affixed to the tape cartridge, or you want to overwrite the internal label on the tape.

KEEP IN MIND: If you overwrite the internal label on a tape, all files stored on that tape will be permanently lost.

There is an option to the "tape_inv" command to check just one piece of media.  If you run "tape_inv -l /dev/sg# -v VOLUMENAME -d drivename", substituting the volume name you wish to check, OSVault will search the library storage slots and import slots for that barcode label, and then will load that tape cartridge into the first drive and check for an internal tape label.  You must specify the drive name of the first drive, found in the output of the "listdrives" command.

Without any other options, running this command will just load the tape cartridge, read the internal label, and then return the tape to the storage/import slot it came from.

If you specify the "-f" flag to tape_inv, such as "tape_inv -l /dev/sg# -v A000001L5 -d /dev/st1 -f", the media will be loaded, and if not already labelled it will be labelled internally with the same label as the barcode on the outside of the tape cartridge.  This option might be easier to use than the "check_inventory" command at times.

If you wish to recycle an empty tape, you can use the "tape_inv" command to overwrite the existing label with a new label, with the "-f -x -x" option.  The two "-x" flags are required to indicate that you really mean to do this destructive operation.  Once this command completes, the tape media is empty in that cartridge, and future writes will start at the beginning of the tape.  It is also necessary to reset the tape cartridge's database entry to start using that tape again.  A command such as
   mysql -e "update archive.media set Free_Space=\"1666704934\" where Label=\"A00001L5\""
will set the free space on the tape to the default for LTO-5 media.

This tape_inv option is available in version 4.1.7 and later of OSVault.

Future releases of tape_inv will allow you to run the command when other tape operations are active and will not require you to specify the device name for the first tape drive.

Recovering OSVault Archived Files Directly From Tape

There are several situations that can occur that will require you to recover files directly from a piece of magnetic tape, rather than through the disk cache.  These include:
- Loss of the disk cache and related database due to catastrophic failure such as fire or damage
- Export and movement of tape cartridge(s) to another location and desire to import those tapes
- Recovery of previous versions of files that may have been overwritten after initial migration

This post will cover the methods to accomplish the last situation: scanning a tape and recovering all the files on that tape that may have been overwritten later.

Recovering Old Versions Of A File
Recovering an old version of a file can be the most tedious process if you do not know exactly when that version was overwritten.  OSVault keeps a history of the last time that a tape cartridge was mounted, so that information can help to reduce the number of tapes that must be scanned to find an old file.
To scan a tape and recover all the files on that tape back to disk, you use the "tape_dump" CLI command.  There are two ways to recover files from that tape, either as "stubs" or as whole files.  There are different reasons to use each option.  A "stub" recovery, using the "-s" option to "tape_dump", will create a stub file in the OSVault maintained file system but will not read in all the data portion of that file until that file stub is read later.  This allows you to import an entire tape while only using a small amount of disk storage.  Then you can browse the files recovered, and copy/use only the file(s) desired, while not reading all the other undesired files into  disk storage.
You can also recover all files directly from a tape onto disk storage.  This would be used if you are reading a tape into a non-OSVault file system or if you know you need all files stored on a particular tape.  Keep in mind that LTO-5 tapes can have 1.5TBytes of more of files on them.
If you are on an OSVault system, you can use the "-v" option to the "tape_dump" command to have the robotics load the select tape cartridge volume into an unused drive for recovery.  Other operations can be active on the OSVault system at the same tape, such as migration and staging.  So, if you wish to recover the stubs of all files on tape A0000325L5 to an OSVault cache directory called "/cache/recovered_files", you would use the CLI command;
    tape_dump -v A0000325L5 -s -f /cache/recovered_files
This will cause the tape to load into a drive, when a drive is available, and to scan the tape and create a stub in the directly /cache/recovered_files for every file it finds on the tape.  After the tape_dump completes, or is killed with a control-c, the files can be displayed with either the web browser, file browser or command line, and the desired file can be retrieved by copying or reading the file.  That reading/copying of the file will reload the tape in the drive and initiate a staging operation, bringing all the data portion of the file back into disk cache.

Exporting Tape Cartridges from an OSVault-Managed Tape Library

This posting is meant to just outline the steps you must follow to export an infrequently-used tape cartridge from your tape library that you wish to store on the shelf.  That shelf-stored (absent) tape cartridge is then not accessible for restoring files, but you will get notified if it is required.

Selecting a Tape To Export
Obviously, you probably don't want to export a tape cartridge that is used frequently.  If you export a tape cartridge that is needed by the system, the request for the file on that tape cartridge will get an error and you will need to do some manual steps to rectify the situation.
There is  a report called "Offline Media Report that will try to find "copy2" volume-group media to export, and also to tell you the least recently used tape cartridge media.  You can run the report by clicking the "Run Now" button as shown below.  The click the "Reports" menu icon to reload the page and look for your report in the "View Reports" side menu.

In the report, select a tape you wish to export.  Then proceed to the next step.

Exporting a Tape
First, don't just open the library and pull a tape out.  OSVault won't know what you did and will have a real problem correcting the situation.  You should use the "Media" menu icon and click the Label button of the tape you wish to export, such as tape A00425L4 in the example below.

Once you click on the tape button, another web-browser window will open with the tape cartridge information.   Just click the "Export Volume" to move the tape cartridge into the I/E slot (sometimes called the I/O slot) of the library.  This will also mark the media as "absent" in the OSVault database, so that requests for files only on that tape cartridge will be returned as an error, and the "Export Media Requests" subwindow will be able to display the status (that the cartridge needs to be imported).

Once you click on that button, the tape cartridge will be in the library import/export slot.  You can then open that slot (how you do this depends on the library model) and remove the tape.

Importing A Shelf Tape Cartridge Back Into the Library
Importing a absent tape cartridge is relatively simple.   You just put the cartridge you desire to import back into the Import/Export slot of the tape library, then go to the "Media" menu icon screen, click on the tape volume you just put into the I/E slot, and then click "Import Volume".  OSVault will then go to the import/export slot, verify that the correct tape is there, then move it to an open storage slot and change the database to show that the tape is now "resident".   Keep in mind that there must be at least one empty storage slot in the library to move the tape to, and that empty storage slot can't be the one that was just used by a tape that is presently in the drive.

What If You Don't Want to Export The Least Recently Used Volume
In some cases, you might have a directory that is all in one volume group and you want to export those tapes that have those files, since you won't be reading them again for a while.  In that case, use the  "File Location Report" option in the Report menu icon screen to get a full list of all files and what tapes they are on.  You can then use your browser tools to locate the files you wish and get a list of the tapes they are on, then export those tape cartridges using the above steps.

How Do I Know If A File is Requested From An Absent Tape Cartridge
There is a browser window you can open that updates continually, and will print out a requested "absent" tape cartridge label when needed.  The following shows what that screen looks like.  You open that browser window by clicking the "Export Media Requests" button on the "Media" screen.

OSVault GUI Version 5-0.31 Release

OSVaultGUI Version 5-0.31 is now available on the YUM repository (http://central.infinitevault.com/yum/base/5/x64_64 or http://central.infinitevault.com/yum/base/4/x64_64 depending on your CentOS level).

This release of the web GUI for OSVault has the following changes, in addition to maintenance changes:
- Color indication of absent/resident media (tape or optical)
- User selectable showing of floating file status windows in the Files section.  This is turned on/off in the Advanced Settings submenu of the Protection screen.
- Addition of a "Net Verify" report in the Reports section for verifying that local files have been migrated successfully to a remote Vault and that the files on the remote vault match the local files.
- Changes to the Offline Media Report to make it more useful in selecting what media to export from the library.
- Addition of a Check Inventory button to the Media GUI under Robotic Management to run the check_inventory CLI via the GUI.  This button will find new media placed in a TAPE library and will label it if it is new to the database.  It will also update media locations.  This operation is significantly faster than a full Inventory on the library.

Getting OSVault to Work from Open-Source

Every some often, we can a request from someone saying, basically, "I downloaded your code and am having troubles getting it to work, can you help me?". The short answer is "Yes, but we do that for pay, not for free."

The longer answer is, "Its not easy building a scalable, hierarchical storage system that can scale from 256GBytes to 10Petabytes, from scratch, and its takes a LOT of time and effort to get it working reliably."

Here are some of the common issues:
1. You need a working DMAPI/XDSM kernel. This code IS NOT in the LINUX mainstream kernel, and you need to get a working copy from somewhere. SGI is the best place, since they used to maintain the DMAPI code as a part of XFS. You will need to go back to the 2.6.24 or 2.6.27 kernel to do it easily. After 2.6.27, SGI removed the DMAPI code from XFS, so its much more difficult to get DMAPI into kernels after that. Plus, if you go back to 2.6.27, then you will have troubles with support for things like the Intel ICH10 chipset on your motherboard. If you have ICH10, then you can't boot from the DVD or the SATA drives with earlier kernels. So you need to patch in ICH10 support, which is another time-consuming process. And while you're at it, you probably need all the other LINUX patches between 2.6.27 and 2.6.39.

2. You need to use the XFS file system. Yes, I know that IBM JFS has DMAPI also, but that code doesn't work with a lot of things, like NFS server. XFS is all we use now for the OSVault work.

3. You need to use programs that DON'T use memory-mapped files. Memory-mapped files can bypass the DMAPI intercept for non-resident files. An example is VSFTPD. Use ProFTPd.

4. You have to get dmapid talking to your kernel. You should run the XFS test routines for DMAPI to make sure your kernel is OK. If you go straight to dmapid and there is a kernel problem, dmapid will probably just give an error and exit (or it may just sit there and do nothing since its not receiving events from the kernel). Make sure you understand how to mount with DMAPI options.

5. You need to use CentOS 4 or 5. We haven't tried it with other distributions. And you need to be careful about updates to the kernel from CentOS, which will replace your DMAPI kernel. Best to setup your own YUM server and put updates into that (and change your /etc/yum.repos.d directory contents).

6. You need to get your tape or optical libraries working correctly. The MCLIB directory has a lot of programs that must work with the library. Don't try migration until those work. You can try network migration, but you have to do all the SSH security setups first. If you don't see your drives and library in /proc/scsi/scsi, go back to square one and start all over again until you do. If "mcstat" doesn't work, do go any further until it does.

7. You need to initialize the MySQL database so that things like "listmedia" report correctly. There are scripts in the open source release that do this. If "listmedia" doesn't work, then migration will never work.

There are many more issues to address, but if you are just starting out with the open source distribution of OSVault from dvdvault.sourceforge.net, you can expect to spend anywhere from 80 to 1000 hours getting a new system up, depending on your skill with the LINUX kernel. The majority of that time is spent in kernel work and in hardware configuration setup. Keep in mind that correctly installing a full LINUX server with RAID boot can take 1-2 days after RAID initialization. Applying all the patches to a LINUX kernel for DMAPI (issue 1) takes at least 3 days of work, since you have to verify each one.

Now, if you want to build OSVault in a high-availability configuration, you're definitely ambitious! Our first HA configuration took 2 months to setup and fully test, even though we know all of the software intimately.

The open source code on Source Forge is the same code we use for our products. Our most active tape-based system using this code restores between 30 and 100 files from tape each day and migrates (writes) over 400GBytes of new data every day, and its been running at that rate for 20 months now using a Qualstar TLS-88264 tape library.

An Example of Backing Up Client Systems to OSVault Servers

In our office, we use a tape-based OSVault server as the destination for our system backups. That OSVault server uses a Qualstar 66-slot, LTO-3 tape library with two drives that we purchased several years ago. It has 4TBytes of spinning disk in a RAID-6 configuration. The backup programs we use (Toucan and Simple Backup Suite) can backup our systems to disk storage and are open-sourced, so they meet our needs adequately.

The structure of our offices is that our servers are located in our Datacenter in downtown Denver, but very few people actually work at the data center. Most work out of other offices with high-speed Internet links. So, the backups were initially setup to write directly to the OSVault server over our company VPN.

We found several limitations with that setup. The first was that the backups needed to be throttled at times, so that other Internet activity (web and VOIP for example) weren't degraded. Also, we found that we were transferring A LOT of data every day, usually at night. A full backup from one employee laptop could exceed 250GBytes.

Our solution was to put a small InfiniDisc system in the office as the backup destination. Those InfiniDisc systems generally have 500GBYtes or 1000GBytes of RAID-1 storage and can receive data at around 50MBytes/second, versus the 1MBytes/second direct Internet links. So, backups ran quickly to the InfiniDisc, then the InfiniDisc system would move the resulting backup at a reduced data rate to the OSVault server. The little InfiniDisc servers cost less than $650 each.

After a couple of years of doing this, we have found the solution works well for us. The costs are very low (less than 3 cents per gigabyte stored) and we get some great advantages:
- We get disk-to-disk-to-tape backup without any licensing costs for a traditional tape backup solution
- We have our most recent backup on the local InfiniDisc for quick restore on a Gigabit Ethernet network
- We don't have to move our incremental backups to the OSVault server, reducing Internet usage
- We have remote access anywhere to our backups stored on the OSVault server
- We can expand out storage capability anytime simply by adding more LTO-3 tape cartridges
- And, its a set-and-forget setup

Fixing the 1TByte inode problem in XFS file systems

If you have an XFS file system that you fill completely full, then add more hard disk space to it, you could run into the situation where the file system reports "no space available", but a "df" command shows plenty of space available. This is caused by the inability to allocate new inodes in that XFS filesystem.

Now, XFS dynamically allocates inodes, so you might be wondering how this could happen. The reason is that unless you say otherwise, inodes are limited to 32-bit values, which means they must fit in the first 1TByte of storage in the file system. But you completely filled that 1TByte previously, so XFS can't allocate more inodes now.

You could just switch to the "inode64" mount option and continue on, but that risks compatibility problems with NFS and with DMAPI (MySQL won't store a 64-bit inode properly in an "int" variable).

To get around this problem is actually not too difficult IF you know how. If you don't know how, it can be a very difficult time.

To fix the situation, you need to move files that occupy some of the storage in the first 1TByte of the file system. To do that, try the following:

1. Run xfs_info on your XFS mount point. For example:

[root@osvault ~]# xfs_info /cache/
meta-data=/dev/CACHE/CACHE               isize=256    agcount=375, agsize=64469728 blks
         =                               sectsz=512   attr=1
data     =                               bsize=4096   blocks=24157093888, imaxpct=25
         =                               sunit=0      swidth=0 blks, unwritten=1
naming   = version 2                     bsize=4096
log      =internal                       bsize=4096   blocks=32768, version=1
         =                               sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                           extsz=4096   blocks=0, rtextents=0

Notice that in this case, the "agsize" is 64 million blocks and the "bsize" is 4K, so the Allocation Groups are 256GBytes each. That means the first 1Tbyte of storage is in the first 4 Allocation Groups. So you want to find the largest files that are in that first 1TByte. If your allocation group is shown differently, then divide 1TByte by the allocation group size and get the number of allocation groups.

2. Figure out if those first allocation groups are full by running:

for ag in `seq 0 1 5`; do echo freespace in AG $ag; xfs_db -r -c "freesp -s -a $ag" /dev/CACHE/CACHE ; grep "total free"; done

If the total free blocks in each Allocation Group (AG) is less than about 40, then you can't create inodes in that allocation group. So now you want to find some files in that allocation group and move them out of the file system and then back in again. Its important that you "mv" the file, rather then "cp", so that the file is deleted from the XFS file system.

2. Now run xfs_bmap -v filename on all of the files in your filesystem. Yes, its tedious, so you probably want to script it. Just run an "ls /mountpoint" and send the output to a temporary file. Edit that temporary file and add the command to the beginning. Beware of spaces, quotes and parenthesis in your filenames.

3. Examine the output from all those xfs_bmap programs and search for lines with " 0 " (thats a space, then a numeral zero, then a space). That will find the files in Allocation Group 0. Repeat this for Allocation group 1, 2, and 3 (in this example). Every file in those first few allocation groups are candidates. All you have to do is copy the file to a temporary location, then copy it back to the XFS file system. The new instance of the file will be placed in storage in other allocation groups, and your XFS file system will now be able to allocate more inodes for new files.

The Challenges of Building a Really Large Storage System

Recently, dropping prices on disk drives have made me wonder if a really large storage system is practical based only on spinning disks, rather than a tape or optical library on the back end. So I decided to work up a 500TByte disk-based storage system that could be counted on to hold data securely for 5-10 years and see what the total cost would be.

Technical Challenges of Large Disk Farms
It is possible to buy a single RAID system today that supports 500TBytes of storage, but I have seen the following disastrous consequences of doing that approach:
1. A single firmware bug, in two instances, resulted in the lost of all storage in the RAID system. This happened on two different manufacturer's platforms and they were both major players in the RAID storage market
2. A bad batch of disk drives at one customer (155 drives out of 1400 installed) had a flawed FLASH memory chip (too much phosphorus in the ceramic coating for the chip caused the leads to the chip to be destroyed). On one VERY important day in 2000, 5 of those drives in a single RAID system (and 3 in one RAID set) decided to die at the same time. The end result was having to reload an Oracle database from tape for 4-8 hours, but the bigger result was a $500,000 fine by a federal judge for failure to process certain legal requirements in time.
3. Cost of these large RAID systems can be prohibitive, with drive trays with 16 drives costing $5000 to $8000, and electricity to drive them also being high (450 watts or higher). One customer swore they had fully redundant, adequate electrical supply for 11 drive trays, but within a week the single circuit breaker covering those 11 drive trays tripped and the system went down and required rewiring to get it back up. A very large school district's IT department was offline during this time.

A Disk-Only Solution
So, how about a different approach to 500TBytes of storage. Say we use 16-drive trays of 2TByte green disk drives at a cost of around $8000 each. Each tray can be RAID-6, giving us about 24TBytes per tray and the tray would be configured as iSCSI volumes. So we would need 21 trays to get our 500TBytes of storage, a couple of redundant network switches to hook everything together, and one tray configured with LINUX as a server. The total cost of all this storage would be about $180,000 (trays, switches, cabinet to hold them, power strips).

Now lets look at the utility costs to run this storage. Placing the storage at a colocation facility would run right around $56,000 per year for floor space and electricity, leaving out a network connection cost. The electricity becomes a major factor since the whole unit pulls more than 62 AMPs at 120 volts, running idle.

So a three year cost to operating this unit (assuming one cold stand-by system) would be about $348,000, or about 67 cents per gigabyte.

Of course backup costs and offsite storage costs aren't included in this calculation.

Now, lets compare this to a 500TByte OSVault implementation....

If we really wanted to stay low cost, we would put in a Qualstar 268 slot tape library and three of the same storage trays (24TByte each). That gives us about 430TBytes of storage in the tape library and 72Tbytes of spinning disk storage. Total purchase price of this unit (tape library, three trays of storage/server, 268 LTO-4 tapes, network switches) is about $105,000 and the OSVault software is free (unless you need our help installing it). The cost to put it in a colocation facility is around $7000 per year, so the three year cost is around $126,000, or about 1/3 the cost of a totally disk based solution.

Think Green
If you are thinking "green", the OSVault solution uses only 11 AMPs to manage 500TBytes of storage (at 120 volts).

So, for those that think the cost delta is not too large, remember that you don't have a disaster recovery plan priced in for the disk only solution, which could again double your costs.

I will grant you that the labor costs of the OSVault solution is greater, since you have to take tapes offsite (second copies, for example), but still you are looking at total managed prices for an OSVault solution at about 17 cents per gigabyte per year, versus around 50-70 cents per gigabyte per year for a totally disk storage system, in the best case.

Infinite Storage Anyone?
And the really nice part of the OSVault solution? It can double to 1Petabyte of storage for only 20% more cost. And that tape-based storage has at least 4 times the shelf life of disk based storage.

Logging NFS Server Activity in LINUX

I recently ran across this problem. I had a situation where files being accessed were being deleted by another system over the NFS protocol. With FTP and SMB/CIFS access, I could just enable greater logging and track the problem down, but with NFS server software on LINUX, there isn't this ability.

I researching it, I was able to determine that "tshark", a part of the Wireshark project, had the ability to capture and print out decoded NFS protocol packets. Running "tshark -R nfs" on the local server allowed me to see all NFS traffic on that local server. The output is formatted to be very readable and I could easily see the REMOVE packets that the remote server was sending, and from that I was able to determine which system was deleting files.

Since my problem seem to be related to an automated process starting at 11:30pm, I didn't have to run tshark for very long but the log was still over 4 Megabytes. If a longer monitoring time was required, I could have just run the following command: "tshark -R nfs | grep REMOVE >/tmp/REMOVE.log"
This would have only stored away NFS file deletions.

Hopefully, this posting will help someone in the future doing a search for a LINUX NFS Log ability (which right now leads to postings that say it can't be done).

Windows Active Directory Integration pitfalls

OSVault integrates into a Microsoft Windows Active Directory environment, but sometimes the SAMBA integration with user permissions can be challenging. For example, unless it is explicitly setup, the Administrator account does not always have unlimited permissions to change the security settings on a file stored in OSVault. This is because Administrator privileges are not always translated to "root" privileges in the OSVault/SAMBA software.

The problem comes from custom environments. Usually, the Domain Administrator is called "Administrator" and is in the group "Domain Admins". So, in the OSVault system, we map those user/group permissions to "root" permission. But if you change the names of that user or that group, OSVault won't know it, and will treat those users as a regular (non-privileged) user.

The file /etc/samba/smb.conf in the OSVault/InfiniDisc appliance has the following lines in it:

winbind separator = +
inherit acls
map acl inherit
username map = /etc/samba/smbusers

And in the Share portion
admin users = @"DOMAIN+Domain Admins", DOMAIN+Administrator

Notice that the "+" in the admin users line is the same character as in the winbind separator line. That is VERY important and not mentioned in a lot of on-line material

XFS B-Tree error and Building a New DMAPI kernel

In LINUX Kernel releases prior to, and including 2.6.29, a large number of file creations and deletions in a single directory uncovers a bug that causes the XFS file system to become unavailable. This is evidenced by the following error message in the system log (/var/log/messages):
infinidiscdev kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 3327 of file fs/xfs/xfs_btree.c. Caller 0xc032b241

This only occurs on 32-bit InfiniDisc systems, and not on any installed OSVault systems. So far the problem has only been seen in lab testing using the BackupPC software to backup a large number of client systems to an InfiniDisc system.

Since nearly all of our customers for OSVault archive systems do not usually do large amounts of file deletions, this problem would be hard to find in that environment.

The fix is to update to a newer kernel version, such as 2.6.31 or 2.6.33. Our InfiniDisc systems will update to 2.6.33, which we get from XFS.ORG but then have to make a couple of patches to. The reason for the patches is that we have legacy systems built with gcc 3.4 and the baseline 2.6.33 kernel will not compile with that kernel. For our more current InfiniDisc appliances built on CentOS 5.4, the changes to the standard kernel source are not required.

The primary change we make to 2.6.33 is in the mptsas message routine, where we move the mptsas_set_rphy routine before the mptsas_port_delete routine.

Printing on Blu-Ray and DVD Media with the HP D7560

We were able to pick up the HP D7560 printer on sale this week and it has the ability to print on ink-jet printable CD, DVD and Blu-Ray media.

Getting it to work took some tweaking. We use the glabels program to create our labels and for that program to print to the media without overrun, you need to create the right size template.

The template goes into the /usr/share/glabels/templates directory on your system. We called our file cd-tray-printing-templates.xml (the extension .xml is IMPORTANT). The contents of the file is as follows:

<?xml version="1.0"?>
<Glabels-templates>
<Template brand="OSStorage" part="CD" size="Other" width="131mm" height="131mm" _description="Disc in Tray (Small Centre)">
<meta category="label"/>
<meta category="media"/>
<label-cd id="0" radius="2.325in" hole="28.5pt" waste="0pt">
<layout nx="1" ny="1" x0="0mm" y0="0mm" dx="4.425in" dy="4.425in"/>
<markup-margin size="9pt"/>
</Label-cd>
</Template>
</Glabels-templates>





As far as plugging in the printer, our Ubuntu 9.10 system recognized it immediately and set it up for us automatically. We changed the default print source to the CD/DVD tray so no one would forget to use that in the future.