Manually Migrating Files from InfiniDisc to Vault

In some cases, you may wish to move a lot of files stored on an InfiniDisc, via an external hard drive, rather than over a network connection to an OSVault server.   There are several reasons for doing this, including initial setup of an InfiniDisc/OSVault configuration or trying to reduce a large backlog on the InfiniDisc, or just to keep from swamping an Internet connection.

To manually move(migrate) files, you follow a four step process.  First you have to select and copy files to the external hard drive attached to the InfiniDisc system.  This is probably best done in  Command Line Interface (CLI) rather than via a GUI.  Then the second step is to ship the drives to the OSVault.  The third step is to copy off the files from the external drive to the OSVault cache; and the fourth step is to confirm that the files are properly copied and to mark them migrated on the InfiniDisc.

Step One
This assumes that you have hooked up an external drive and mounted it as /tmp/mnt on the InfiniDisc system.  Once this is done, you can run the "find" command on the InfiniDisc to move files to the external drive.  The "find" command used is something like this:
          find /cache/ -type f +mtime +30  -exec /root/test_migrated {} \; -exec cp {} /tmp/mnt/ \; -exec attr -s nomigrate -V 1 {} \;
This single command will search the /cache directory on your InfiniDisc for regular files ( -type f ) older than 30 days (+mtime +30), will see if they have been migrated (-exec /root/test_migrated {} \;) and if they are NOT migrated, will copy them to the external hard drive (-exec cp {} /tmp/mnt \;) and then mark the file as non-migratable (-exec attr -s nomigrate -V 1 {} \;).  Setting the "nomigrate" extended attribute on the file will insure that the InfiniDisc does not try to migrate the file while the external hard drive is in-transit.

Step Two
Unmount the external hard drive, disconnect it, and move it to the OSVault server and plug it in there.  It then gets mounted a /tmp/mnt on the OSVault system.

Step Three
You now need to copy the files from the external hard drive to the OSVault /cache for management.  There are many ways to do this, but a favorite for us is to use RSYNC, so that if anything happens to the copy operation, it can be restarted where it left off, rather than starting from scratch again.  The RSYNC command would be:
           rsync -avz --ignore-existing /tmp/mnt /cache
The "--ignore-existing" flag keeps the system from copying any files that may have already migrated to the OSVault system, or allows you to kill this command and restart it anytime.

Step Four
You now need to tell the InfiniDisc system to check that all the files marked "nomigrate" in step one are successfully copied to the OSVault and to mark them "copied".  You do this with the "net_verify" CLI command.  Net_+verify will search a directory and sub-directories, will find any files marked "nomigrate" and will run a SSH check to the OSVault to make sure that the file is truly copied to the OSVault.  The syntax of the net_verify command is:

                    net_verify -n -s /cache  -f

The "-n" flag tells net_verify to look for "nomigrate" files, and the "-f" flag tells net_verify to mark the files as "copied" if they are confirmed to be on the remote Vault.

The /root/test_migrated script may not be on your InfiniDisc box.  If you, you can create it with a text editor in the /root directory and mark it executable (chmod 0755 /root/test_migrated).  The contents of test_migrated are:

attr -q -g migrated "$1">/dev/null

if [ $? -eq 0 ]

  then

exit 1

  else

exit 0

fi

Labeling Individual Tape Media in OSVault

When adding new media to a tape library, you can always just put the media into unused storage slots using the importmedia CLI command, and then run "check_inventory -l /dev/sg# -f" to find that new media and internally label it.

However, sometimes you need to just put one piece of media in a library, or you just want to check the label written on a tape with the barcode affixed to the tape cartridge, or you want to overwrite the internal label on the tape.

KEEP IN MIND: If you overwrite the internal label on a tape, all files stored on that tape will be permanently lost.

There is an option to the "tape_inv" command to check just one piece of media.  If you run "tape_inv -l /dev/sg# -v VOLUMENAME -d drivename", substituting the volume name you wish to check, OSVault will search the library storage slots and import slots for that barcode label, and then will load that tape cartridge into the first drive and check for an internal tape label.  You must specify the drive name of the first drive, found in the output of the "listdrives" command.

Without any other options, running this command will just load the tape cartridge, read the internal label, and then return the tape to the storage/import slot it came from.

If you specify the "-f" flag to tape_inv, such as "tape_inv -l /dev/sg# -v A000001L5 -d /dev/st1 -f", the media will be loaded, and if not already labelled it will be labelled internally with the same label as the barcode on the outside of the tape cartridge.  This option might be easier to use than the "check_inventory" command at times.

If you wish to recycle an empty tape, you can use the "tape_inv" command to overwrite the existing label with a new label, with the "-f -x -x" option.  The two "-x" flags are required to indicate that you really mean to do this destructive operation.  Once this command completes, the tape media is empty in that cartridge, and future writes will start at the beginning of the tape.  It is also necessary to reset the tape cartridge's database entry to start using that tape again.  A command such as
   mysql -e "update archive.media set Free_Space=\"1666704934\" where Label=\"A00001L5\""
will set the free space on the tape to the default for LTO-5 media.

This tape_inv option is available in version 4.1.7 and later of OSVault.

Future releases of tape_inv will allow you to run the command when other tape operations are active and will not require you to specify the device name for the first tape drive.

Recovering OSVault Archived Files Directly From Tape

There are several situations that can occur that will require you to recover files directly from a piece of magnetic tape, rather than through the disk cache.  These include:
- Loss of the disk cache and related database due to catastrophic failure such as fire or damage
- Export and movement of tape cartridge(s) to another location and desire to import those tapes
- Recovery of previous versions of files that may have been overwritten after initial migration

This post will cover the methods to accomplish the last situation: scanning a tape and recovering all the files on that tape that may have been overwritten later.

Recovering Old Versions Of A File
Recovering an old version of a file can be the most tedious process if you do not know exactly when that version was overwritten.  OSVault keeps a history of the last time that a tape cartridge was mounted, so that information can help to reduce the number of tapes that must be scanned to find an old file.
To scan a tape and recover all the files on that tape back to disk, you use the "tape_dump" CLI command.  There are two ways to recover files from that tape, either as "stubs" or as whole files.  There are different reasons to use each option.  A "stub" recovery, using the "-s" option to "tape_dump", will create a stub file in the OSVault maintained file system but will not read in all the data portion of that file until that file stub is read later.  This allows you to import an entire tape while only using a small amount of disk storage.  Then you can browse the files recovered, and copy/use only the file(s) desired, while not reading all the other undesired files into  disk storage.
You can also recover all files directly from a tape onto disk storage.  This would be used if you are reading a tape into a non-OSVault file system or if you know you need all files stored on a particular tape.  Keep in mind that LTO-5 tapes can have 1.5TBytes of more of files on them.
If you are on an OSVault system, you can use the "-v" option to the "tape_dump" command to have the robotics load the select tape cartridge volume into an unused drive for recovery.  Other operations can be active on the OSVault system at the same tape, such as migration and staging.  So, if you wish to recover the stubs of all files on tape A0000325L5 to an OSVault cache directory called "/cache/recovered_files", you would use the CLI command;
    tape_dump -v A0000325L5 -s -f /cache/recovered_files
This will cause the tape to load into a drive, when a drive is available, and to scan the tape and create a stub in the directly /cache/recovered_files for every file it finds on the tape.  After the tape_dump completes, or is killed with a control-c, the files can be displayed with either the web browser, file browser or command line, and the desired file can be retrieved by copying or reading the file.  That reading/copying of the file will reload the tape in the drive and initiate a staging operation, bringing all the data portion of the file back into disk cache.

Exporting Tape Cartridges from an OSVault-Managed Tape Library

This posting is meant to just outline the steps you must follow to export an infrequently-used tape cartridge from your tape library that you wish to store on the shelf.  That shelf-stored (absent) tape cartridge is then not accessible for restoring files, but you will get notified if it is required.

Selecting a Tape To Export
Obviously, you probably don't want to export a tape cartridge that is used frequently.  If you export a tape cartridge that is needed by the system, the request for the file on that tape cartridge will get an error and you will need to do some manual steps to rectify the situation.
There is  a report called "Offline Media Report that will try to find "copy2" volume-group media to export, and also to tell you the least recently used tape cartridge media.  You can run the report by clicking the "Run Now" button as shown below.  The click the "Reports" menu icon to reload the page and look for your report in the "View Reports" side menu.

In the report, select a tape you wish to export.  Then proceed to the next step.

Exporting a Tape
First, don't just open the library and pull a tape out.  OSVault won't know what you did and will have a real problem correcting the situation.  You should use the "Media" menu icon and click the Label button of the tape you wish to export, such as tape A00425L4 in the example below.

Once you click on the tape button, another web-browser window will open with the tape cartridge information.   Just click the "Export Volume" to move the tape cartridge into the I/E slot (sometimes called the I/O slot) of the library.  This will also mark the media as "absent" in the OSVault database, so that requests for files only on that tape cartridge will be returned as an error, and the "Export Media Requests" subwindow will be able to display the status (that the cartridge needs to be imported).

Once you click on that button, the tape cartridge will be in the library import/export slot.  You can then open that slot (how you do this depends on the library model) and remove the tape.

Importing A Shelf Tape Cartridge Back Into the Library
Importing a absent tape cartridge is relatively simple.   You just put the cartridge you desire to import back into the Import/Export slot of the tape library, then go to the "Media" menu icon screen, click on the tape volume you just put into the I/E slot, and then click "Import Volume".  OSVault will then go to the import/export slot, verify that the correct tape is there, then move it to an open storage slot and change the database to show that the tape is now "resident".   Keep in mind that there must be at least one empty storage slot in the library to move the tape to, and that empty storage slot can't be the one that was just used by a tape that is presently in the drive.

What If You Don't Want to Export The Least Recently Used Volume
In some cases, you might have a directory that is all in one volume group and you want to export those tapes that have those files, since you won't be reading them again for a while.  In that case, use the  "File Location Report" option in the Report menu icon screen to get a full list of all files and what tapes they are on.  You can then use your browser tools to locate the files you wish and get a list of the tapes they are on, then export those tape cartridges using the above steps.

How Do I Know If A File is Requested From An Absent Tape Cartridge
There is a browser window you can open that updates continually, and will print out a requested "absent" tape cartridge label when needed.  The following shows what that screen looks like.  You open that browser window by clicking the "Export Media Requests" button on the "Media" screen.

OSVault GUI Version 5-0.31 Release

OSVaultGUI Version 5-0.31 is now available on the YUM repository (http://central.infinitevault.com/yum/base/5/x64_64 or http://central.infinitevault.com/yum/base/4/x64_64 depending on your CentOS level).

This release of the web GUI for OSVault has the following changes, in addition to maintenance changes:
- Color indication of absent/resident media (tape or optical)
- User selectable showing of floating file status windows in the Files section.  This is turned on/off in the Advanced Settings submenu of the Protection screen.
- Addition of a "Net Verify" report in the Reports section for verifying that local files have been migrated successfully to a remote Vault and that the files on the remote vault match the local files.
- Changes to the Offline Media Report to make it more useful in selecting what media to export from the library.
- Addition of a Check Inventory button to the Media GUI under Robotic Management to run the check_inventory CLI via the GUI.  This button will find new media placed in a TAPE library and will label it if it is new to the database.  It will also update media locations.  This operation is significantly faster than a full Inventory on the library.

Getting OSVault to Work from Open-Source

Every some often, we can a request from someone saying, basically, "I downloaded your code and am having troubles getting it to work, can you help me?". The short answer is "Yes, but we do that for pay, not for free."

The longer answer is, "Its not easy building a scalable, hierarchical storage system that can scale from 256GBytes to 10Petabytes, from scratch, and its takes a LOT of time and effort to get it working reliably."

Here are some of the common issues:
1. You need a working DMAPI/XDSM kernel. This code IS NOT in the LINUX mainstream kernel, and you need to get a working copy from somewhere. SGI is the best place, since they used to maintain the DMAPI code as a part of XFS. You will need to go back to the 2.6.24 or 2.6.27 kernel to do it easily. After 2.6.27, SGI removed the DMAPI code from XFS, so its much more difficult to get DMAPI into kernels after that. Plus, if you go back to 2.6.27, then you will have troubles with support for things like the Intel ICH10 chipset on your motherboard. If you have ICH10, then you can't boot from the DVD or the SATA drives with earlier kernels. So you need to patch in ICH10 support, which is another time-consuming process. And while you're at it, you probably need all the other LINUX patches between 2.6.27 and 2.6.39.

2. You need to use the XFS file system. Yes, I know that IBM JFS has DMAPI also, but that code doesn't work with a lot of things, like NFS server. XFS is all we use now for the OSVault work.

3. You need to use programs that DON'T use memory-mapped files. Memory-mapped files can bypass the DMAPI intercept for non-resident files. An example is VSFTPD. Use ProFTPd.

4. You have to get dmapid talking to your kernel. You should run the XFS test routines for DMAPI to make sure your kernel is OK. If you go straight to dmapid and there is a kernel problem, dmapid will probably just give an error and exit (or it may just sit there and do nothing since its not receiving events from the kernel). Make sure you understand how to mount with DMAPI options.

5. You need to use CentOS 4 or 5. We haven't tried it with other distributions. And you need to be careful about updates to the kernel from CentOS, which will replace your DMAPI kernel. Best to setup your own YUM server and put updates into that (and change your /etc/yum.repos.d directory contents).

6. You need to get your tape or optical libraries working correctly. The MCLIB directory has a lot of programs that must work with the library. Don't try migration until those work. You can try network migration, but you have to do all the SSH security setups first. If you don't see your drives and library in /proc/scsi/scsi, go back to square one and start all over again until you do. If "mcstat" doesn't work, do go any further until it does.

7. You need to initialize the MySQL database so that things like "listmedia" report correctly. There are scripts in the open source release that do this. If "listmedia" doesn't work, then migration will never work.

There are many more issues to address, but if you are just starting out with the open source distribution of OSVault from dvdvault.sourceforge.net, you can expect to spend anywhere from 80 to 1000 hours getting a new system up, depending on your skill with the LINUX kernel. The majority of that time is spent in kernel work and in hardware configuration setup. Keep in mind that correctly installing a full LINUX server with RAID boot can take 1-2 days after RAID initialization. Applying all the patches to a LINUX kernel for DMAPI (issue 1) takes at least 3 days of work, since you have to verify each one.

Now, if you want to build OSVault in a high-availability configuration, you're definitely ambitious! Our first HA configuration took 2 months to setup and fully test, even though we know all of the software intimately.

The open source code on Source Forge is the same code we use for our products. Our most active tape-based system using this code restores between 30 and 100 files from tape each day and migrates (writes) over 400GBytes of new data every day, and its been running at that rate for 20 months now using a Qualstar TLS-88264 tape library.

An Example of Backing Up Client Systems to OSVault Servers

In our office, we use a tape-based OSVault server as the destination for our system backups. That OSVault server uses a Qualstar 66-slot, LTO-3 tape library with two drives that we purchased several years ago. It has 4TBytes of spinning disk in a RAID-6 configuration. The backup programs we use (Toucan and Simple Backup Suite) can backup our systems to disk storage and are open-sourced, so they meet our needs adequately.

The structure of our offices is that our servers are located in our Datacenter in downtown Denver, but very few people actually work at the data center. Most work out of other offices with high-speed Internet links. So, the backups were initially setup to write directly to the OSVault server over our company VPN.

We found several limitations with that setup. The first was that the backups needed to be throttled at times, so that other Internet activity (web and VOIP for example) weren't degraded. Also, we found that we were transferring A LOT of data every day, usually at night. A full backup from one employee laptop could exceed 250GBytes.

Our solution was to put a small InfiniDisc system in the office as the backup destination. Those InfiniDisc systems generally have 500GBYtes or 1000GBytes of RAID-1 storage and can receive data at around 50MBytes/second, versus the 1MBytes/second direct Internet links. So, backups ran quickly to the InfiniDisc, then the InfiniDisc system would move the resulting backup at a reduced data rate to the OSVault server. The little InfiniDisc servers cost less than $650 each.

After a couple of years of doing this, we have found the solution works well for us. The costs are very low (less than 3 cents per gigabyte stored) and we get some great advantages:
- We get disk-to-disk-to-tape backup without any licensing costs for a traditional tape backup solution
- We have our most recent backup on the local InfiniDisc for quick restore on a Gigabit Ethernet network
- We don't have to move our incremental backups to the OSVault server, reducing Internet usage
- We have remote access anywhere to our backups stored on the OSVault server
- We can expand out storage capability anytime simply by adding more LTO-3 tape cartridges
- And, its a set-and-forget setup