OSVault can be configured to either stage files back to hard disk on access or to read directly from optical media for files not resident on hard disk.
Generally, the OSVault appliance needs to be installed either one way or the other, it won't support both types of file access. Once a system is configured as a staging (also called DMAPI) system, it is difficult to change that configuration. Also, once a system is configured for direct reads from optical disk, it is difficult to convert it to stage files.
Direct reads from media are accomplished via symbolic links that are on the hard disk and point to the optical disk file. The migration.c source code has a DEFINE called PURGEIT that, if defined, makes the migration program able to migrate files and replace them with symbolic links. When using direct reads/symbolic links, you do not use the "purge" program. The "purge" program is only used with staging configurations.
Generally, reading directly from optical media or staging files works well with Windows shares, but NFS shares require files to be staged back to disk. If you wish to read directly from optical disk and use and NFS share, you will have to NFS mount both the /cache share and the /archive share.
Friday, December 1, 2006
I mixed up the media in the library! HELP!
If you removed media from the optical library and placed it back in the library and aren't sure if you put it in the right place, you can remedy this easily.
First, you need to have the library physically check each slot to see if media is there. Most optical libraries have the ability to do this from some menu. We also did a program called "initelem" that will cause a library to check each slot for media.
After running "initelem", you can then verify volume labels of media in each slot by running "inventory". "inventory" will load every piece of media into a drive and check to see if it is formatted and what its label is. Then "inventory" will update the media database (archive.media in MySQL) with the location and name of each piece of media.
Time wise, running a full "initelem" followed by "inventory" can take a while. An ASACA 750 slot library with three drives will take about 45 minutes to run "initelem" and then about 1 minute per loaded media per drive for "inventory", or 4 hours for 750 pieces of media.
First, you need to have the library physically check each slot to see if media is there. Most optical libraries have the ability to do this from some menu. We also did a program called "initelem" that will cause a library to check each slot for media.
After running "initelem", you can then verify volume labels of media in each slot by running "inventory". "inventory" will load every piece of media into a drive and check to see if it is formatted and what its label is. Then "inventory" will update the media database (archive.media in MySQL) with the location and name of each piece of media.
Time wise, running a full "initelem" followed by "inventory" can take a while. An ASACA 750 slot library with three drives will take about 45 minutes to run "initelem" and then about 1 minute per loaded media per drive for "inventory", or 4 hours for 750 pieces of media.
Where can I get OSVault Source Code?
When we build an archive appliance for a customer, we put the source code in the /root/optivault directory of that appliance. We also post OSVault releases on the web at http://dvdvault.sourceforge.net
Keep in mind that OSVault is based on software that is released under the GPL and the original software in OSVault is released under GPL. So if you use it to build a commercial product, any changes you make to OSVault must be released to the public and published by you. Your commercial product must also credit the authors in your copyright notices.
Keep in mind that OSVault is based on software that is released under the GPL and the original software in OSVault is released under GPL. So if you use it to build a commercial product, any changes you make to OSVault must be released to the public and published by you. Your commercial product must also credit the authors in your copyright notices.
OSVault support for WORM media
Testing is just about complete on this update to OSVault to support BD-R, DVD+RW, DVD+R, DVD-R and CD-RW media. There is a new migraiton program called "batch_migration" that complements the existing "migration" program.
Batch_migration is used just like "migration" and will copy files in a block from hard disk cache to optical media. "batch_migration" uses the same database of optical media to figure out which media to use (archive.media) and uses the same "inventory" program to figure out what media is loaded into the library. The "batch_migration" has been tested with 50GByte BD-R, 25GByte BD-R, 4.7GByte DVD+RW, 700MB CD-RW, and 4.7GByte DVD+R.
"batch_migration" reads thru the files stored in /cache and locates either the oldest or largest files to migrate. Oldest is the default. By default, batch_migration will only a maximum of 2000 files onto a piece of optical media. "batch_migration" will try to fill the optical media, but you should specify a minimum amount of data required before an optical burn begins to avoid wasting optical disk space.
"batch_migration" requires temporary hard disk space to create an ISO9660 image before the burn of the optical media starts, so it checks to make sure that free space exists. You can specify a different workspace location with the "-w" flag (the default is /var/tmp). DON'T PUT YOUR WORKSPACE in /cache, it will run slowly.
The current testing version of "batch_migration" is set to only do a single session to optical media. This works in most deep archive situations, but we realize that some customers may not generate enough data at a time to hold off migrating to optical only when a full disk is available. So we will be adding multi-session support in the next release. Multi-session support means, for example, that while using 50GByte media, you can move smaller amounts of data to a single disk over several sessions, rather than waiting for a full 50Gbytes on hard disk before migrating.
Batch_migration is used just like "migration" and will copy files in a block from hard disk cache to optical media. "batch_migration" uses the same database of optical media to figure out which media to use (archive.media) and uses the same "inventory" program to figure out what media is loaded into the library. The "batch_migration" has been tested with 50GByte BD-R, 25GByte BD-R, 4.7GByte DVD+RW, 700MB CD-RW, and 4.7GByte DVD+R.
"batch_migration" reads thru the files stored in /cache and locates either the oldest or largest files to migrate. Oldest is the default. By default, batch_migration will only a maximum of 2000 files onto a piece of optical media. "batch_migration" will try to fill the optical media, but you should specify a minimum amount of data required before an optical burn begins to avoid wasting optical disk space.
"batch_migration" requires temporary hard disk space to create an ISO9660 image before the burn of the optical media starts, so it checks to make sure that free space exists. You can specify a different workspace location with the "-w" flag (the default is /var/tmp). DON'T PUT YOUR WORKSPACE in /cache, it will run slowly.
The current testing version of "batch_migration" is set to only do a single session to optical media. This works in most deep archive situations, but we realize that some customers may not generate enough data at a time to hold off migrating to optical only when a full disk is available. So we will be adding multi-session support in the next release. Multi-session support means, for example, that while using 50GByte media, you can move smaller amounts of data to a single disk over several sessions, rather than waiting for a full 50Gbytes on hard disk before migrating.
How OSVault works with multiple libraries
So, the upcoming 4.20 release of OSVault is planned to support multiple ASACA or DISC Blu-Ray libraries attached to a single server. Some people have had questions about how this will work.
If you have two or more Blu-Ray libraries attached to a single OSVault server, the software will store files on optical media, by default, starting with the first slot of the first library. The libraries are named during installation as library1 or library2.
If you wish to store files on optical media in a different order than this, you can setup "Volume Groups" in the libraries. Each piece of media can belong to the default Volume Group 0, or you can put pieces of media in different Volume Groups. Volume Groups have names you create,
such as "HD Video", "SD Video", "Post Edit", etc.
For example, you could put the first 100 pieces of media in library1 into a volume group called "Post Edit", and the second 100 pieces of media in library1 in a group called "Production", and put the last 100 pieces of media in both libraries in a Volume Group of 200 pieces called "Backup Files". The grouping is very flexible and can be changed at any time.
When you store files in the OSVault server, all files reside in a single name space with subdirectories. OSVault moves individual files to optical media and preserves the naming on the optical media. When a file is migrated (copied) to optical media, the migration command is
scheduled to run on a Volume Group. After a file is migrated it is then purged to free up more hard disk space, but the file still exists on the hard disk but its data space is not there. If you then try to read that file later, the OSVault server will copy the file data back to hard disk space in the same location it was originally. There is a database file on the OSVault server called "archive.files" which keeps track of this mapping of file name to optical location. There is a copy of that database mapping contained in the file name entry on the hard disk, also.
If you have two or more Blu-Ray libraries attached to a single OSVault server, the software will store files on optical media, by default, starting with the first slot of the first library. The libraries are named during installation as library1 or library2.
If you wish to store files on optical media in a different order than this, you can setup "Volume Groups" in the libraries. Each piece of media can belong to the default Volume Group 0, or you can put pieces of media in different Volume Groups. Volume Groups have names you create,
such as "HD Video", "SD Video", "Post Edit", etc.
For example, you could put the first 100 pieces of media in library1 into a volume group called "Post Edit", and the second 100 pieces of media in library1 in a group called "Production", and put the last 100 pieces of media in both libraries in a Volume Group of 200 pieces called "Backup Files". The grouping is very flexible and can be changed at any time.
When you store files in the OSVault server, all files reside in a single name space with subdirectories. OSVault moves individual files to optical media and preserves the naming on the optical media. When a file is migrated (copied) to optical media, the migration command is
scheduled to run on a Volume Group. After a file is migrated it is then purged to free up more hard disk space, but the file still exists on the hard disk but its data space is not there. If you then try to read that file later, the OSVault server will copy the file data back to hard disk space in the same location it was originally. There is a database file on the OSVault server called "archive.files" which keeps track of this mapping of file name to optical location. There is a copy of that database mapping contained in the file name entry on the hard disk, also.
Subscribe to:
Posts (Atom)