Well I finally got an RPM build with LINUX 2.6.24-rc3. Now upgrades to the kernel in OSVault can be done much more easily (and safely). Still a lot of testing to run on it. The new kernel also includes a change for the DM_REMOVE_EVENT (deleting a file) that removes the migration database entry of files that are already migrated, then deleted. This keeps the database smaller and is generally better practice. IBM also put this same modification into their DMAPI kernel for the GPFS software.
The mod for the inodes goes in vfs_vnodeops.c like the following:
file_vp = XFS_ITOV(ip);
if (DM_EVENT_ENABLED(dp, DM_EVENT_REMOVE)) {
error = XFS_SEND_NAMESP(mp, DM_EVENT_REMOVE, dir_vp,
DM_RIGHT_NULL, file_vp, DM_RIGHT_NULL,
name, NULL, dm_di_mode, 0, 0);
if (error)
return error;
}
This passes the inode to the dmapid program so that it doesn't have to thrash around trying to find out which file was deleted.
Saturday, February 16, 2008
Thursday, December 20, 2007
Monday, September 17, 2007
Generating Video Thumbnails during Migration
OK, so we want the next released version of OSVault to generate video thumbnails during migration from hard drive to optical/tape media. Here's how the prototype implementation works:
1. During migration, the file type is determined using the "file" command. This has been extended to recognize various video formats.
2. If the file being migrated is a video file that we support, then during migration (not just as a separate task), each block copied out is run thru the ffmpeg program. ffmpeg generates video thumbnails with the command
ffmpeg -y -i video.avi -f singlejpeg -ss 10 -vframes 1 -s 150×100 -an thumbnail.jpg
3. The output thumbnail is placed in /var/www/html for the file browser to find. WHen you use the file browser, the thumbnail is displayed if it exists.
1. During migration, the file type is determined using the "file" command. This has been extended to recognize various video formats.
2. If the file being migrated is a video file that we support, then during migration (not just as a separate task), each block copied out is run thru the ffmpeg program. ffmpeg generates video thumbnails with the command
ffmpeg -y -i video.avi -f singlejpeg -ss 10 -vframes 1 -s 150×100 -an thumbnail.jpg
3. The output thumbnail is placed in /var/www/html for the file browser to find. WHen you use the file browser, the thumbnail is displayed if it exists.
Friday, December 1, 2006
Reading direct from Optical Media
OSVault can be configured to either stage files back to hard disk on access or to read directly from optical media for files not resident on hard disk.
Generally, the OSVault appliance needs to be installed either one way or the other, it won't support both types of file access. Once a system is configured as a staging (also called DMAPI) system, it is difficult to change that configuration. Also, once a system is configured for direct reads from optical disk, it is difficult to convert it to stage files.
Direct reads from media are accomplished via symbolic links that are on the hard disk and point to the optical disk file. The migration.c source code has a DEFINE called PURGEIT that, if defined, makes the migration program able to migrate files and replace them with symbolic links. When using direct reads/symbolic links, you do not use the "purge" program. The "purge" program is only used with staging configurations.
Generally, reading directly from optical media or staging files works well with Windows shares, but NFS shares require files to be staged back to disk. If you wish to read directly from optical disk and use and NFS share, you will have to NFS mount both the /cache share and the /archive share.
Generally, the OSVault appliance needs to be installed either one way or the other, it won't support both types of file access. Once a system is configured as a staging (also called DMAPI) system, it is difficult to change that configuration. Also, once a system is configured for direct reads from optical disk, it is difficult to convert it to stage files.
Direct reads from media are accomplished via symbolic links that are on the hard disk and point to the optical disk file. The migration.c source code has a DEFINE called PURGEIT that, if defined, makes the migration program able to migrate files and replace them with symbolic links. When using direct reads/symbolic links, you do not use the "purge" program. The "purge" program is only used with staging configurations.
Generally, reading directly from optical media or staging files works well with Windows shares, but NFS shares require files to be staged back to disk. If you wish to read directly from optical disk and use and NFS share, you will have to NFS mount both the /cache share and the /archive share.
I mixed up the media in the library! HELP!
If you removed media from the optical library and placed it back in the library and aren't sure if you put it in the right place, you can remedy this easily.
First, you need to have the library physically check each slot to see if media is there. Most optical libraries have the ability to do this from some menu. We also did a program called "initelem" that will cause a library to check each slot for media.
After running "initelem", you can then verify volume labels of media in each slot by running "inventory". "inventory" will load every piece of media into a drive and check to see if it is formatted and what its label is. Then "inventory" will update the media database (archive.media in MySQL) with the location and name of each piece of media.
Time wise, running a full "initelem" followed by "inventory" can take a while. An ASACA 750 slot library with three drives will take about 45 minutes to run "initelem" and then about 1 minute per loaded media per drive for "inventory", or 4 hours for 750 pieces of media.
First, you need to have the library physically check each slot to see if media is there. Most optical libraries have the ability to do this from some menu. We also did a program called "initelem" that will cause a library to check each slot for media.
After running "initelem", you can then verify volume labels of media in each slot by running "inventory". "inventory" will load every piece of media into a drive and check to see if it is formatted and what its label is. Then "inventory" will update the media database (archive.media in MySQL) with the location and name of each piece of media.
Time wise, running a full "initelem" followed by "inventory" can take a while. An ASACA 750 slot library with three drives will take about 45 minutes to run "initelem" and then about 1 minute per loaded media per drive for "inventory", or 4 hours for 750 pieces of media.
Where can I get OSVault Source Code?
When we build an archive appliance for a customer, we put the source code in the /root/optivault directory of that appliance. We also post OSVault releases on the web at http://dvdvault.sourceforge.net
Keep in mind that OSVault is based on software that is released under the GPL and the original software in OSVault is released under GPL. So if you use it to build a commercial product, any changes you make to OSVault must be released to the public and published by you. Your commercial product must also credit the authors in your copyright notices.
Keep in mind that OSVault is based on software that is released under the GPL and the original software in OSVault is released under GPL. So if you use it to build a commercial product, any changes you make to OSVault must be released to the public and published by you. Your commercial product must also credit the authors in your copyright notices.
OSVault support for WORM media
Testing is just about complete on this update to OSVault to support BD-R, DVD+RW, DVD+R, DVD-R and CD-RW media. There is a new migraiton program called "batch_migration" that complements the existing "migration" program.
Batch_migration is used just like "migration" and will copy files in a block from hard disk cache to optical media. "batch_migration" uses the same database of optical media to figure out which media to use (archive.media) and uses the same "inventory" program to figure out what media is loaded into the library. The "batch_migration" has been tested with 50GByte BD-R, 25GByte BD-R, 4.7GByte DVD+RW, 700MB CD-RW, and 4.7GByte DVD+R.
"batch_migration" reads thru the files stored in /cache and locates either the oldest or largest files to migrate. Oldest is the default. By default, batch_migration will only a maximum of 2000 files onto a piece of optical media. "batch_migration" will try to fill the optical media, but you should specify a minimum amount of data required before an optical burn begins to avoid wasting optical disk space.
"batch_migration" requires temporary hard disk space to create an ISO9660 image before the burn of the optical media starts, so it checks to make sure that free space exists. You can specify a different workspace location with the "-w" flag (the default is /var/tmp). DON'T PUT YOUR WORKSPACE in /cache, it will run slowly.
The current testing version of "batch_migration" is set to only do a single session to optical media. This works in most deep archive situations, but we realize that some customers may not generate enough data at a time to hold off migrating to optical only when a full disk is available. So we will be adding multi-session support in the next release. Multi-session support means, for example, that while using 50GByte media, you can move smaller amounts of data to a single disk over several sessions, rather than waiting for a full 50Gbytes on hard disk before migrating.
Batch_migration is used just like "migration" and will copy files in a block from hard disk cache to optical media. "batch_migration" uses the same database of optical media to figure out which media to use (archive.media) and uses the same "inventory" program to figure out what media is loaded into the library. The "batch_migration" has been tested with 50GByte BD-R, 25GByte BD-R, 4.7GByte DVD+RW, 700MB CD-RW, and 4.7GByte DVD+R.
"batch_migration" reads thru the files stored in /cache and locates either the oldest or largest files to migrate. Oldest is the default. By default, batch_migration will only a maximum of 2000 files onto a piece of optical media. "batch_migration" will try to fill the optical media, but you should specify a minimum amount of data required before an optical burn begins to avoid wasting optical disk space.
"batch_migration" requires temporary hard disk space to create an ISO9660 image before the burn of the optical media starts, so it checks to make sure that free space exists. You can specify a different workspace location with the "-w" flag (the default is /var/tmp). DON'T PUT YOUR WORKSPACE in /cache, it will run slowly.
The current testing version of "batch_migration" is set to only do a single session to optical media. This works in most deep archive situations, but we realize that some customers may not generate enough data at a time to hold off migrating to optical only when a full disk is available. So we will be adding multi-session support in the next release. Multi-session support means, for example, that while using 50GByte media, you can move smaller amounts of data to a single disk over several sessions, rather than waiting for a full 50Gbytes on hard disk before migrating.
Subscribe to:
Posts (Atom)