Getting OSVault to Work from Open-Source

Every some often, we can a request from someone saying, basically, "I downloaded your code and am having troubles getting it to work, can you help me?". The short answer is "Yes, but we do that for pay, not for free."

The longer answer is, "Its not easy building a scalable, hierarchical storage system that can scale from 256GBytes to 10Petabytes, from scratch, and its takes a LOT of time and effort to get it working reliably."

Here are some of the common issues:
1. You need a working DMAPI/XDSM kernel. This code IS NOT in the LINUX mainstream kernel, and you need to get a working copy from somewhere. SGI is the best place, since they used to maintain the DMAPI code as a part of XFS. You will need to go back to the 2.6.24 or 2.6.27 kernel to do it easily. After 2.6.27, SGI removed the DMAPI code from XFS, so its much more difficult to get DMAPI into kernels after that. Plus, if you go back to 2.6.27, then you will have troubles with support for things like the Intel ICH10 chipset on your motherboard. If you have ICH10, then you can't boot from the DVD or the SATA drives with earlier kernels. So you need to patch in ICH10 support, which is another time-consuming process. And while you're at it, you probably need all the other LINUX patches between 2.6.27 and 2.6.39.

2. You need to use the XFS file system. Yes, I know that IBM JFS has DMAPI also, but that code doesn't work with a lot of things, like NFS server. XFS is all we use now for the OSVault work.

3. You need to use programs that DON'T use memory-mapped files. Memory-mapped files can bypass the DMAPI intercept for non-resident files. An example is VSFTPD. Use ProFTPd.

4. You have to get dmapid talking to your kernel. You should run the XFS test routines for DMAPI to make sure your kernel is OK. If you go straight to dmapid and there is a kernel problem, dmapid will probably just give an error and exit (or it may just sit there and do nothing since its not receiving events from the kernel). Make sure you understand how to mount with DMAPI options.

5. You need to use CentOS 4 or 5. We haven't tried it with other distributions. And you need to be careful about updates to the kernel from CentOS, which will replace your DMAPI kernel. Best to setup your own YUM server and put updates into that (and change your /etc/yum.repos.d directory contents).

6. You need to get your tape or optical libraries working correctly. The MCLIB directory has a lot of programs that must work with the library. Don't try migration until those work. You can try network migration, but you have to do all the SSH security setups first. If you don't see your drives and library in /proc/scsi/scsi, go back to square one and start all over again until you do. If "mcstat" doesn't work, do go any further until it does.

7. You need to initialize the MySQL database so that things like "listmedia" report correctly. There are scripts in the open source release that do this. If "listmedia" doesn't work, then migration will never work.

There are many more issues to address, but if you are just starting out with the open source distribution of OSVault from dvdvault.sourceforge.net, you can expect to spend anywhere from 80 to 1000 hours getting a new system up, depending on your skill with the LINUX kernel. The majority of that time is spent in kernel work and in hardware configuration setup. Keep in mind that correctly installing a full LINUX server with RAID boot can take 1-2 days after RAID initialization. Applying all the patches to a LINUX kernel for DMAPI (issue 1) takes at least 3 days of work, since you have to verify each one.

Now, if you want to build OSVault in a high-availability configuration, you're definitely ambitious! Our first HA configuration took 2 months to setup and fully test, even though we know all of the software intimately.

The open source code on Source Forge is the same code we use for our products. Our most active tape-based system using this code restores between 30 and 100 files from tape each day and migrates (writes) over 400GBytes of new data every day, and its been running at that rate for 20 months now using a Qualstar TLS-88264 tape library.