XFS B-Tree error and Building a New DMAPI kernel

In LINUX Kernel releases prior to, and including 2.6.29, a large number of file creations and deletions in a single directory uncovers a bug that causes the XFS file system to become unavailable. This is evidenced by the following error message in the system log (/var/log/messages):
infinidiscdev kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 3327 of file fs/xfs/xfs_btree.c. Caller 0xc032b241

This only occurs on 32-bit InfiniDisc systems, and not on any installed OSVault systems. So far the problem has only been seen in lab testing using the BackupPC software to backup a large number of client systems to an InfiniDisc system.

Since nearly all of our customers for OSVault archive systems do not usually do large amounts of file deletions, this problem would be hard to find in that environment.

The fix is to update to a newer kernel version, such as 2.6.31 or 2.6.33. Our InfiniDisc systems will update to 2.6.33, which we get from XFS.ORG but then have to make a couple of patches to. The reason for the patches is that we have legacy systems built with gcc 3.4 and the baseline 2.6.33 kernel will not compile with that kernel. For our more current InfiniDisc appliances built on CentOS 5.4, the changes to the standard kernel source are not required.

The primary change we make to 2.6.33 is in the mptsas message routine, where we move the mptsas_set_rphy routine before the mptsas_port_delete routine.

Printing on Blu-Ray and DVD Media with the HP D7560

We were able to pick up the HP D7560 printer on sale this week and it has the ability to print on ink-jet printable CD, DVD and Blu-Ray media.

Getting it to work took some tweaking. We use the glabels program to create our labels and for that program to print to the media without overrun, you need to create the right size template.

The template goes into the /usr/share/glabels/templates directory on your system. We called our file cd-tray-printing-templates.xml (the extension .xml is IMPORTANT). The contents of the file is as follows:

<?xml version="1.0"?>
<Glabels-templates>
<Template brand="OSStorage" part="CD" size="Other" width="131mm" height="131mm" _description="Disc in Tray (Small Centre)">
<meta category="label"/>
<meta category="media"/>
<label-cd id="0" radius="2.325in" hole="28.5pt" waste="0pt">
<layout nx="1" ny="1" x0="0mm" y0="0mm" dx="4.425in" dy="4.425in"/>
<markup-margin size="9pt"/>
</Label-cd>
</Template>
</Glabels-templates>





As far as plugging in the printer, our Ubuntu 9.10 system recognized it immediately and set it up for us automatically. We changed the default print source to the CD/DVD tray so no one would forget to use that in the future.

Notes on Joining OSVault to a Windows 2003 Active Directory

The hardest part about joining OSVault to a Windows 2003 Active Directory setup is understanding the configuration of your Domain and making sure the Windows 2003 server is setup correctly. If you have an existing Active Directory setup (with a Domain Controller), then the methods used to join OSVault to that domain is detailed further on.

Edit /etc/samba.smb.conf and make sure you have the following
WORKGROUP = MYDOMAIN
REALM = MYDOMAIN.MYCOMPANY.COM
SECURITY = ADS
netbios name = MYCOMPUTER

Make sure your /etc/krb5.conf file has at least the following:
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = MYDOMAIN.MYCOMPANY.COM
dns_lookup_realm = false
dns_lookup_kdc = false

[realms]
MYDOMAIN.MYCOMPANY.COM = {
kdc = MYDOMAIN.MYCOMPANY.COM:88
admin_server = MYDOMAIN.MYCOMPANY.COM:749
default_domain = MYDOMAIN.MYCOMPANY.COM
}

[domain_realm]
.kerberos.server = MYDOMAIN.MYCOMPANY.COM

[kdc]
profile = /var/kerberos/krb5kdc/kdc.conf

[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}


From command line you should do the following:

• kinit administrator@MYDOMAIN.MYCOMPANY.COM

where administrator is any account on the Domain that has "Domain Admin" privileges

• net join -U administrator

again, where administrator is any account on the Domain that has "Domain Admin" privileges

Your OSVault hostname MUST match your full host name on the PDC (i.e. the output from `hostname` command is MYMACHINE.MYDOMAIN.MYCOMPANY.COM


Go to http://www.joeware.net/freetools to get a set of tools for your Windows 2003 server, such as adfind, admod, oldcmp, findexpacc, and memberof.

Power Consumption in a Very Large Vault Architecture

Our largest single box server architecture is a 24-drive, Quad Processor system with 16GBytes of memory. Fully filled with 1.5TByte Seagate drives, this monster has around 25TBytes of usable high speed storage space. Overall performance is about 500MBytes/second across 6 gigabit Ethernet ports. This server can be directly attached to a tape or Blu-Ray library with SCSI or Fibre Channel interfaces.

With high speed access to an extremely large immediate storage space, and access to hundreds of terabytes of files, the default configuration will store over 1 Billion files, each file up to a terabyte is size, for a total Vault storage space is excess of 1 Zettabyte (thats 1,000,000,000,000,000,000,000 bytes).

The really exciting part about this is the total power consumption for this server. Less than 600 watts typically, as shown in the picture below of a KilloWatt monitor on the power line of a fully configured system running all 24 drives in a mixture of read and write operations. The system is running all power supplies on a single cable in this example, in a normal configuration you would run four redundant power cables.

Controlling network migration traffic

When migration is configured to copy files to a network storage device, rather than to optical disc or tape, it can be preferred to limit the amount of bandwidth that OSVault uses. This writeup explains how to throttle network usage by migration.
For example, if you are migrating data from a remote office to a central site, and that office has a 7MBit/second (A)DSL link, then you uplink speed would probably be around 800Kbits/second. Running a migration from that office to a central site without some throttling might make response time unacceptable for email and web browsing at that office.
To throttle a migration, there is a configuration parameter, mig_delay, stored in /etc/sysconfig/osvault/mig_delay that sets an amount of time to pause in data transfers. The pause is executed after every block of data, where the default size of a block of data is 1MByte. The vault in mig_delay in in microseconds, so putting a vault of 1,000,000 (1 million) in mig_delay would cause migration to pause for 1 second after every 1MByte (default) of data transferred.
In recent releases of migration, you can set the default data block size with the -i option to migration. So, to set migration to send 64KBytes of data, then pause for one second, you would run migration with the -i 65536 option and set /etc/sysconfig/osvault/mig_delay to 1000000.

About archives and memory resources

In some ways, archive systems are uniquely different than run of the mill NAS storage. A case in point is some observations from a recent customer installation. The customer wanted to store about 1 billion (that's 1024 million) records on Blu-Ray optical disc. The average file size was about 5Kbytes (thats right, about 5120 bytes). So, if you do simple math, you think you can store about 10 million files on a single 50GByte Blu-Ray optical disc. But you would be wrong, because you also need to store the file names on that disc and that overhead (along with other file system overhead) means you store less than 7 million files on the disk. That's quite a bit of overhead.

Now, this is an extreme example, but it shows that planning is required for many installations.

Also, some other observations are interesting. In generating the ISO9660 file system (level 4, with iso8859 character set), the system uses over 8Gbytes of system memory during that file system generation (just before the optical disc is actually burned). OSVault uses the mkisofs program, called by growisofs, to create an ISO9660 file system.

Running a directory operation on a large archive with 1.6million files in a single directory will cause the "ls" program to use 1Gbyte of memory to store the file names alphabetically.

Blu-Ray drive firmware levels

I have been running into problems with older firmware on Blu-Ray drives installed in both DISC/NSM and ASACA libraries. The problems manifest themselves in the inability of the drive to write to media from Verbatim and SONY corporation, primarily BD-RE media, and the inability to write to Panasonic BD-R media. This is firmware that is installed on Panasonic SW-5582 model Blu-ray drives. These are drives with parallel ATA interfaces, connected to SCSI thru IODATA cards.

The following firmware levels are known to work with all media we have tested so far:
BKF1 - (ASACA libraries) works with all tested media (TDK, Verbatim, Panasonic, both BD-RE and BD-R)
BKE8 -(DISC libraries) works with all tested media (TDK, Verbatim, Panasonic, both BD-RE and BD-R)

The following firmware levels have produced problems with various types of Blu-Ray media:
BZA6 - only does Panasonic BD-RE (installed at some ASACA sites)
BDB2 - Seen on some DISC and on some ASACA libraries