The Challenges of Building a Really Large Storage System

Recently, dropping prices on disk drives have made me wonder if a really large storage system is practical based only on spinning disks, rather than a tape or optical library on the back end. So I decided to work up a 500TByte disk-based storage system that could be counted on to hold data securely for 5-10 years and see what the total cost would be.

Technical Challenges of Large Disk Farms
It is possible to buy a single RAID system today that supports 500TBytes of storage, but I have seen the following disastrous consequences of doing that approach:
1. A single firmware bug, in two instances, resulted in the lost of all storage in the RAID system. This happened on two different manufacturer's platforms and they were both major players in the RAID storage market
2. A bad batch of disk drives at one customer (155 drives out of 1400 installed) had a flawed FLASH memory chip (too much phosphorus in the ceramic coating for the chip caused the leads to the chip to be destroyed). On one VERY important day in 2000, 5 of those drives in a single RAID system (and 3 in one RAID set) decided to die at the same time. The end result was having to reload an Oracle database from tape for 4-8 hours, but the bigger result was a $500,000 fine by a federal judge for failure to process certain legal requirements in time.
3. Cost of these large RAID systems can be prohibitive, with drive trays with 16 drives costing $5000 to $8000, and electricity to drive them also being high (450 watts or higher). One customer swore they had fully redundant, adequate electrical supply for 11 drive trays, but within a week the single circuit breaker covering those 11 drive trays tripped and the system went down and required rewiring to get it back up. A very large school district's IT department was offline during this time.

A Disk-Only Solution
So, how about a different approach to 500TBytes of storage. Say we use 16-drive trays of 2TByte green disk drives at a cost of around $8000 each. Each tray can be RAID-6, giving us about 24TBytes per tray and the tray would be configured as iSCSI volumes. So we would need 21 trays to get our 500TBytes of storage, a couple of redundant network switches to hook everything together, and one tray configured with LINUX as a server. The total cost of all this storage would be about $180,000 (trays, switches, cabinet to hold them, power strips).

Now lets look at the utility costs to run this storage. Placing the storage at a colocation facility would run right around $56,000 per year for floor space and electricity, leaving out a network connection cost. The electricity becomes a major factor since the whole unit pulls more than 62 AMPs at 120 volts, running idle.

So a three year cost to operating this unit (assuming one cold stand-by system) would be about $348,000, or about 67 cents per gigabyte.

Of course backup costs and offsite storage costs aren't included in this calculation.

Now, lets compare this to a 500TByte OSVault implementation....

If we really wanted to stay low cost, we would put in a Qualstar 268 slot tape library and three of the same storage trays (24TByte each). That gives us about 430TBytes of storage in the tape library and 72Tbytes of spinning disk storage. Total purchase price of this unit (tape library, three trays of storage/server, 268 LTO-4 tapes, network switches) is about $105,000 and the OSVault software is free (unless you need our help installing it). The cost to put it in a colocation facility is around $7000 per year, so the three year cost is around $126,000, or about 1/3 the cost of a totally disk based solution.

Think Green
If you are thinking "green", the OSVault solution uses only 11 AMPs to manage 500TBytes of storage (at 120 volts).

So, for those that think the cost delta is not too large, remember that you don't have a disaster recovery plan priced in for the disk only solution, which could again double your costs.

I will grant you that the labor costs of the OSVault solution is greater, since you have to take tapes offsite (second copies, for example), but still you are looking at total managed prices for an OSVault solution at about 17 cents per gigabyte per year, versus around 50-70 cents per gigabyte per year for a totally disk storage system, in the best case.

Infinite Storage Anyone?
And the really nice part of the OSVault solution? It can double to 1Petabyte of storage for only 20% more cost. And that tape-based storage has at least 4 times the shelf life of disk based storage.

Logging NFS Server Activity in LINUX

I recently ran across this problem. I had a situation where files being accessed were being deleted by another system over the NFS protocol. With FTP and SMB/CIFS access, I could just enable greater logging and track the problem down, but with NFS server software on LINUX, there isn't this ability.

I researching it, I was able to determine that "tshark", a part of the Wireshark project, had the ability to capture and print out decoded NFS protocol packets. Running "tshark -R nfs" on the local server allowed me to see all NFS traffic on that local server. The output is formatted to be very readable and I could easily see the REMOVE packets that the remote server was sending, and from that I was able to determine which system was deleting files.

Since my problem seem to be related to an automated process starting at 11:30pm, I didn't have to run tshark for very long but the log was still over 4 Megabytes. If a longer monitoring time was required, I could have just run the following command: "tshark -R nfs | grep REMOVE >/tmp/REMOVE.log"
This would have only stored away NFS file deletions.

Hopefully, this posting will help someone in the future doing a search for a LINUX NFS Log ability (which right now leads to postings that say it can't be done).

Windows Active Directory Integration pitfalls

OSVault integrates into a Microsoft Windows Active Directory environment, but sometimes the SAMBA integration with user permissions can be challenging. For example, unless it is explicitly setup, the Administrator account does not always have unlimited permissions to change the security settings on a file stored in OSVault. This is because Administrator privileges are not always translated to "root" privileges in the OSVault/SAMBA software.

The problem comes from custom environments. Usually, the Domain Administrator is called "Administrator" and is in the group "Domain Admins". So, in the OSVault system, we map those user/group permissions to "root" permission. But if you change the names of that user or that group, OSVault won't know it, and will treat those users as a regular (non-privileged) user.

The file /etc/samba/smb.conf in the OSVault/InfiniDisc appliance has the following lines in it:

winbind separator = +
inherit acls
map acl inherit
username map = /etc/samba/smbusers

And in the Share portion
admin users = @"DOMAIN+Domain Admins", DOMAIN+Administrator

Notice that the "+" in the admin users line is the same character as in the winbind separator line. That is VERY important and not mentioned in a lot of on-line material

XFS B-Tree error and Building a New DMAPI kernel

In LINUX Kernel releases prior to, and including 2.6.29, a large number of file creations and deletions in a single directory uncovers a bug that causes the XFS file system to become unavailable. This is evidenced by the following error message in the system log (/var/log/messages):
infinidiscdev kernel: XFS internal error XFS_WANT_CORRUPTED_GOTO at line 3327 of file fs/xfs/xfs_btree.c. Caller 0xc032b241

This only occurs on 32-bit InfiniDisc systems, and not on any installed OSVault systems. So far the problem has only been seen in lab testing using the BackupPC software to backup a large number of client systems to an InfiniDisc system.

Since nearly all of our customers for OSVault archive systems do not usually do large amounts of file deletions, this problem would be hard to find in that environment.

The fix is to update to a newer kernel version, such as 2.6.31 or 2.6.33. Our InfiniDisc systems will update to 2.6.33, which we get from XFS.ORG but then have to make a couple of patches to. The reason for the patches is that we have legacy systems built with gcc 3.4 and the baseline 2.6.33 kernel will not compile with that kernel. For our more current InfiniDisc appliances built on CentOS 5.4, the changes to the standard kernel source are not required.

The primary change we make to 2.6.33 is in the mptsas message routine, where we move the mptsas_set_rphy routine before the mptsas_port_delete routine.

Printing on Blu-Ray and DVD Media with the HP D7560

We were able to pick up the HP D7560 printer on sale this week and it has the ability to print on ink-jet printable CD, DVD and Blu-Ray media.

Getting it to work took some tweaking. We use the glabels program to create our labels and for that program to print to the media without overrun, you need to create the right size template.

The template goes into the /usr/share/glabels/templates directory on your system. We called our file cd-tray-printing-templates.xml (the extension .xml is IMPORTANT). The contents of the file is as follows:

<?xml version="1.0"?>
<Glabels-templates>
<Template brand="OSStorage" part="CD" size="Other" width="131mm" height="131mm" _description="Disc in Tray (Small Centre)">
<meta category="label"/>
<meta category="media"/>
<label-cd id="0" radius="2.325in" hole="28.5pt" waste="0pt">
<layout nx="1" ny="1" x0="0mm" y0="0mm" dx="4.425in" dy="4.425in"/>
<markup-margin size="9pt"/>
</Label-cd>
</Template>
</Glabels-templates>





As far as plugging in the printer, our Ubuntu 9.10 system recognized it immediately and set it up for us automatically. We changed the default print source to the CD/DVD tray so no one would forget to use that in the future.

Notes on Joining OSVault to a Windows 2003 Active Directory

The hardest part about joining OSVault to a Windows 2003 Active Directory setup is understanding the configuration of your Domain and making sure the Windows 2003 server is setup correctly. If you have an existing Active Directory setup (with a Domain Controller), then the methods used to join OSVault to that domain is detailed further on.

Edit /etc/samba.smb.conf and make sure you have the following
WORKGROUP = MYDOMAIN
REALM = MYDOMAIN.MYCOMPANY.COM
SECURITY = ADS
netbios name = MYCOMPUTER

Make sure your /etc/krb5.conf file has at least the following:
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log

[libdefaults]
default_realm = MYDOMAIN.MYCOMPANY.COM
dns_lookup_realm = false
dns_lookup_kdc = false

[realms]
MYDOMAIN.MYCOMPANY.COM = {
kdc = MYDOMAIN.MYCOMPANY.COM:88
admin_server = MYDOMAIN.MYCOMPANY.COM:749
default_domain = MYDOMAIN.MYCOMPANY.COM
}

[domain_realm]
.kerberos.server = MYDOMAIN.MYCOMPANY.COM

[kdc]
profile = /var/kerberos/krb5kdc/kdc.conf

[appdefaults]
pam = {
debug = false
ticket_lifetime = 36000
renew_lifetime = 36000
forwardable = true
krb4_convert = false
}


From command line you should do the following:

• kinit administrator@MYDOMAIN.MYCOMPANY.COM

where administrator is any account on the Domain that has "Domain Admin" privileges

• net join -U administrator

again, where administrator is any account on the Domain that has "Domain Admin" privileges

Your OSVault hostname MUST match your full host name on the PDC (i.e. the output from `hostname` command is MYMACHINE.MYDOMAIN.MYCOMPANY.COM


Go to http://www.joeware.net/freetools to get a set of tools for your Windows 2003 server, such as adfind, admod, oldcmp, findexpacc, and memberof.

Power Consumption in a Very Large Vault Architecture

Our largest single box server architecture is a 24-drive, Quad Processor system with 16GBytes of memory. Fully filled with 1.5TByte Seagate drives, this monster has around 25TBytes of usable high speed storage space. Overall performance is about 500MBytes/second across 6 gigabit Ethernet ports. This server can be directly attached to a tape or Blu-Ray library with SCSI or Fibre Channel interfaces.

With high speed access to an extremely large immediate storage space, and access to hundreds of terabytes of files, the default configuration will store over 1 Billion files, each file up to a terabyte is size, for a total Vault storage space is excess of 1 Zettabyte (thats 1,000,000,000,000,000,000,000 bytes).

The really exciting part about this is the total power consumption for this server. Less than 600 watts typically, as shown in the picture below of a KilloWatt monitor on the power line of a fully configured system running all 24 drives in a mixture of read and write operations. The system is running all power supplies on a single cable in this example, in a normal configuration you would run four redundant power cables.