The Challenges of Building a Really Large Storage System

Recently, dropping prices on disk drives have made me wonder if a really large storage system is practical based only on spinning disks, rather than a tape or optical library on the back end. So I decided to work up a 500TByte disk-based storage system that could be counted on to hold data securely for 5-10 years and see what the total cost would be.

Technical Challenges of Large Disk Farms
It is possible to buy a single RAID system today that supports 500TBytes of storage, but I have seen the following disastrous consequences of doing that approach:
1. A single firmware bug, in two instances, resulted in the lost of all storage in the RAID system. This happened on two different manufacturer's platforms and they were both major players in the RAID storage market
2. A bad batch of disk drives at one customer (155 drives out of 1400 installed) had a flawed FLASH memory chip (too much phosphorus in the ceramic coating for the chip caused the leads to the chip to be destroyed). On one VERY important day in 2000, 5 of those drives in a single RAID system (and 3 in one RAID set) decided to die at the same time. The end result was having to reload an Oracle database from tape for 4-8 hours, but the bigger result was a $500,000 fine by a federal judge for failure to process certain legal requirements in time.
3. Cost of these large RAID systems can be prohibitive, with drive trays with 16 drives costing $5000 to $8000, and electricity to drive them also being high (450 watts or higher). One customer swore they had fully redundant, adequate electrical supply for 11 drive trays, but within a week the single circuit breaker covering those 11 drive trays tripped and the system went down and required rewiring to get it back up. A very large school district's IT department was offline during this time.

A Disk-Only Solution
So, how about a different approach to 500TBytes of storage. Say we use 16-drive trays of 2TByte green disk drives at a cost of around $8000 each. Each tray can be RAID-6, giving us about 24TBytes per tray and the tray would be configured as iSCSI volumes. So we would need 21 trays to get our 500TBytes of storage, a couple of redundant network switches to hook everything together, and one tray configured with LINUX as a server. The total cost of all this storage would be about $180,000 (trays, switches, cabinet to hold them, power strips).

Now lets look at the utility costs to run this storage. Placing the storage at a colocation facility would run right around $56,000 per year for floor space and electricity, leaving out a network connection cost. The electricity becomes a major factor since the whole unit pulls more than 62 AMPs at 120 volts, running idle.

So a three year cost to operating this unit (assuming one cold stand-by system) would be about $348,000, or about 67 cents per gigabyte.

Of course backup costs and offsite storage costs aren't included in this calculation.

Now, lets compare this to a 500TByte OSVault implementation....

If we really wanted to stay low cost, we would put in a Qualstar 268 slot tape library and three of the same storage trays (24TByte each). That gives us about 430TBytes of storage in the tape library and 72Tbytes of spinning disk storage. Total purchase price of this unit (tape library, three trays of storage/server, 268 LTO-4 tapes, network switches) is about $105,000 and the OSVault software is free (unless you need our help installing it). The cost to put it in a colocation facility is around $7000 per year, so the three year cost is around $126,000, or about 1/3 the cost of a totally disk based solution.

Think Green
If you are thinking "green", the OSVault solution uses only 11 AMPs to manage 500TBytes of storage (at 120 volts).

So, for those that think the cost delta is not too large, remember that you don't have a disaster recovery plan priced in for the disk only solution, which could again double your costs.

I will grant you that the labor costs of the OSVault solution is greater, since you have to take tapes offsite (second copies, for example), but still you are looking at total managed prices for an OSVault solution at about 17 cents per gigabyte per year, versus around 50-70 cents per gigabyte per year for a totally disk storage system, in the best case.

Infinite Storage Anyone?
And the really nice part of the OSVault solution? It can double to 1Petabyte of storage for only 20% more cost. And that tape-based storage has at least 4 times the shelf life of disk based storage.