Current mood:

accomplished

My experience using Linux software RAID on my Addonics storage tower.
Note: During this process, I emailed Addonics Tech Support for assistance, and in addition to their (quick) email response I received a phone call the next day to discuss the situation. Apparently this was a configuration they had not encountered before and so I offered to write up the results. (If you are not here for that technical information, now would be a good time to leave. :) )

Hardware:
  Addonics JBOD USB tower with four drives (2x300GB, 1x320GB, 1x180GB).
  IBM Thinkpad Pentium 3-M laptop with USB 2.0 served through a PCMCIA card.

Software:
  Xubuntu 7.10 with all updates as of 4/2008.

Goal:  
  RAID 5 with the first three drives (around 300GB each.)

HOWTO:
  http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch26_:_Linux_Software_RAID.

Tools:
  /proc/mdstat
  /sbin/mdadm
  /sbin/mkfs
  /bin/dmesg
  /sbin/fdisk

Attempt 1: At first I wasn't aware that there was a lot of initializing to do for the array, and attempted to continue to mkfs as soon as the mdadm command completed. The mkfs repeatedly failed:


root@bow:~# mkfs -t xfs /dev/md0
...
mkfs.xfs: pwrite64 failed: Input/output error
root@bow:~#


...or ...


root@bow:~# mkfs -t ext3 /dev/md0
...
Warning: could not erase sector 2: Attempt to write block from filesystem resulted in short write
...
Warning: could not read block 0: Attempt to read block from filesystem resulted in short read
Warning: could not erase sector 0: Attempt to write block from filesystem resulted in short write
...
ext2fs_update_bb_inode: Illegal triply indirect block found while setting bad block inode
root@bow:~#


Attempt 2: Around this time I noticed that the CPU meter was still pegged even though I wasn't doing anything. Seeing mdadm still running made me look up mdadm's man page, which directed me to /proc/mdstat, which indicated the initialization was still proceeding:

tdb@bow:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdb1[3] sdc1[1] sdd1[0]
      586067072 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
      [===>.................]  recovery = 16.4% (48195320/293033536) finish=1010.3min speed=4035K/sec

unused devices: 
tdb@bow:~$

Eventually I noticed that the recovery percentage never went above around 2% (unlike the above sample, taken later) before getting back to a completed state:

tdb@bow:~$ cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sdb1[2] sdc1[1] sdd1[0]
      586067072 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

unused devices: 
tdb@bow:~$

This process took about 10-15 minutes. Since there were no error messages in /proc/mdstat or dmesg, I assumed that the work was all front-loaded and the "other" 98% of the work was trivial, like setting up a partition table, rather than calculating and writing parity blocks. However, the errors from mkfs didn't change, and the result was still unmountable. I also tried altering the partition type from fd (Linux raid automount) to 83 (plain Linux) but there was no effect.

Attempt 3: At this point I was starting to wonder if one of the drives had gone bad, although I wasn't sold on the idea because they had worked right up to the start of this process. Still, I repartitioned the drives to start at cylinder 100 (leaving the first 100 cylinders blank on each disk) to skip past any initial problem spots on the drives. There were still no changes to the error messages, indicating the problem did not shift geometry with the partition change, which in turn suggested that the problem was probably not physical.

Attempt 4: I emailed Addonics tech support. Though they had not previously encountered my configuration, within that conversation there was a clue that the creation of a RAID 5 array takes a long time ("3 500GB drives takes as much as 24 hours.") I went back to thinking of the mdadm command as the culprit, wondering why my initialization step terminated at 2%. Again, attempting to force some kind of change in geometry, I recreated the RAID array with the drives in reverse order: sdd1, sdc1, sdb1.

This worked. Completely. I was rather gobsmacked. I was able to create an XFS filesystem and mount it.

I have since repartitioned the drives back to use the complete drive geometry (20gb is wasted on one drive, shucky-darn) and everything is still working fine. I also note that the partition type (83 or fd) doesn't seem to affect the creation of the array, although it may have other consequences for mdadm and related tools (don't know yet.)

tdb@bow:~$ mount
...
/dev/md0 on /media/AddonicsRaid type xfs (rw,noexec,nosuid,nodev)
tdb@bow:~$ df -h /media/AddonicsRaid
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              559G  1.5M  559G   1% /media/AddonicsRaid
tdb@bow:~$ sudo mdadm --query --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Fri Apr 18 10:56:23 2008
     Raid Level : raid5
     Array Size : 586067072 (558.92 GiB 600.13 GB)
  Used Dev Size : 293033536 (279.46 GiB 300.07 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sat Apr 19 08:44:38 2008
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : b3c7a2c5:d065d41c:80d35bfa:21d4c790 (local to host bow)
         Events : 0.4

    Number   Major   Minor   RaidDevice State
       0       8       49        0      active sync   /dev/sdd1
       1       8       33        1      active sync   /dev/sdc1
       2       8       17        2      active sync   /dev/sdb1
tdb@bow:~$ touch /media/AddonicsRaid/testit
tdb@bow:~$ ls /media/AddonicsRaid
testit
tdb@bow:~$

Et voila! A new use for those old drives. See next entry for relative performance measurements...