Wednesday, February 15, 2012

Sans Digital TowerRAID TR8M-B (wip)

Testing out this enclosure in an attempt to move to a ZFS 6-disk dual parity cluster.

WARNING:  THIS WILL BE A MESS OF A POST while I'm working through the various tests.  I will cleanup and organize my thoughts after I'm happy with the correct set of tests and order for this.

eSATA supports up to 5 devices via port multiplication, and the TR8M has 8 disk slots, split 4 disk slots per eSATA port (2), with a rather cheap insertion mechanism (but it seems to be working fine so far in my tests).  I have it temporarily connect directly to the sole esata port my motherboard offers

I also have acquired a StarTech PEXESAT32 card, based on a Marvel 9128 chip, for dual eSATA ports. When booting the system, the post message says to hit Ctrl+M to enter setup but I just can't get this to work ever.  Identified by the following in lspci:

0b:00.0 SATA controller: Device 1b4b:9123 (rev 11)

Motherboard is a Quad Core Intel i7 based, ASUS P7P55D-E Deluxe LGA1156 Intel P55 DDR3 - 2133 ATX.

First test was with a single 3TB Hitachi 5400-rpm disk loaded into position one.  I transferred, without interruption, an entire 3TB volume to it.

Second test, dd if=/dev/zero of=/dev/sdi bs=1M (where sdi is the device identifier for a second 3TB Hitachi 5400 RPM drive I have inserted into position 2 in the enclosure.  Currently humming along at a 71.4 MB/s rate, and we'll see what the rate looks like at the edge of the disk.

After 730GB transferred, rate is now 74.7MB/s.
After 2.5TB, rate is 88.8MB/s.
And finally, 3000592982016 bytes (3.0 TB) copied, 35449.4 s, 84.6 MB/s

Next, I formatted drive2 with ext4 and transferred 500+ GB.  Average at 500GB mark was 65.25MB/s.  For the next 100GB or so store rate was 65.8 MB/s averaged.

Using "rsync -av --delete-excluded " the results:
   sent 783607353796 bytes  received 2161398 bytes  63,029,118.46 bytes/sec

Ran another 260GB test copy:
   sent 264064015788 bytes  received 14133 bytes  69,591,258.38 bytes/sec

Looking solid.  Before I could do my next tests, I found drive slot 3 failed to work.  Since I picked this enclosure up at Frys and it was a markdown due to a return, I now figured out why it was returned.  Took it back and replaced with a new unit.  Also, I ordered some Syba cables (SY-CAB-ESA e-SATA to e-SATA Cable, 21-Inch) since some people have reported the cables that came with the enclosure are substandard.  So far my tests haven't shown any troubles with them, but I'll give the Syba cables a run for comparison.

Next test, using the Syba cables, dd from drive 2 (Hitachi 3TB) to drive 1 (WD 3TB), and drive 4 (Seagate 1.5TB) to drive 3 (WD 3TB) simultaneously.  Here's some early results:

   3745513472 bytes (3.7 GB) copied, 214.557 s, 17.5 MB/s
   3451912192 bytes (3.5 GB) copied, 206.387 s, 16.7 MB/s

   58474889216 bytes (58 GB) copied, 3477.81 s, 16.8 MB/s
   58770587648 bytes (59 GB) copied, 3486.02 s, 16.9 MB/s

   125004939264 bytes (125 GB) copied, 7447.71 s, 16.8 MB/s
   124681977856 bytes (125 GB) copied, 7439.6 s, 16.8 MB/s

   451648946176 bytes (452 GB) copied, 27096.5 s, 16.7 MB/s
   451353247744 bytes (451 GB) copied, 27088.4 s, 16.7 MB/s

   605514891264 bytes (606 GB) copied, 36413.1 s, 16.6 MB/s
   605839949824 bytes (606 GB) copied, 36421.3 s, 16.6 MB/s

Interesting my speeds are down compared to the early single drive test, but this isn't like for like yet.  Will have to retest with A) a single WD drive and B) a single Hitachi drive.

Let's try single drive reads.  Read from WD 3TB to /dev/null:

    29657923584 bytes (30 GB) copied, 242.405 s, 122 MB/s

This seems to continue around the same pace.  But when I start a read from TWO drives at once, speeds nearly cut in half combined: 

    drive1: 3714056192 bytes (3.7 GB) copied, 112.843 s, 32.9 MB/s
    drive2: 3716153344 bytes (3.7 GB) copied, 112.893 s, 32.9 MB/s

Same results using the PEXESAT32 controller as JBOD:

  Single drive only: 5524946944 bytes (5.5 GB) copied, 42.835 s, 129 MB/s
  Dual drive read:
     drive1: 3511681024 bytes (3.5 GB) copied, 55.2967 s, 63.5 MB/s
     drive2: 3511681024 bytes (3.5 GB) copied, 55.2987 s, 63.5 MB/s

That's a big improvement for the PEXESAT32 (Marvel 9128 based) versus the Onboard Marvel.

How about 3 drives on the expansion card?

   3764387840 bytes (3.8 GB) copied, 88.9512 s, 42.3 MB/s
   3764387840 bytes (3.8 GB) copied, 88.9532 s, 42.3 MB/s
   3764387840 bytes (3.8 GB) copied, 88.9552 s, 42.3 MB/s

Again, overall the eSATA port is managing to pull about 127-129 MB/s on the Expansion card no matter how many drives I spread it across.

What's 4 drives on the onboard Marvell 9123 reading simultaneously look like you ask?  Yep, more speed degradation:

   1883242496 bytes (1.9 GB) copied, 135.96 s, 13.9 MB/s
   1952448512 bytes (2.0 GB) copied, 136.024 s, 14.4 MB/s
   1929379840 bytes (1.9 GB) copied, 136.047 s, 14.2 MB/s
   1933574144 bytes (1.9 GB) copied, 136.051 s, 14.2 MB/s

That's a combined speed of 56.7 MB/s, a drop-off of ~53.5% from single drive reads!  NOTE TO SELF: Test my onboard mirrored drives within my server for speed comparison.

Kernel is 2.6.34-gentoo-r12, Quad-core system.  Not sure what sort of tweaking might need to be done to improve things.

Now, i needed to revisit the single drive write, this time with the WD 3TB drive for comparison to the Hitachi test earlier.
  # dd if=/dev/zero of=/dev/sdg bs=1M
  16183070720 bytes (16 GB) copied, 137.352 s, 118 MB/s
  2980546994176 bytes (3.0 TB) copied, 30124.9 s, 98.9 MB/s
  3000592982016 bytes (3.0 TB) copied, 30456.3 s, 98.5 MB/s

Quite a bit faster than the Hitachi, but let's keep in mind I'm now in a new enclosure (same brand and model though) and using Syba cables versus the no-name in that comes with the enclosure.  I will retest the Hitachi a final time to compare results, after letting this WD run through it's paces.

Next tests will be with the Western Digital 3TB Caviar Green drives, in position 5 and 6, where I will be using a PEXESAT32 PCIe dual-port eSATA controller.

I will continue to update this post as more data is collected.

No comments: