Testing out this enclosure in an attempt to move to a ZFS 6-disk dual parity cluster.
WARNING: THIS WILL BE A MESS OF A POST while I'm working through the various tests. I will cleanup and organize my thoughts after I'm happy with the correct set of tests and order for this.
eSATA supports up to 5 devices via port multiplication, and the TR8M has 8 disk slots, split 4 disk slots per eSATA port (2), with a rather cheap insertion mechanism (but it seems to be working fine so far in my tests). I have it temporarily connect directly to the sole esata port my motherboard offers
I also have acquired a StarTech PEXESAT32 card, based on a Marvel 9128 chip, for dual eSATA ports. When booting the system, the post message says to hit Ctrl+M to enter setup but I just can't get this to work ever. Identified by the following in lspci:
0b:00.0 SATA controller: Device 1b4b:9123 (rev 11)
Motherboard is a Quad Core Intel i7 based, ASUS P7P55D-E Deluxe LGA1156 Intel P55 DDR3 - 2133 ATX.
First test was with a single 3TB Hitachi 5400-rpm disk loaded into position one. I transferred, without interruption, an entire 3TB volume to it.
Second test, dd if=/dev/zero of=/dev/sdi bs=1M (where sdi is the device identifier for a second 3TB Hitachi 5400 RPM drive I have inserted into position 2 in the enclosure. Currently humming along at a 71.4 MB/s rate, and we'll see what the rate looks like at the edge of the disk.
After 730GB transferred, rate is now 74.7MB/s.
After 2.5TB, rate is 88.8MB/s.
And finally, 3000592982016 bytes (3.0 TB) copied, 35449.4 s, 84.6 MB/s
Next, I formatted drive2 with ext4 and transferred 500+ GB. Average at 500GB mark was 65.25MB/s. For the next 100GB or so store rate was 65.8 MB/s averaged.
Using "rsync -av --delete-excluded " the results:
sent 783607353796 bytes received 2161398 bytes 63,029,118.46 bytes/sec
Ran another 260GB test copy:
sent 264064015788 bytes received 14133 bytes 69,591,258.38 bytes/sec
Looking solid. Before I could do my next tests, I found drive slot 3 failed to work. Since I picked this enclosure up at Frys and it was a markdown due to a return, I now figured out why it was returned. Took it back and replaced with a new unit. Also, I ordered some Syba cables (SY-CAB-ESA e-SATA to e-SATA Cable, 21-Inch) since some people have reported the cables that came with the enclosure are substandard. So far my tests haven't shown any troubles with them, but I'll give the Syba cables a run for comparison.
Next test, using the Syba cables, dd from drive 2 (Hitachi 3TB) to drive 1 (WD 3TB), and drive 4 (Seagate 1.5TB) to drive 3 (WD 3TB) simultaneously. Here's some early results:
3745513472 bytes (3.7 GB) copied, 214.557 s, 17.5 MB/s
3451912192 bytes (3.5 GB) copied, 206.387 s, 16.7 MB/s
58474889216 bytes (58 GB) copied, 3477.81 s, 16.8 MB/s
58770587648 bytes (59 GB) copied, 3486.02 s, 16.9 MB/s
125004939264 bytes (125 GB) copied, 7447.71 s, 16.8 MB/s
124681977856 bytes (125 GB) copied, 7439.6 s, 16.8 MB/s
451648946176 bytes (452 GB) copied, 27096.5 s, 16.7 MB/s
451353247744 bytes (451 GB) copied, 27088.4 s, 16.7 MB/s
605514891264 bytes (606 GB) copied, 36413.1 s, 16.6 MB/s
605839949824 bytes (606 GB) copied, 36421.3 s, 16.6 MB/s
Interesting my speeds are down compared to the early single drive test, but this isn't like for like yet. Will have to retest with A) a single WD drive and B) a single Hitachi drive.
Let's try single drive reads. Read from WD 3TB to /dev/null:
29657923584 bytes (30 GB) copied, 242.405 s, 122 MB/s
This seems to continue around the same pace. But when I start a read from TWO drives at once, speeds nearly cut in half combined:
drive1: 3714056192 bytes (3.7 GB) copied, 112.843 s, 32.9 MB/s
drive2: 3716153344 bytes (3.7 GB) copied, 112.893 s, 32.9 MB/s
Same results using the PEXESAT32 controller as JBOD:
Single drive only: 5524946944 bytes (5.5 GB) copied, 42.835 s, 129 MB/s
Dual drive read:
drive1: 3511681024 bytes (3.5 GB) copied, 55.2967 s, 63.5 MB/s
drive2: 3511681024 bytes (3.5 GB) copied, 55.2987 s, 63.5 MB/s
That's a big improvement for the PEXESAT32 (Marvel 9128 based) versus the Onboard Marvel.
How about 3 drives on the expansion card?
3764387840 bytes (3.8 GB) copied, 88.9512 s, 42.3 MB/s
3764387840 bytes (3.8 GB) copied, 88.9532 s, 42.3 MB/s
3764387840 bytes (3.8 GB) copied, 88.9552 s, 42.3 MB/s
Again, overall the eSATA port is managing to pull about 127-129 MB/s on the Expansion card no matter how many drives I spread it across.
What's 4 drives on the onboard Marvell 9123 reading simultaneously look like you ask? Yep, more speed degradation:
1883242496 bytes (1.9 GB) copied, 135.96 s, 13.9 MB/s
1952448512 bytes (2.0 GB) copied, 136.024 s, 14.4 MB/s
1929379840 bytes (1.9 GB) copied, 136.047 s, 14.2 MB/s
1933574144 bytes (1.9 GB) copied, 136.051 s, 14.2 MB/s
That's a combined speed of 56.7 MB/s, a drop-off of ~53.5% from single drive reads! NOTE TO SELF: Test my onboard mirrored drives within my server for speed comparison.
Kernel is 2.6.34-gentoo-r12, Quad-core system. Not sure what sort of tweaking might need to be done to improve things.
Now, i needed to revisit the single drive write, this time with the WD 3TB drive for comparison to the Hitachi test earlier.
# dd if=/dev/zero of=/dev/sdg bs=1M
16183070720 bytes (16 GB) copied, 137.352 s, 118 MB/s
...
2980546994176 bytes (3.0 TB) copied, 30124.9 s, 98.9 MB/s
3000592982016 bytes (3.0 TB) copied, 30456.3 s, 98.5 MB/s
Quite a bit faster than the Hitachi, but let's keep in mind I'm now in a new enclosure (same brand and model though) and using Syba cables versus the no-name in that comes with the enclosure. I will retest the Hitachi a final time to compare results, after letting this WD run through it's paces.
Next tests will be with the Western Digital 3TB Caviar Green drives, in position 5 and 6, where I will be using a PEXESAT32 PCIe dual-port eSATA controller.
I will continue to update this post as more data is collected.
No comments:
Post a Comment