Ok now this is weird. I tried building with with LVM and got good speeds: 1.8GBps read and write.
Code: Select all
# pvcreate /dev/sd[b-e]
# vgcreate vg01 /dev/sd[b-e]
# lvcreate -i 4 -L 1.86T vg01
Then I wiped the drives and rebuilt it again with mdadm and now the speed is still good. I've tried several variations, but I can't figure out why it was slow before.
Other observations: XFS is marginally faster than Ext4.
With mdadm, the default chunk size of 512K works best. I tried 64K, 32K, and 4096K.
noatime does not make any speed difference. noadirtime gives an error, I think that option doesn't exist.
I was using dd with conv=fdatasync for the write test, but now I realize that was limiting the speed. If I instead use oflag=direct then I see the true write speed. Not sure why, maybe this method is using the drive's write cache. But the speed is sustained for many GB so I think it's true.