Software RAID0 slow

General support questions
hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Software RAID0 slow

Post by hunter86_bg » 2018/10/17 19:05:15

What does iostat show?
You can use

Code: Select all

iostat  -dxN
Also, did you stop the drives cache?
Create an LV with stripe of 4 (-i flag) , create XFS ontop, mount it and run 'dd' with 'oflag=direct' and small(64KB) and large(1-4MB) 'bs' value against a file on this LV. If the dd is creating it faster, then it should be an alignment problem.

Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Re: Software RAID0 slow

Post by Elliott » 2018/10/17 20:34:03

Here's a sample of iostat output. I'm not sure what you mean about creating an LV. Do you mean to use lvcreate instead of mdadm? I'm using ext4 for now and I don't have a strong preference either way, but do you expect XFS to be faster?

Code: Select all

Linux 3.10.0-862.el7.x86_64 (alt-pix-la) 	10/17/2018 	_x86_64_	(48 CPU)
Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    9.00     0.00     0.02     4.44     0.02    2.33    0.00    2.33   2.33   2.10
sdb               0.00     0.00  411.00    0.00   205.42     0.00  1023.61     4.64   11.18   11.18    0.00   1.02  42.10
sdc               0.00     0.00  418.00    0.00   203.48     0.00   996.98     4.59   10.84   10.84    0.00   1.00  41.90
sdd               0.00     0.00  423.00    0.00   200.99     0.00   973.11     4.58   10.68   10.68    0.00   0.98  41.60
sde               0.00     0.00  403.00    0.00   199.55     0.00  1014.09     4.31   10.59   10.59    0.00   1.01  40.70
centos-root       0.00     0.00    0.00    9.00     0.00     0.02     4.44     0.02    2.33    0.00    2.33   2.33   2.10
centos-swap       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sdf               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
centos-home       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
md0               0.00     0.00 1667.00    0.00   814.47     0.00  1000.62     0.00    0.00    0.00    0.00   0.00   0.00

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Software RAID0 slow

Post by hunter86_bg » 2018/10/18 03:56:00

In your case test both ways:
A) mdadm -> raid0 -> you need to decide the stripe width depending on workload (read/write)-> 4 disks -> XFS
B) no MD (wipe the MD metadata) -> no partitions -> pvcreate on all SSDs -> vgcreate VG with all PVs -> create lv with stripe of 4 ( I think the switch is '-i 4') and pick the stripe width based on workload (read/ write).
What type of load (or what command) was used when iostat output was generated? This is very important, as based on the application IO requests you should change the stripe width.
If you need high read -> use stripe width larger than 1MB, for faster writes, usually you need smaller stripe size , which reduces read speed.
What mount options are you using ? You can consider using noatime & noadirtime .
XFS will definitely be faster than ext-based systems.

Edit: Test the performance with larger tests ( 10GB or more) as filesystem cache will initially show unreal performance.In your case flags like 'direct' bypass it.
Also , there are synthetic benchmarks like bonnie++.
Keep in mind that best tests are done with the application that will use your storage.

Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Re: Software RAID0 slow

Post by Elliott » 2018/10/18 06:54:15

Ok now this is weird. I tried building with with LVM and got good speeds: 1.8GBps read and write.

Code: Select all

# pvcreate /dev/sd[b-e]
# vgcreate vg01 /dev/sd[b-e]
# lvcreate -i 4 -L 1.86T vg01
Then I wiped the drives and rebuilt it again with mdadm and now the speed is still good. I've tried several variations, but I can't figure out why it was slow before.

Other observations: XFS is marginally faster than Ext4.
With mdadm, the default chunk size of 512K works best. I tried 64K, 32K, and 4096K.
noatime does not make any speed difference. noadirtime gives an error, I think that option doesn't exist.
I was using dd with conv=fdatasync for the write test, but now I realize that was limiting the speed. If I instead use oflag=direct then I see the true write speed. Not sure why, maybe this method is using the drive's write cache. But the speed is sustained for many GB so I think it's true.

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Software RAID0 slow

Post by hunter86_bg » 2018/10/18 15:09:11

Another optimization for ssds that came to my mind : disk scheduler.
Try using 'noop' as disk scheduler , as the default settings of the deadline focus on the reads and provides sustainable read/write latencies but is not optimal for writes without tweaking.
About noadirtime, check the man page as it might have been misspelled by me or even deprecated.

As you don't have battery backed write cache - keep the write barriers (which is the default now).
I prefer the new LVM approach as it is a more flexible approach and you can migrate your data live, or even reshape the raid - if such need occurs in the future.

Edit: your test is not 100% accurate as dd is testing sequential reads & writes but not random ones. Try bonnie++ for random read/write performance. As you didn't specify the application that will use it, I'm recommending to test anything that comes to your mind.

Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Re: Software RAID0 slow

Post by Elliott » 2020/01/23 01:27:47

I've been away on other projects but I'm coming back to this now. This time I put 7 disks in the same system, but the limit is still around 1100MBps.

How can I set noop scheduler? I tried echo 'noop' > /sys/block/md0/queue/scheduler (running as root of course)
But cat /sys/block/md0/queue/scheduler still shows 'none'.

Post Reply