Software RAID0 slow

General support questions
Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Software RAID0 slow

Post by Elliott » 2018/10/16 06:05:22

Hi all. I'm building a server and getting poor performance from the software RAID. Any tips to improve it? This is my first time learning mdadm; I'm more familiar with storcli and hardware RAIDs.

I have four SSDs connected to the motherboard. SATA ports are set to RAID mode using Intel RST, although I'm not sure if that even matters. The system is a Dell 7920. I build the virtual disk like this:

Code: Select all

# mdadm --create /dev/md0 --level 0 -n 4 /dev/sd[b-e]
# cat /proc/mdstat
Personalities : [raid0]
md0 : active raid0 sdb[0] sde[3] sdd[2] sdc[1]
      1999900672 blocks super 1.2 512k chunks
# parted -s /dev/md0 gpt
# parted -s /dev/md0 mklabel gpt
# parted -s -a optimal /dev/md0p1 mkpart primary 0% 100%
# mkfs -t ext4 /dev/md0
# mount /dev/md0p1 /mnt/RAID
      
I expect to see near 2GBps, but I get half that.

Code: Select all

# dd if=/dev/zero of=/mnt/RAID/testfile bs=1M count=2000 conv=fdatasync
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 2.20422 s, 951 MB/s
# dd if=/mnt/RAID/testfile of=/dev/null bs=1M iflag=direct
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 2.24319 s, 935 MB/s

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Software RAID0 slow

Post by TrevorH » 2018/10/16 06:14:05

You definitely don't want the BIOS set to RAID mode. You may need to run `dmraid -r -R` against each member disk after turning it off to remove the FakeRAID metadata from them. Though having said that I would have expected to see some indication in /proc/mdstat like "imsm" in the text somewhere if it was using FakeRAID.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Re: Software RAID0 slow

Post by Elliott » 2018/10/16 06:19:17

Interesting — I thought the "hybrid" RST might help if anything. Care to elaborate on why it's bad? I did try with AHCI mode too, and didn't notice any speed difference.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Software RAID0 slow

Post by TrevorH » 2018/10/16 06:58:39

If it's set up using RST then mdadm detects that and the RAID operations then become a cooperative effort between it and the BIOS code. It's effectively using the FakeRAID code path and not the mdadm one.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Re: Software RAID0 slow

Post by Elliott » 2018/10/16 07:02:54

Ok thanks. I'm happy to switch back to AHCI, but I did try that already and got the same performance. I need to look around for more clues. These are brand new SSDs, latest firmware.

Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Re: Software RAID0 slow

Post by Elliott » 2018/10/17 08:12:43

AHCI mode allows me to enable write cache, and I can see that the cache is enabled using hdparm.
smartctl shows they are all connected at SATA 3.1 (6Gbps) speed.

I tried building the array with mdadm using imsm metadata as suggested by Intel. No difference.

If I benchmark each disk individually I get 420MBps, which is a little slow because these drives do 490MBps on another Windows system.
Two drives in RAID0 get 630MBps.
Four drives in RAID0 get 960MBps. So the performance benefit per drive is decreasing. To check controller bandwidth, I tried running four dd instances to test all four individual drives at once, and they stay at 420MBps. So this tells me the controller can handle at least 1680MBps.

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Software RAID0 slow

Post by hunter86_bg » 2018/10/17 10:17:57

Are these NVME ?
You might have shared PCI and M.2 slot.

Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Re: Software RAID0 slow

Post by Elliott » 2018/10/17 14:59:32

No, they are SATA drives. Samsung 850 Pro. I don't think PCI bandwidth is the issue because I get much faster performance when I bench them all at once individually, without RAID.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Software RAID0 slow

Post by TrevorH » 2018/10/17 15:11:01

One thing could be alignment problems. Your mdadm is using metadata version 1.2 which is stored at 4K from the front of each device. That's almost certainly going to push things out of alignment. I notice that you're not partitioning these devices so you may want to do that and then adjust the starting position of the partition to cater for the metadata on the front and align everything to (usually 1MB) a recommended offset. I believe most SSDs use a 64KB block internally so you probably want to try to get the data portion of each device aligned to a 64KB boundary. Maybe...

mdadm -E $device will show a data offset and you can use a --data-offset as part of the create command.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Elliott
Posts: 44
Joined: 2018/09/14 01:30:14

Re: Software RAID0 slow

Post by Elliott » 2018/10/17 17:16:10

Thanks for the suggestion. mdadm -E showed the offset was 264192 sectors, which is 129MB. It's odd that this shows in sectors, while the command-line argument --data-offset uses KB.
I tried again with --data-offset 64K and I get the same performance. mdadm -E now shows the data offset is 128 sectors.

Post Reply