Software RAID5 with 2tb drives
-
- Posts: 72
- Joined: 2010/08/09 16:10:26
Re: Software RAID5 with 2tb drives
I took 3 different drives (80gb, 1tb, 1.5tb - but only 20gb / data size), and ran the same tests. I got:
Read speed: 67 MB/s
Write speed: 43 MB/s
I also ran hdparm:
[code]
# /sbin/hdparm -tT /dev/md2
/dev/md2:
Timing cached reads: 6560 MB in 2.00 seconds = 3282.08 MB/sec
Timing buffered disk reads: 200 MB in 3.03 seconds = 65.91 MB/sec
[/code]
So this makes me think it might be the motherboard or something else. Any ideas to look for?
Read speed: 67 MB/s
Write speed: 43 MB/s
I also ran hdparm:
[code]
# /sbin/hdparm -tT /dev/md2
/dev/md2:
Timing cached reads: 6560 MB in 2.00 seconds = 3282.08 MB/sec
Timing buffered disk reads: 200 MB in 3.03 seconds = 65.91 MB/sec
[/code]
So this makes me think it might be the motherboard or something else. Any ideas to look for?
-
- Retired Moderator
- Posts: 18276
- Joined: 2006/12/13 20:15:34
- Location: Tidewater, Virginia, North America
- Contact:
Re: Software RAID5 with 2tb drives
Are these perchance AFD (Advanced Format Disk) drives? If so, the partition alignment is a critical factor for performance. Also check that you have optimal BIOS settings - i.e. AHCI and not legacy IDE mode.
-
- Posts: 72
- Joined: 2010/08/09 16:10:26
Re: Software RAID5 with 2tb drives
I have no idea if they are AFD, how would I align the partitions?
-
- Retired Moderator
- Posts: 18276
- Joined: 2006/12/13 20:15:34
- Location: Tidewater, Virginia, North America
- Contact:
Re: Software RAID5 with 2tb drives
This [url=https://www.centos.org/modules/newbb/viewtopic.php?topic_id=25914&forum=37&post_id=105418#forumpost105418]post[/url] addresses the issue as well as I have seen. Perhaps an easier way would be to do any partitioning with a later distro (such as recent Fedora) that has partition tools that understand 4k sectors. If I understand correctly it is only an issue if your are not using the whole disk for a single partition, as you may or may not be doing with RAID.
-
- Posts: 72
- Joined: 2010/08/09 16:10:26
Re: Software RAID5 with 2tb drives
I may have fixed it!
I used fdisk to change the heads/sectors from 256/63 (old CHS/LBA settings) to 224/56 (4k setting).
In fdisk on each disk:
x for expert mode
h for heads, change to 224
s for sectors, change to 56
w to write and exit
Now speeds on a single drive were 108 MB/s for reading, and 118 MB/s for writing (old was 97/71 MB/s). I am rebuilding the system now, will update tomorrow when its done and I can test.
This post was helpful:
http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/index.html?ca=dgr-lnxw074KB-Disksdth-LX
I used fdisk to change the heads/sectors from 256/63 (old CHS/LBA settings) to 224/56 (4k setting).
In fdisk on each disk:
x for expert mode
h for heads, change to 224
s for sectors, change to 56
w to write and exit
Now speeds on a single drive were 108 MB/s for reading, and 118 MB/s for writing (old was 97/71 MB/s). I am rebuilding the system now, will update tomorrow when its done and I can test.
This post was helpful:
http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/index.html?ca=dgr-lnxw074KB-Disksdth-LX
-
- Posts: 72
- Joined: 2010/08/09 16:10:26
Re: Software RAID5 with 2tb drives
Right now the recovery on the fresh install is going at 62-65 MB/s, better than before but not the big increase I was hoping for. I'm not sure if this fixed it.
Re: Software RAID5 with 2tb drives
The recovery is, by default, throttled unless you have changed it.
[code]
# grep -i raid /etc/sysctl.conf
# RAID rebuild min speed
dev.raid.speed_limit_min = 400000
dev.raid.speed_limit_max = 400000
# grep -i raid /etc/rc.local
echo 8192 > /sys/block/md0/md/stripe_cache_size
[/code]
Run sysctl -p to implement any changes you make to /etc/sysctl.conf immediately.
[code]
# grep -i raid /etc/sysctl.conf
# RAID rebuild min speed
dev.raid.speed_limit_min = 400000
dev.raid.speed_limit_max = 400000
# grep -i raid /etc/rc.local
echo 8192 > /sys/block/md0/md/stripe_cache_size
[/code]
Run sysctl -p to implement any changes you make to /etc/sysctl.conf immediately.
-
- Posts: 72
- Joined: 2010/08/09 16:10:26
Re: Software RAID5 with 2tb drives
I found a way to fix it, basically I found a script that calculates the best settings and modifies it for you. I had to modify the script to use /sbin/blockdev and not just blockdev, and my array was /dev/(sda3,sdb3,sdc3) and not sda1/sdb1/sdc1. Once I modified those lines and changed it, it worked!
Before:
Write speed 55 MB/s
Read speed 115 MB/s
After
Write speed is 107 MB/s
Read speed is 164 MB/s
A significant improvement!
Before:
Write speed 55 MB/s
Read speed 115 MB/s
After
Write speed is 107 MB/s
Read speed is 164 MB/s
A significant improvement!
-
- Posts: 72
- Joined: 2010/08/09 16:10:26
Re: Software RAID5 with 2tb drives
Oh, the only thing I've noticed is, now the drive heads click about every 10 seconds. I found an old post about Ubuntu that does this, but I wanted to ask if anyone here knew of a fix.
http://www.linuxquestions.org/questions/ubuntu-63/hdparm-apm-hard-drive-clicking-sound-588770/
http://www.linuxquestions.org/questions/ubuntu-63/hdparm-apm-hard-drive-clicking-sound-588770/
-
- Posts: 72
- Joined: 2010/08/09 16:10:26
Re: Software RAID5 with 2tb drives
I added the hdparm -B 255 /dev/sda command to my startup, along with that script to tune the HDD's. Seems to work for now, but I was hoping for something better. That is apparently new firmware for the drive, I'll try that as well.