Software RAID5 with 2tb drives

Installing, Configuring, Troubleshooting server daemons such as Web and Mail
nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/28 21:02:55

I took 3 different drives (80gb, 1tb, 1.5tb - but only 20gb / data size), and ran the same tests. I got:
Read speed: 67 MB/s
Write speed: 43 MB/s

I also ran hdparm:
[code]
# /sbin/hdparm -tT /dev/md2

/dev/md2:
Timing cached reads: 6560 MB in 2.00 seconds = 3282.08 MB/sec
Timing buffered disk reads: 200 MB in 3.03 seconds = 65.91 MB/sec
[/code]

So this makes me think it might be the motherboard or something else. Any ideas to look for?

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: Software RAID5 with 2tb drives

Post by pschaff » 2011/03/28 21:22:46

Are these perchance AFD (Advanced Format Disk) drives? If so, the partition alignment is a critical factor for performance. Also check that you have optimal BIOS settings - i.e. AHCI and not legacy IDE mode.

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/28 21:57:08

I have no idea if they are AFD, how would I align the partitions?

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: Software RAID5 with 2tb drives

Post by pschaff » 2011/03/28 22:07:15

This [url=https://www.centos.org/modules/newbb/viewtopic.php?topic_id=25914&forum=37&post_id=105418#forumpost105418]post[/url] addresses the issue as well as I have seen. Perhaps an easier way would be to do any partitioning with a later distro (such as recent Fedora) that has partition tools that understand 4k sectors. If I understand correctly it is only an issue if your are not using the whole disk for a single partition, as you may or may not be doing with RAID.

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/29 03:52:10

I may have fixed it!

I used fdisk to change the heads/sectors from 256/63 (old CHS/LBA settings) to 224/56 (4k setting).

In fdisk on each disk:
x for expert mode
h for heads, change to 224
s for sectors, change to 56
w to write and exit

Now speeds on a single drive were 108 MB/s for reading, and 118 MB/s for writing (old was 97/71 MB/s). I am rebuilding the system now, will update tomorrow when its done and I can test.

This post was helpful:
http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/index.html?ca=dgr-lnxw074KB-Disksdth-LX

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/29 04:41:36

Right now the recovery on the fresh install is going at 62-65 MB/s, better than before but not the big increase I was hoping for. I'm not sure if this fixed it.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Software RAID5 with 2tb drives

Post by TrevorH » 2011/03/29 13:02:13

The recovery is, by default, throttled unless you have changed it.

[code]
# grep -i raid /etc/sysctl.conf
# RAID rebuild min speed
dev.raid.speed_limit_min = 400000
dev.raid.speed_limit_max = 400000

# grep -i raid /etc/rc.local
echo 8192 > /sys/block/md0/md/stripe_cache_size
[/code]

Run sysctl -p to implement any changes you make to /etc/sysctl.conf immediately.

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/30 23:42:41

I found a way to fix it, basically I found a script that calculates the best settings and modifies it for you. I had to modify the script to use /sbin/blockdev and not just blockdev, and my array was /dev/(sda3,sdb3,sdc3) and not sda1/sdb1/sdc1. Once I modified those lines and changed it, it worked!

Before:
Write speed 55 MB/s
Read speed 115 MB/s

After
Write speed is 107 MB/s
Read speed is 164 MB/s

A significant improvement!

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/30 23:45:19

Oh, the only thing I've noticed is, now the drive heads click about every 10 seconds. I found an old post about Ubuntu that does this, but I wanted to ask if anyone here knew of a fix.
http://www.linuxquestions.org/questions/ubuntu-63/hdparm-apm-hard-drive-clicking-sound-588770/

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/31 01:24:16

I added the hdparm -B 255 /dev/sda command to my startup, along with that script to tune the HDD's. Seems to work for now, but I was hoping for something better. That is apparently new firmware for the drive, I'll try that as well.

Post Reply