Software RAID5 with 2tb drives

Installing, Configuring, Troubleshooting server daemons such as Web and Mail
nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/26 16:11:23

I'm trying to build a new system with 3 2tb drives in raid5. I was following the centos raid howto, and it talks about using dmraid and how the boot volume must be on a raid1 array. I'd really rather not do this, since I plan to add 3 more 2tb drives later and expand the array, and it would be nice to add a hot spare and not have to worry that the 2 drives that failed are the boot drives. My second issue is that disk druid doesn't like the GPT partition table on the 2tb drives, but I thought drives of this size had to be GPT due to a 2tb limit of MBR?

1) Is there a way to do software raid5 without the raid1 boot volume?
2) Can anyone comment on the 2tb drive size and GPT partitions?

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Software RAID5 with 2tb drives

Post by pschaff » 2011/03/26 17:10:18

I think you mean [b]mdraid[/b] not [b]dmraid[/b] (used for hardware/HostRAID/FakeRAID), but you did not say which of the several [url=http://wiki.centos.org/?action=fullsearch&context=180&value=RAID&titlesearch=Titles]Wiki RAID articles[/url] you are following.

As stated in [url=http://wiki.centos.org/HowTos/SoftwareRAIDonCentOS5]How to Setup a Software RAID on CentOS 5[/url] you cannot have /boot on RAID other than RAID1. You can make a separate /boot partition and make other partitions RAID5, also as stated in the article.

IIRC the anaconda installer does not understand GPT.

I'd be prone to install the OS on a separate smaller disk (two if you want RAID1) and leave the big RAID5 array to be configured later and mounted where you need it as storage space.

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/27 17:46:03

I was following the following article, also from centos.org!
[url=http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-raid-config.html]Configuring Software RAID[/url]

The installer did not understand GPT, so I ended up booting off a Ubuntu 9.10 livecd and changed it to msdos, and that worked fine. I created a 100mb /boot on two drives and set for raid1, and then created a 16gb swap partition on all 3 drives as raid5, and then the rest of the 2tb drives became / as raid5. I'd rather not have extra drives in this, the power supply isn't that large.

Once mdadm said the array was done building, I did some speed tests. I got:
Reading was 115MB/s, totally acceptable.
Writing was 55MB/s, kinda not acceptable.

I have another box, similar specs but it had a 3ware 9650 hardware raid card in it and it is set for RAID10 with 6x 500gb drives. I ran the same tests on it and got:

Reading was 161MB/s
Writing was 107MB/s

So I was kind of hoping for 100MB/s write speeds from the new system, since the old one with the old raid card was getting those speeds. I'll post again once I do the tests on a single drive to see what the write speeds are.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Software RAID5 with 2tb drives

Post by TrevorH » 2011/03/27 17:58:29

You need to wait until it's finished building before you can speed test it. This is a 4 drive RAID 5 array:

[code]
# hdparm -tT /dev/md0

/dev/md0:
Timing cached reads: 41004 MB in 2.00 seconds = 20532.68 MB/sec
Timing buffered disk reads: 782 MB in 3.00 seconds = 260.23 MB/sec
[/code]

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/27 19:17:37

I did wait, it took 10 hours!

Ok, so my tests with a single drive are as follows:
Write speed: 71 MB/s
Read speed: 97 MB/s

So thats not great, but not bad considering these are 5900rpm drives. So why is the RAID5 taking so much overhead? Shouldn't it in theory be twice that for write speeds since it is 2 data drives?

260 MB/s would be fantastic, I only have 3 drives now though. I'm going to start the rebuild of the array to try this again.

Update: After doing some googling, the 55 MB/s write speed may in fact be all I can get from these drives.
http://www.z-a-recovery.com/art-raid.htm

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: Software RAID5 with 2tb drives

Post by pschaff » 2011/03/27 20:47:11

[quote]
nosebreaker wrote:
I was following the following article, also from centos.org!
[url=http://www.centos.org/docs/5/html/Deployment_Guide-en-US/s1-raid-config.html]Configuring Software RAID[/url][/quote]
That is indeed on centos.org, but is an old copy of upstream documentation as the CentOS copies have not been kept up to date. If you want upstream docs I'd just go to the [url=http://docs.redhat.com/docs/en-US/index.html]source[/url].

P.S. Please do not change the title in mid-thread to something nondescript. It is very confusing.

Edit: I see some fellow moderator has anonymously acted on that issue - never mind.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Software RAID5 with 2tb drives

Post by TrevorH » 2011/03/28 01:17:41

What sort of drives are you using? My array here is 4 x 1.5TB Western Digital WD15EADS "Green" 5400 rpm drives and tested individually with hdparm will pull ~95MB/s each. I don't think these are particularly new or fast - they're probably going on 18 months old already. There's nowhere in this thread where you've posted the information about your configuration - which physical disks? what SATA ports are they attached to? Are they seen as /dev/sd* or /dev/hd*? What are you using to measure read/write speeds? What speeds do you get if you test each disk individually - e.g. if the array is /dev/md0 and consists of /dev/sd[bcd] then what speed do you get from /dev/sdb?

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/28 02:19:11

I am using (Seagate Barracuda LP ST32000542AS 2TB 5900 RPM 32MB Cache SATA 3.0Gb/s 3.5" Hard Drive -Bare Drive):
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148413

The motherboard (BIOSTAR A770E3 AM3 AMD 770 ATX AMD)
http://www.newegg.com/Product/Product.aspx?Item=N82E16813138179

Write test (16gb file):
dd if=/dev/zero of=testfile.out bs=1M count=16384
Read test (that same 16gb file you just made):
dd if=testfile.out of=/dev/null bs=1M

When I tested read speeds individually they were 97 MB/s as I would expect, but read speed was only 71 MB/s. I did a basic install of CentOS onto the system to pull these numbers.

They are attached to ports 0 through 2 on the motherboard, they show up as /dev/sda /dev/sdb and /dev/sdc

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Software RAID5 with 2tb drives

Post by TrevorH » 2011/03/28 04:10:05

Ok, so I repeated (more or less) your write test here

[code]
# time { sync; dd if=/dev/zero of=/mnt/r3tb/test1.file bs=1M count=16384; sync; }
16384+0 records in
16384+0 records out
17179869184 bytes (17 GB) copied, 95.8845 seconds, 179 MB/s

real 1m47.669s
user 0m0.017s
sys 0m14.262s
[/code]

As you can see, dd's timing is a little faster than it should be because not all data had hit the disk platters (96 secs vs the actual elapsed 107 so more like 152MB/s). Don't have to bother with that on reading

[code]
# dd if=/mnt/r3tb/test1.file of=/dev/null bs=1M
16384+0 records in
16384+0 records out
17179869184 bytes (17 GB) copied, 97.0281 seconds, 177 MB/s
[/code]

Even accounting for your 3 spindles vs my 4 I think your rates are low.

nosebreaker
Posts: 72
Joined: 2010/08/09 16:10:26

Re: Software RAID5 with 2tb drives

Post by nosebreaker » 2011/03/28 13:43:20

I agree that I would think it would be faster, but if the max write speed is 71 MB/s, then it is possible the 55 MB/s write speed is the max. As I understand it, RAID5 has to do a read on the other drives before it can write the parity bit, even though it should know that anyway. But I am hoping I am wrong!

Do you have any ideas on what I can try to make this work? I suppose I could take some older drives and see if they are any faster.

Post Reply