network transfer slow

Issues related to configuring your network
paabloop
Posts: 8
Joined: 2011/08/29 21:04:34

network transfer slow

Post by paabloop » 2011/08/29 21:32:59

Hi i have a troubles about network transfers, i have two servers with the same hardware.

i have the eth0 connected to gigalan switch with the net 192.168.1.13 named st1 and 192.168.1.14 named st2 if i download with wget:

[code][root@st2 shm]# wget http://192.168.1.13/test.iso
--2011-08-29 18:12:18-- http://192.168.1.13/test.iso
Connecting to 192.168.1.13:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 285069312 (272M) [application/octet-stream]
Saving to: `test.iso.3'

100%[========================================================>] 285,069,312 113M/s in 2.4s

2011-08-29 18:12:21 (113 MB/s) - `test.iso.3' saved [285069312/285069312]

[root@st2 shm]#[/code]

and i have a infiniband cards with the interface ib0 and download a file with wget:
[code][root@st2 shm]# wget http://10.10.0.13/test.iso
--2011-08-29 18:17:08-- http://10.10.0.13/test.iso
Connecting to 10.10.0.13:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 285069312 (272M) [application/octet-stream]
Saving to: `test.iso.4'

100%[========================================================>] 285,069,312 278M/s in 1.0s

2011-08-29 18:17:09 (278 MB/s) - `test.iso.4' saved [285069312/285069312]

[root@st2 shm][/code]

this are fine, but if i copy the same files through scp or with any other protocol (included drbd, i need for drdb) the transfer through eth0 is:

[code][root@st2 shm]# scp test.iso root@192.168.1.13:/dev/shm/
test.iso 100% 272MB 38.8MB/s 00:07
[root@st2 shm]# [/code]


and the same do with the infiniband ib0 interface.

[code][root@st2 shm]# scp test.iso root@10.10.0.13:/dev/shm/
test.iso 100% 272MB 38.8MB/s 00:07
[root@st2 shm]#[/code]


The saddest thing is that with drbd is:

[code]version: 8.4.0 (api:1/proto:86-100)
GIT-hash: 28753f559ab51b549d16bcf487fe625d5919c49c build by root@st2.caltrad.cl, 2011-08-24 14:44:40
0: cs:SyncTarget ro:Secondary/Secondary ds:Inconsistent/UpToDate C r-----
ns:0 nr:447232 dw:447232 dr:0 al:0 bm:27 lo:0 pe:0 ua:0 ap:0 ep:1 wo:d oos:2992340500
[>....................] sync'ed: 0.1% (2922204/2922644)M
finish: 27:51:19 speed: 29,812 (29,812) want: 34,000 K/sec[/code]

the drbd config (/etc/drbd.d/global_common.conf):

[code][root@st1 shm]# cat /etc/drbd.d/global_common.conf
global {
usage-count no;
# minor-count dialog-refresh disable-ip-verification
}

common {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
# fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
split-brain "/usr/lib/drbd/notify-split-brain.sh root";
out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
# before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
# after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
}

startup {
wfc-timeout 600;
degr-wfc-timeout 600;
#outdated-wfc-timeout wait-after-sb
}

options {
cpu-mask f;
#on-no-data-accessible
}

disk {
on-io-error detach;
disk-barrier no;
disk-flushes no;
# size max-bio-bvecs on-io-error fencing disk-barrier disk-flushes
# disk-drain md-flushes resync-rate resync-after al-extents
# c-plan-ahead c-delay-target c-fill-target c-max-rate
# c-min-rate disk-timeout
}

net {
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 0;
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
unplug-watermark 16;
# protocol timeout max-epoch-size max-buffers unplug-watermark
# connect-int ping-int sndbuf-size rcvbuf-size ko-count
# allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
# after-sb-1pri after-sb-2pri always-asbp rr-conflict
# ping-timeout data-integrity-alg tcp-cork on-congestion
# congestion-fill congestion-extents csums-alg verify-alg
# use-rle
}
syncer {
rate 900M;
al-extents 3389;
verify-alg md5;
}
}[/code]



Very bad i need more speed.

the get info is here: http://pastebin.centos.org/37804

i test with centos 5.3, 5.4, 5.5, 5,6 and 6.0 and i have the same results. What could it be?

Regards

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: network transfer slow

Post by TrevorH » 2011/08/29 22:22:51

[quote]
test.iso 100% 272MB 38.8MB/s 00:07
[/quote]

Most probably you are hitting the limit on the speed of your cpu here as scp has to encrypt/decrypt the data.

[quote]
syncer {
rate 900M;
[/quote]

Unless you have 10Ge ethernet cards there, that is [b]way[/b] too high a value. DRBD uses mega[u]bytes[/u] in its syncer numbers not megabits. From your posting of /proc/drbd where it says "Want: 34,000KB/s" you must have that overridden somewhere - can you post the resource definition too.

paabloop
Posts: 8
Joined: 2011/08/29 21:04:34

Re: network transfer slow

Post by paabloop » 2011/08/30 02:01:48

[quote]
TrevorH wrote:
[quote]
test.iso 100% 272MB 38.8MB/s 00:07
[/quote]

Most probably you are hitting the limit on the speed of your cpu here as scp has to encrypt/decrypt the data.

[quote]
syncer {
rate 900M;
[/quote]

Unless you have 10Ge ethernet cards there, that is [b]way[/b] too high a value. DRBD uses mega[u]bytes[/u] in its syncer numbers not megabits. From your posting of /proc/drbd where it says "Want: 34,000KB/s" you must have that overridden somewhere - can you post the resource definition too.[/quote]

i have infiniband card with 2 ports of 10Gb (i'm use one, but i will do a bonding), if you see trough wget download files to 276M/s, centos have a limitation about some type of transfers. but i don't know where :S

Thanks

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: network transfer slow

Post by TrevorH » 2011/08/30 07:50:32

Post the rest of your DRBD config files please.

paabloop
Posts: 8
Joined: 2011/08/29 21:04:34

Re: network transfer slow

Post by paabloop » 2011/08/30 14:26:52

[quote]
TrevorH wrote:
Post the rest of your DRBD config files please.[/quote]

Ok, the /etc/drbd.d/r0.res
[code]resource r0 {

on st1.storage.loc {
device /dev/drbd0;
disk /dev/nfs/st1;
address 10.10.0.13:7789;
meta-disk internal;
}

on st2.storage.loc {
device /dev/drbd0;
disk /dev/nfs/st1;
address 10.10.0.14:7789;
meta-disk internal;
}
}[/code]

i don't have more files of drbd

Thanks

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: network transfer slow

Post by TrevorH » 2011/08/30 15:22:58

I am not sure why your /proc/drbd says "Want: 34,000 K/sec" and this looks like the limiting factor in your current setup. Can you post the output from `drbdsetup /dev/drbd0 show` (assuming the syntax for this is still valid in 8.4.0)

Oh, and BTW, the recommendation is that you set the syncer rate to 30% of the maximum available bandwidth of either the network link between the two machine or the throughput of the backing device, whichever is the lower. So to set it to 900M means you need to be seeing about 3000M/s on your network link and the same on the disk based backing store - and yes, those figures are megabytes not bits.

paabloop
Posts: 8
Joined: 2011/08/29 21:04:34

Re: network transfer slow

Post by paabloop » 2011/08/30 17:30:15

[quote]
TrevorH wrote:
I am not sure why your /proc/drbd says "Want: 34,000 K/sec" and this looks like the limiting factor in your current setup. Can you post the output from `drbdsetup /dev/drbd0 show` (assuming the syntax for this is still valid in 8.4.0)

Oh, and BTW, the recommendation is that you set the syncer rate to 30% of the maximum available bandwidth of either the network link between the two machine or the throughput of the backing device, whichever is the lower. So to set it to 900M means you need to be seeing about 3000M/s on your network link and the same on the disk based backing store - and yes, those figures are megabytes not bits.[/quote]

Thanks for you help!!!

here is the out of "drbdsetup /dev/drbd0 show":

[code][root@st1 ~]# drbdsetup /dev/drbd0 show
resource r0 {
options {
cpu-mask "f";
}
net {
max-epoch-size 8000;
max-buffers 8000;
unplug-watermark 16;
allow-two-primaries yes;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
verify-alg "md5";
}
_remote_host {
address ipv4 10.10.0.14:7789;
}
_this_host {
address ipv4 10.10.0.13:7789;
volume 0 {
device minor 0;
disk "/dev/nfs/st1";
meta-disk internal;
disk {
disk-barrier no;
disk-flushes no;
resync-rate 4096000k; # bytes/second
al-extents 3389;
}
}
}
}
[root@st1 ~][/code]

i run a iperf test:

[code][root@st2 ~]# iperf -c 10.10.0.13
------------------------------------------------------------
Client connecting to 10.10.0.13, TCP port 5001
TCP window size: 9.54 MByte (default)
------------------------------------------------------------
[ 3] local 10.10.0.14 port 58912 connected with 10.10.0.13 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 3.92 GBytes 3.37 Gbits/sec
[root@st2 ~]#[/code]

why the transfer over scp, rsync, ftp are so slow?

i run a hdparm -tT /dev/md127:

[code][root@st2 ~]# hdparm -Tt /dev/md127

/dev/md127:
Timing cached reads: 1634 MB in 2.00 seconds = 3816.18 MB/sec
Timing buffered disk reads: 368 MB in 3.00 seconds = 367.62 MB/sec
[root@st2 ~]#[/code]

:S i don't know will do!!!

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

network transfer slow

Post by TrevorH » 2011/08/30 20:36:12

Afraid I have no clue as to what is going on. The output from your drbdsetup command is a lot shorter than the one I get from 8.3.11 which is the latest release that I've used (I tend to steer clear of x.0 releases!).

I can understand scp being slow as that usually ends up being cpu limited as it has to encrypt/decrypt the data and if you are using rsync over ssh then the same applies to that. I've seen no figures for ftp and I wouldn't be able to explain why that might be slow if I did!

Did you try the 8.3.x releases of DRBD?

paabloop
Posts: 8
Joined: 2011/08/29 21:04:34

Re: network transfer slow

Post by paabloop » 2011/08/31 13:07:18

i have drbd 8.4.0 version... i had to compile. centos 6 do not include drbd rpm's

thanks

paabloop
Posts: 8
Joined: 2011/08/29 21:04:34

Re: network transfer slow

Post by paabloop » 2011/08/31 14:11:17

UPDATE: i install drbd 8.3 from El Repo and i have the same results...

Thanks

Post Reply