Dell Server Communication - not getting GB speeds at all

[uncdubdiver]

I have 2 Dell Servers connected on my network, each has 2x 1Gb NIC. I've connected both dell servers on eth1 together with a standard Cat5E and Cat6 cable, and attempting to transfer file(s) between them. The fastest I'm able to receive in terms of transfer speeds is 10.8MB/s. Using ethtool on both servers, they are both stating: Speed 1000Mb/s, Duplex Full.

I don't know what to do, I have multiple +100G files that I need to transfer from 1 server to another (VMs), and it's going to take probably a few days to do it at this speed.

Any help is greatly appreciated, and my apologies if this is already answered somewhere else, I'm new to the CentOS formum.

[TrevorH]

What are using to do the transfer?

[uncdubdiver]

scp command.

And another thing I'm noticing...when I try to transfer something over eth0, then try to transfer something else over the eth1 (both transferring out from the server), both speeds drop down to <6MB/s. They should be over completely separate NICs, but you'd never know that based on how slow it's going.

[TrevorH]

First step, verify if it's network related by using something that doesn't use encryption. I'd recommend installing iperf3 from the EPEL repo and running that between your servers to test the link speed. Read the fine man page once it's installed and run one side in server mode and connect to it in client mode from the other one. That will give you numbers just about the network connection. On a Gigabit connection you should see iperf3 figures approaching that speed.

The reason for not using scp for testing this is that the communication is encrypted and that can very easily bottleneck on cpu as it's single threaded and thus will only use one core.

[uncdubdiver]

Thanks for the response. I'm installing it now on both machines and will run a test and get back to you guys shortly...

[uncdubdiver]

That must be it, must be the fact that it's being sent over encrypted...it's not transferring anywhere near the speed it should be.

Code: Select all

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.0.0.95, port 43406
[  5] local 10.0.0.90 port 5201 connected to 10.0.0.95 port 43408
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   108 MBytes   905 Mbits/sec
[  5]   1.00-2.00   sec   112 MBytes   941 Mbits/sec
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec
[  5]   4.00-5.00   sec   112 MBytes   942 Mbits/sec
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec
[  5]   6.00-7.00   sec   112 MBytes   942 Mbits/sec
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec
[  5]   9.00-10.00  sec   112 MBytes   942 Mbits/sec
[  5]  10.00-10.04  sec  4.58 MBytes   939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.04  sec  1.10 GBytes   938 Mbits/sec                  receiver

Thank you for the help on understanding this one!!!!

[TrevorH]

Well that does show you the connection is ok.

So you have a few possibilities about how to do this then. One is to use the least cpu intensive algorithm for scp from the list in man ssh_config then search for Ciphers. Another is to find a transfer method that doesn't involve encryption - for example, setting it up as an NFS export on one machine and mounting it on the other as a remote filesystem.

[uncdubdiver]

I didn't even think of doing NFS. I did it temporarily with FTP and it seemed to move a lot faster than SCP did. I have 1 more large 199G VM to transfer over tonight, so I will use NFS for that one.

[TrevorH]

Using ftp would work, I suggested nfs as probably being easier to set up but if you already have ftp up and running then it's probably easier just to use that.