Page 1 of 1

Poor performance with HP NC552SFP 10GbE NIC

Posted: 2018/06/09 21:05:28
by Diglet
I have CentOS 7.5 running on two HP DL360-G7 servers, each with 96GB RAM. The BIOS and all the firmware are up-to-date on the servers. I recently installed one NC552SFP 10GbE fiber NIC in each server. The NIC's firmware is also up-to-date. I then configured IP on the new NIC interface and deactivated the old interfaces. Finally, I disconnected the no-longer-in-use 1Gb copper connections so everything is definitely running over the 10GbE fiber connection.

The data transfer rates between the two servers is about the same as it was with the 1Gb copper connections; about 102MB/s. Obviously there is a performance problem here. The port status on the switch as well as both servers show 10Gb full-duplex.

Any ideas why this is happening and how to correct it? Any help is appreciated. I've searched the internet high and low and cannot find a solution.

Re: Poor performance with HP NC552SFP 10GbE NIC

Posted: 2018/06/09 22:56:27
by TrevorH
What is the output from ethtool $interface (where $interface is the name of the right interface) ?

What tool are you using to measure the throughput and to generate that traffic?

Re: Poor performance with HP NC552SFP 10GbE NIC

Posted: 2018/06/12 18:19:00
by Diglet
TrevorH wrote:
2018/06/09 22:56:27
What is the output from ethtool $interface (where $interface is the name of the right interface) ?

What tool are you using to measure the throughput and to generate that traffic?
I created a 10GB ramdisk on each server and transferred an 8GB file between the ramdisks using scp. I did this to eliminate the possibility of slow disk I/O affecting results. I calculated the overall throughput simply by dividing 8192MB by the number of seconds it took to complete the transfer.

Here is the ethtool output:

# ethtool ens2f0
Settings for ens2f0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 1
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: g
Wake-on: g
Current message level: 0x00002000 (8192)
hw
Link detected: yes

# ethtool -i ens2f0
driver: be2net
version: 11.4.0.0r
firmware-version: 11.1.183.23
expansion-rom-version:
bus-info: 0000:09:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: no
supports-priv-flags: yes

Re: Poor performance with HP NC552SFP 10GbE NIC

Posted: 2018/06/12 19:48:13
by TrevorH
Yeah, scp is not a good test tool for this. It uses encryption and is limited to a single core for that so the bottleneck in that case becomes "how fast can you encrypt the data".

Try installing and using iperf3 (from EPEL) or configuring e.g an httpd server to serve the files you want to transfer and use curl/wget to pull the data,

Re: Poor performance with HP NC552SFP 10GbE NIC

Posted: 2018/06/12 23:16:57
by Diglet
TrevorH wrote:
2018/06/12 19:48:13
Yeah, scp is not a good test tool for this. It uses encryption and is limited to a single core for that so the bottleneck in that case becomes "how fast can you encrypt the data".

Try installing and using iperf3 (from EPEL) or configuring e.g an httpd server to serve the files you want to transfer and use curl/wget to pull the data,
Well, that fixed the "problem". LOL. scp is definitely NOT suitable for network performance testing.

I performed the exact same file transfer as I described previously (ramdisk to ramdisk) but with httpd and wget. wget reported a transfer rate of 279MB/s which is much faster than the scp transfer, but I suspect slower than what can be achieved on a 10 GbE network. So, what is the preferred/recommended or most reliable means to performance test data transfers between servers? And, what is a reasonable expectation I should have of my new 10 GbE network?

And thank you for your help.

Re: Poor performance with HP NC552SFP 10GbE NIC

Posted: 2018/06/18 04:14:33
by hunter86_bg
The 'iperf3' , as mentioned by Trevor, is your swiss army knife.