Centos 7 Ethernet 10G running at 1G

[lyzhao]

Dear all,
I have two Linux system Dell T440 servers (CentOS 7.8). Each server has a Gigabit and a 10G network card. Now the Gigabit cards are interconnected through switches, and the 10G network cards are directly interconnected through optical fibers. Now the transmission rate between the two servers can only reach Gigabit (~100MB/s) speed using both Gigabit network cards and 10 Gigabit network cards.
ethtool shows: lyzs7 same as lyzs8

Any help will be appreciated.
Youzhao Lan

Code: Select all

[root@lyzs8 lyzhao]# ethtool  p5p2
Settings for p5p2:
 Supported ports: [ FIBRE ]
 Supported link modes:   1000baseT/Full 
                         10000baseT/Full 
 Supported pause frame use: Symmetric Receive-only
 Supports auto-negotiation: Yes
 Supported FEC modes: Not reported
 Advertised link modes:  1000baseT/Full 
                         10000baseT/Full 
 Advertised pause frame use: No
 Advertised auto-negotiation: Yes
 Advertised FEC modes: Not reported
 Speed: 10000Mb/s
 Duplex: Full
 Port: FIBRE
 PHYAD: 1
 Transceiver: internal
 Auto-negotiation: on
 Supports Wake-on: d
 Wake-on: d
 Current message level: 0x00000000 (0)
          
 Link detected: yes
[root@lyzs8 lyzhao]# ethtool  em2
Settings for em2:
 Supported ports: [ TP ]
 Supported link modes:   10baseT/Half 10baseT/Full 
                         100baseT/Half 100baseT/Full 
                         1000baseT/Half 1000baseT/Full 
 Supported pause frame use: No
 Supports auto-negotiation: Yes
 Supported FEC modes: Not reported
 Advertised link modes:  10baseT/Half 10baseT/Full 
                         100baseT/Half 100baseT/Full 
                         1000baseT/Half 1000baseT/Full 
 Advertised pause frame use: Symmetric
 Advertised auto-negotiation: Yes
 Advertised FEC modes: Not reported
 Link partner advertised link modes:  10baseT/Half 10baseT/Full 
                                      100baseT/Half 100baseT/Full 
                                      1000baseT/Full 
 Link partner advertised pause frame use: Symmetric
 Link partner advertised auto-negotiation: Yes
 Link partner advertised FEC modes: Not reported
 Speed: 1000Mb/s
 Duplex: Full
 Port: Twisted Pair
 PHYAD: 2
 Transceiver: internal
 Auto-negotiation: on
 MDI-X: off
 Supports Wake-on: g
 Wake-on: d
 Current message level: 0x000000ff (255)
          drv probe link timer ifdown ifup rx_err tx_err
 Link detected: yes
 

[TrevorH]

That info clearly shows that p5p2 is up and running at 10000Mbps and em2 is up at 1000Mbps. What leads you to believe that it is not running at 10Gbps?

Also, CentOS 7.8 is deader than dead and needs to be updated to 7.9 plus all the other patches that have come out in the last 3 years since 7.9 was released.

Are you sure your routing is correct and the traffic is going over the interface you think it is?

Did you try installing iperf3 on both and running tests to see what throughput you get?

[jlehtone]

One can also look at interface statistics:

Code: Select all

ip -s li
ethtool  -S p5p2

Lots of errors could be a hint. Also, if packet counts go up only in the 1G link, then traffic does not route through the 10G link ...

If the NIC has tranceiver module for fiber, then ethtool -m p5p2 would tell about the module.

[lyzhao]

Great thanks for your help. I used scp to transfer files to test the speed, the interconnection network based on the 10G network card can only reach ~120MB/s. Now I use iperf3 to test, and the results show that it can reach ~9.8Gbits/s, so I think it is normal. I suspect that the write rate of other hardware such as hard drives cannot match.

Best regards,
Youzhao Lan

[jlehtone]

You can run iotop on the side. It may show disk access speed.
iftop (from EPEL) shows interface transfer stat.


The scp is somewhat deprecated as it has (had) security issues (despite being part of ssh).
I do recommend rsync as alternative. It does use ssh for transport (just like scp).
(At least with rsync it is clear that many small files transfer slower than one huge file. Might depend on involved filesystems too.)

[TrevorH]

Using scp will encrypt all the traffic before it is sent and that encryption will run on only a single core. It might be a different core each time the process is despatched too so any cache benefits of running on a single core will be lost - you could use taskset to pin it to one core to see if it makes any difference but basically the bottleneck will be cpu. You may be able to make it marginally quicker by specifying -c arcfour (I think that still works) which is the fastest but most crackable encryption that ssh offers. Or test using a non-encrypted transfer mechanism like plain http for example.