10GbE connectivity and speed issues

Issues related to configuring your network
Post Reply
agi
Posts: 5
Joined: 2018/02/22 00:23:04
Contact:

10GbE connectivity and speed issues

Post by agi » 2018/04/05 23:28:45

Hello all,

We have a server with two 1GbE interfaces, and two 10GbE interfaces (only one used). We are having a couple problems. First, it isn't possible so far to have the 1Gb and 10Gb interfaces enabled at the same time, otherwise only one of the interfaces is reachable. Secondly, the upload speed is always <1 Mbps but download is good (over 5,000 Mbps depending on the speedtest.net server). The 10Gb card is the SolarFlare in lspci (below), a Flareon Ultra SFN8722.

I ran the getinfo.sh script with the machine configured how I'd like it to work, with eth0 and eth2 enabled. eth1 is just an internal 1GbE connection. With this configuration, only the 1Gb 128.x.x.3 address is reachable and the 10Gb 128.x.x.4 address is not (eth1 is also reachable)

To work around the reachability issue, i just disabled eth0 and assigned that address to the 10Gb NIC with eth2:0, at least all our services are reachable that way. I was reading through some guides to optimize 10Gb connection, and have set the MTU to 9000 and txqueuelength to 3000. Tried setting those back to 1500/1000 but didn't change anything.

I have checked with network operations and we have gone thru everything and they believe it is a config issue with our machine. We wonder if it's an issue with asymmetric routing, since both NICs are on the same subnet, requests are going out one NIC and are coming back through another? Perhaps there needs to be some updates to the routing table? It looks like it is routing everything thru eth0. Both NICs land on the same VLAN and are configured for the same subnet, but are homed to two different switches.



Information for networking problems.

Code: Select all

== BEGIN uname -rmi ==
2.6.32-696.13.2.el6.x86_64 x86_64 x86_64
== END   uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
centos-release-6-9.el6.12.3.x86_64
epel-release-6-8.noarch
== END   rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
CentOS release 6.9 (Final)
== END   cat /etc/redhat-release ==

== BEGIN getenforce ==
Disabled
== END   getenforce ==

== BEGIN free -m ==
             total       used       free     shared    buffers     cached
Mem:         32010      23705       8304          1         53      16804
-/+ buffers/cache:       6847      25162
Swap:         8191         10       8181
== END   free -m ==

== BEGIN lspci -nn ==
00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2 [8086:6f00] (rev 01)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 [8086:6f02] (rev 01)
00:02.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2 [8086:6f04] (rev 01)
00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 [8086:6f08] (rev 01)
00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3 [8086:6f0a] (rev 01)
00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management [8086:6f28] (rev 01)
00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug [8086:6f29] (rev 01)
00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors [8086:6f2a] (rev 01)
00:05.4 PIC [0800]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D I/O APIC [8086:6f2c] (rev 01)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05)
00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05)
00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05)
00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05)
00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
00:1c.5 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 [8086:8d1a] (rev d5)
00:1c.7 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 [8086:8d1e] (rev d5)
00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05)
02:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] [1000:005d] (rev 02)
04:00.0 Ethernet controller [0200]: Solarflare Communications SFC9220 [1924:0a03] (rev 02)
04:00.1 Ethernet controller [0200]: Solarflare Communications SFC9220 [1924:0a03] (rev 02)
06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
08:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
09:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f81] (rev 01)
ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f36] (rev 01)
ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1 [8086:6f37] (rev 01)
ff:0b.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug [8086:6f76] (rev 01)
ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe0] (rev 01)
ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe1] (rev 01)
ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe2] (rev 01)
ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe3] (rev 01)
ff:0c.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe4] (rev 01)
ff:0c.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe5] (rev 01)
ff:0c.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe6] (rev 01)
ff:0c.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6fe7] (rev 01)
ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ff8] (rev 01)
ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ff9] (rev 01)
ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffc] (rev 01)
ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffd] (rev 01)
ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffe] (rev 01)
ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent [8086:6f1d] (rev 01)
ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent [8086:6f34] (rev 01)
ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f1e] (rev 01)
ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f7d] (rev 01)
ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox [8086:6f1f] (rev 01)
ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6fa0] (rev 01)
ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6f30] (rev 01)
ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6fa8] (rev 01)
ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6f71] (rev 01)
ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6faa] (rev 01)
ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fab] (rev 01)
ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fac] (rev 01)
ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fad] (rev 01)
ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast [8086:6fae] (rev 01)
ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6faf] (rev 01)
ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control [8086:6fb0] (rev 01)
ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control [8086:6fb1] (rev 01)
ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error [8086:6fb2] (rev 01)
ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error [8086:6fb3] (rev 01)
ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbc] (rev 01)
ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbd] (rev 01)
ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbe] (rev 01)
ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface [8086:6fbf] (rev 01)
ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control [8086:6fb4] (rev 01)
ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control [8086:6fb5] (rev 01)
ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error [8086:6fb6] (rev 01)
ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error [8086:6fb7] (rev 01)
ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS [8086:6f68] (rev 01)
ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Broadcast [8086:6f6e] (rev 01)
ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6f6f] (rev 01)
ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 1 - Channel 0 Thermal Control [8086:6fd0] (rev 01)
ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface [8086:6fb8] (rev 01)
ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface [8086:6fb9] (rev 01)
ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface [8086:6fba] (rev 01)
ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 2/3 Interface [8086:6fbb] (rev 01)
ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f98] (rev 01)
ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f99] (rev 01)
ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f9a] (rev 01)
ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6fc0] (rev 01)
ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f9c] (rev 01)
ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f88] (rev 01)
ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit [8086:6f8a] (rev 01)
== END   lspci -nn ==

== BEGIN lsusb ==
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:800a Intel Corp. 
Bus 002 Device 002: ID 8087:8002 Intel Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
== END   lsusb ==

== BEGIN rpm -qa kmod\* kmdl\* ==
== END   rpm -qa kmod\* kmdl\* ==

== BEGIN ifconfig -a ==
eth0      Link encap:Ethernet  HWaddr 14:DD:A9:D4:B9:BC  
          inet addr:128.x.x.3  Bcast:128.x.x.15  Mask:255.255.255.240
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1386710 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2659414 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:179564398 (171.2 MiB)  TX bytes:2473382015 (2.3 GiB)
          Memory:fbe00000-fbe7ffff 

eth1      Link encap:Ethernet  HWaddr 14:DD:A9:D4:B9:BD  
          inet addr:10.10.10.247  Bcast:10.10.10.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:100671082 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9545110 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:149620420717 (139.3 GiB)  TX bytes:1471833446 (1.3 GiB)
          Memory:fbd00000-fbd7ffff 

eth2      Link encap:Ethernet  HWaddr 00:0F:53:4C:8E:60  
          inet addr:128.x.x.4  Bcast:128.x.x.15  Mask:255.255.255.240
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:30124184 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9248702 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3000 
          RX bytes:39795705592 (37.0 GiB)  TX bytes:3840901530 (3.5 GiB)
          Interrupt:42 

eth3      Link encap:Ethernet  HWaddr 00:0F:53:4C:8E:61  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:45 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:5729695 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5729695 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4477043947 (4.1 GiB)  TX bytes:4477043947 (4.1 GiB)

== END   ifconfig -a ==

== BEGIN brctl show ==
bridge name	bridge id		STP enabled	interfaces
== END   brctl show ==

== BEGIN route -n ==
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
128.x.x.0    0.0.0.0         255.255.255.240 U     0      0        0 eth0
128.x.x.0    0.0.0.0         255.255.255.240 U     0      0        0 eth2
10.10.10.0      0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U     1004   0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1005   0        0 eth1
0.0.0.0         128.x.x.1    0.0.0.0         UG    0      0        0 eth0
== END   route -n ==

== BEGIN sysctl -a | grep .rp_filter ==
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth2.rp_filter = 1
net.ipv4.conf.eth2.arp_filter = 0
net.ipv4.conf.eth1.rp_filter = 1
net.ipv4.conf.eth1.arp_filter = 0
net.ipv4.conf.eth3.rp_filter = 1
net.ipv4.conf.eth3.arp_filter = 0
== END   sysctl -a | grep .rp_filter ==

== BEGIN ip rule show ==
0:	from all lookup local 
32766:	from all lookup main 
32767:	from all lookup default 
== END   ip rule show ==

== BEGIN ip route show ==
128.x.x.0/28 dev eth0  proto kernel  scope link  src 128.x.x.3 
128.x.x.0/28 dev eth2  proto kernel  scope link  src 128.x.x.4 
10.10.10.0/24 dev eth1  proto kernel  scope link  src 10.10.10.247 
169.254.0.0/16 dev eth2  scope link  metric 1002 
169.254.0.0/16 dev eth0  scope link  metric 1004 
169.254.0.0/16 dev eth1  scope link  metric 1005 
default via 128.x.x.1 dev eth0 
== END   ip route show ==

== BEGIN egrep 'net|hosts' /etc/nsswitch.conf ==
#hosts:     db files nisplus nis dns
hosts:      files dns
#networks:   nisplus [NOTFOUND=return] files
#netmasks:   nisplus [NOTFOUND=return] files     
netmasks:   files
networks:   files
netgroup:   files ldap
== END   egrep 'net|hosts' /etc/nsswitch.conf ==

== BEGIN chkconfig --list | grep -Ei 'network|wpa' ==
network        	0:off	1:off	2:on	3:on	4:on	5:on	6:off
== END   chkconfig --list | grep -Ei 'network|wpa' ==


tomkep
Posts: 38
Joined: 2018/04/25 13:30:50

Re: 10GbE connectivity and speed issues

Post by tomkep » 2018/04/25 13:35:06

It looks like you are trying to connect both interfaces to the same network segments. In this case you very likely need to disable rp_filter on BOTH of them:

Code: Select all

echo 0 > /proc/sys/net/ipv4/conf/<interface-1-name>/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/<interface-2-name>/rp_filter

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: 10GbE connectivity and speed issues

Post by TrevorH » 2018/04/25 14:26:55

I'm not entirely sure what you're trying to achieve but having two interfaces with ip addresses that reside in the same subnet is almost always problematical. I'm not sure why you're not using bonding to do what you need, perhaps mode=1 with the 10GbE interface as primary and the 1GbE interface as backup with primary_reselect=always.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply