Tg3 Network Link Unstable

Issues related to configuring your network
User avatar
TrevorH
Forum Moderator
Posts: 21162
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Tg3 Network Link Unstable

Postby TrevorH » 2017/12/06 22:56:15

What does modinfo tg3 say now?
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Takx
Posts: 9
Joined: 2017/12/06 00:17:17

Re: Tg3 Network Link Unstable

Postby Takx » 2017/12/07 00:01:37

Here it go, the version is not the 3.137s anymore:

Code: Select all

[root@infraefika Downloads]# modinfo tg3
filename:       /lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/net/ethernet/broadcom/tg3.ko.xz
firmware:       tigon/tg3_tso5.bin
firmware:       tigon/tg3_tso.bin
firmware:       tigon/tg3.bin
version:        3.137
license:        GPL
description:    Broadcom Tigon3 ethernet driver
author:         David S. Miller (davem@redhat.com) and Jeff Garzik (jgarzik@pobox.com)
rhelversion:    7.4
srcversion:     98953AED539811404DF56C6
alias:          pci:v000010CFd000011A2sv*sd*bc*sc*i*
alias:          pci:v0000106Bd00001645sv*sd*bc*sc*i*
alias:          pci:v0000173Bd000003EAsv*sd*bc*sc*i*
alias:          pci:v0000173Bd000003EBsv*sd*bc*sc*i*
alias:          pci:v0000173Bd000003E9sv*sd*bc*sc*i*
alias:          pci:v0000173Bd000003E8sv*sd*bc*sc*i*
alias:          pci:v00001148d00004500sv*sd*bc*sc*i*
alias:          pci:v00001148d00004400sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016B3sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016B7sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001641sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001683sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001642sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016F3sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001643sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001687sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001686sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001682sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000165Fsv*sd*bc*sc*i*
alias:          pci:v000014E4d00001657sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016B6sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016B2sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016B4sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016B0sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016B5sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016B1sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001656sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001665sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001655sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001691sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001694sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001690sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001692sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001692sv00001025sd00000612bc*sc*i*
alias:          pci:v000014E4d00001692sv00001025sd00000601bc*sc*i*
alias:          pci:v000014E4d000016A0sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001699sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001689sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001688sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001680sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001681sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000165Bsv*sd*bc*sc*i*
alias:          pci:v000014E4d00001684sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001698sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001713sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001712sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016DDsv*sd*bc*sc*i*
alias:          pci:v000014E4d0000166Bsv*sd*bc*sc*i*
alias:          pci:v000014E4d0000166Asv*sd*bc*sc*i*
alias:          pci:v000014E4d00001679sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001678sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001669sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001668sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000167Fsv*sd*bc*sc*i*
alias:          pci:v000014E4d00001693sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001693sv000017AAsd00003056bc*sc*i*
alias:          pci:v000014E4d0000169Bsv*sd*bc*sc*i*
alias:          pci:v000014E4d0000169Asv*sd*bc*sc*i*
alias:          pci:v000014E4d00001674sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001673sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000167Bsv*sd*bc*sc*i*
alias:          pci:v000014E4d00001672sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000167Asv*sd*bc*sc*i*
alias:          pci:v000014E4d000016FEsv*sd*bc*sc*i*
alias:          pci:v000014E4d000016FDsv*sd*bc*sc*i*
alias:          pci:v000014E4d000016F7sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001601sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001600sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000167Esv*sd*bc*sc*i*
alias:          pci:v000014E4d0000167Dsv*sd*bc*sc*i*
alias:          pci:v000014E4d00001677sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001676sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000165Asv*sd*bc*sc*i*
alias:          pci:v000014E4d00001659sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000166Esv*sd*bc*sc*i*
alias:          pci:v000014E4d00001649sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000170Esv*sd*bc*sc*i*
alias:          pci:v000014E4d0000170Dsv*sd*bc*sc*i*
alias:          pci:v000014E4d0000169Dsv*sd*bc*sc*i*
alias:          pci:v000014E4d0000169Csv*sd*bc*sc*i*
alias:          pci:v000014E4d00001696sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016C7sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016C6sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016A8sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016A7sv*sd*bc*sc*i*
alias:          pci:v000014E4d000016A6sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000165Esv*sd*bc*sc*i*
alias:          pci:v000014E4d0000165Dsv*sd*bc*sc*i*
alias:          pci:v000014E4d00001654sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001653sv*sd*bc*sc*i*
alias:          pci:v000014E4d0000164Dsv*sd*bc*sc*i*
alias:          pci:v000014E4d00001648sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001647sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001646sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001645sv*sd*bc*sc*i*
alias:          pci:v000014E4d00001644sv*sd*bc*sc*i*
depends:        ptp
intree:         Y
vermagic:       3.10.0-693.el7.x86_64 SMP mod_unload modversions
signer:         CentOS Linux kernel signing key
sig_key:        DA:18:7D:CA:7D:BE:53:AB:05:BD:13:BD:0C:4E:21:F4:22:B6:A4:9C
sig_hashalgo:   sha256
parm:           tg3_debug:Tigon3 bitmapped debugging message enable value (int)
[root@infraefika Downloads]#



Was looking at dmesg, it actually maps the interface mac address on the startup.

Code: Select all

[root@infraefika network-scripts]# dmesg -e -H | grep tg3
[  +0,017521] tg3.c:v3.137 (May 11, 2014)
[  +0,000735] tg3 0000:02:00.0 eth0: Tigon3 [partno(N/A) rev 5719001] (PCI Express) MAC address 94:18:82:7b:b3:8c
[  +0,000014] tg3 0000:02:00.0 eth0: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[  +0,000012] tg3 0000:02:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[  +0,000010] tg3 0000:02:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit]
[  +0,010487] tg3 0000:02:00.1 eth1: Tigon3 [partno(N/A) rev 5719001] (PCI Express) MAC address 94:18:82:7b:b3:8d
[  +0,000011] tg3 0000:02:00.1 eth1: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[  +0,000008] tg3 0000:02:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[  +0,000007] tg3 0000:02:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit]
[  +0,007692] tg3 0000:02:00.2 eth2: Tigon3 [partno(N/A) rev 5719001] (PCI Express) MAC address 94:18:82:7b:b3:8e
[  +0,000011] tg3 0000:02:00.2 eth2: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[  +0,000007] tg3 0000:02:00.2 eth2: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[  +0,000006] tg3 0000:02:00.2 eth2: dma_rwctrl[00000001] dma_mask[64-bit]
[  +0,014451] tg3 0000:02:00.3 eth3: Tigon3 [partno(N/A) rev 5719001] (PCI Express) MAC address 94:18:82:7b:b3:8f
[  +0,000008] tg3 0000:02:00.3 eth3: attached PHY is 5719C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[  +0,000005] tg3 0000:02:00.3 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[  +0,000004] tg3 0000:02:00.3 eth3: dma_rwctrl[00000001] dma_mask[64-bit]
[  +0,326008] tg3 0000:02:00.0: irq 133 for MSI/MSI-X
[  +0,000025] tg3 0000:02:00.0: irq 134 for MSI/MSI-X
[  +0,000027] tg3 0000:02:00.0: irq 135 for MSI/MSI-X
[  +0,000021] tg3 0000:02:00.0: irq 136 for MSI/MSI-X
[  +0,000019] tg3 0000:02:00.0: irq 137 for MSI/MSI-X
[  +3,802772] tg3 0000:02:00.0 eno1: Link is up at 1000 Mbps, full duplex
[  +0,000014] tg3 0000:02:00.0 eno1: Flow control is off for TX and off for RX
[  +0,000006] tg3 0000:02:00.0 eno1: EEE is enabled
[  +1,916827] tg3 0000:02:00.1: irq 138 for MSI/MSI-X
[  +0,000030] tg3 0000:02:00.1: irq 139 for MSI/MSI-X
[  +0,000019] tg3 0000:02:00.1: irq 140 for MSI/MSI-X
[  +0,000019] tg3 0000:02:00.1: irq 141 for MSI/MSI-X
[  +0,000019] tg3 0000:02:00.1: irq 142 for MSI/MSI-X
[  +6,211297] tg3 0000:02:00.3: irq 143 for MSI/MSI-X
[  +0,000036] tg3 0000:02:00.3: irq 144 for MSI/MSI-X
[  +0,000020] tg3 0000:02:00.3: irq 145 for MSI/MSI-X
[  +0,000029] tg3 0000:02:00.3: irq 146 for MSI/MSI-X
[  +0,000019] tg3 0000:02:00.3: irq 147 for MSI/MSI-X
[  +3,371791] tg3 0000:02:00.3 eno4: Link is up at 1000 Mbps, full duplex
[  +0,000016] tg3 0000:02:00.3 eno4: Flow control is off for TX and off for RX
[  +0,000006] tg3 0000:02:00.3 eno4: EEE is enabled
[root@infraefika network-scripts]#



Where I should check the interfaces configs (only place I looked up to today is /etc/sysconfig/network-scripts/) and over there is all good.


Trying to up it manually:

Code: Select all

[root@infraefika network-scripts]# pwd
/etc/sysconfig/network-scripts


[root@infraefika network-scripts]# cat ifcfg-eno3
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eno3
UUID=5088ae60-e4a0-41bd-890c-5202d99a7b7a
DEVICE=eno3
ONBOOT=yes
HWADDR=94:18:82:7b:b3:8e
IPADDR=10.40.198.34
GATEWAY=10.40.196.1
NETMASK=255.255.252.0
NM_CONTROLLED=no


[root@infraefika network-scripts]# ifup ifcfg-eno3
ERROR     : [/etc/sysconfig/network-scripts/ifup-eth] Device eno3 not present, initializing delayed.
[root@infraefika network-scripts]#

User avatar
TrevorH
Forum Moderator
Posts: 21162
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Tg3 Network Link Unstable

Postby TrevorH » 2017/12/07 00:24:51

/lib/modules/3.10.0-693.el7.x86_64/kernel/drivers/net/ethernet/broadcom/tg3.ko.xz


So that shows two things, first that you're back to using the CentOS kernel supplied tg3 module and also that you're still running the original 7.4 kernel and you need to yum update. If any of your links are bonded then kernel 3.10.0-693.5.2 has a fix for bonding problems and on my machine that made the entire slave interface disappear completely. There's now a newer kernel still which you'll get on update and that's an 'important' security update.

There'll also be some messages in your log about the renaming of the ethX interfaces to their enoY versions and your grep missed those.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Takx
Posts: 9
Joined: 2017/12/06 00:17:17

Re: Tg3 Network Link Unstable

Postby Takx » 2017/12/07 17:16:35

Hey there,

I did update, but got even a newer kernel version

Code: Select all

[root@infraefika network-scripts]# uname -r
3.10.0-693.11.1.el7.x86_64
[root@infraefika network-scripts]#


But at the same time I edited the "/etc/sysconfig/network-scripts" config files, removing the MAC lock I had placed, after one more reboot and the interfaces were all UP again (tried this because you talked about interface name changes during boot).

I removed some configs on those files, but I guess there is a "default" value for all those parameters somewhere on the system.

Now I have all the interfaces UP but something changed on the routes, I keeps laying Eno1 for default, even I changing the config on the ifcfg file.

Code: Select all

[root@infraefika network-scripts]# cat ifcfg-eno1
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=no
IPV6INIT=no
NAME=eno1
UUID=36b5427c-ae83-4d90-8594-903497a6efb6
DEVICE=eno1
ONBOOT=yes
IPADDR=10.40.198.88
GATEWAY=10.40.196.1
NETMASK=255.255.252.0
NM_CONTROLLED=no
METRIC=10
[root@infraefika network-scripts]# cat ifcfg-ovirtmgmt
# Generated by VDSM version 4.19.37-1.el7.centos
DEVICE=ovirtmgmt
TYPE=Bridge
DELAY=0
STP=off
ONBOOT=yes
BOOTPROTO=static
MTU=1500
DEFROUTE=yes
NM_CONTROLLED=no
IPV6INIT=no
DNS1=10.41.24.17
DNS2=10.41.22.176
IPADDR=10.40.196.188
GATEWAY=10.40.196.1
NETMASK=255.255.252.0
METRIC=0
[root@infraefika network-scripts]#


And the routing table after a systemctl restart network.service:

Code: Select all

[root@infraefika network-scripts]# route
Tabela de Roteamento IP do Kernel
Destino         Roteador        MáscaraGen.    Opções Métrica Ref   Uso Iface
default         gateway         0.0.0.0         UG    0      0        0 eno1
10.40.196.0     0.0.0.0         255.255.252.0   U     0      0        0 eno1
10.40.196.0     0.0.0.0         255.255.252.0   U     0      0        0 eno2
10.40.196.0     0.0.0.0         255.255.252.0   U     0      0        0 ovirtmgmt
10.40.196.0     0.0.0.0         255.255.252.0   U     0      0        0 ovirtmgmt-eno3
link-local      0.0.0.0         255.255.0.0     U     1002   0        0 eno1
link-local      0.0.0.0         255.255.0.0     U     1003   0        0 eno2
link-local      0.0.0.0         255.255.0.0     U     1021   0        0 ovirtmgmt-eno3
link-local      0.0.0.0         255.255.0.0     U     1023   0        0 ovirtmgmt
[root@infraefika network-scripts]#



Reading around, saw people talking about interface.route file on the same folder, but on my version of CentOS it doesn't exist, is there anywhere else I could see the routing configuration? Also, how can I check if dhcp is really down and not messing around?


Thanks!

User avatar
TrevorH
Forum Moderator
Posts: 21162
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Tg3 Network Link Unstable

Postby TrevorH » 2017/12/07 19:01:56

Don't specify GATEWAY= in more than one ifcfg file or they fight for which one is used and it's random which of them it will be. Specify it in one file only.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Takx
Posts: 9
Joined: 2017/12/06 00:17:17

Re: Tg3 Network Link Unstable

Postby Takx » 2017/12/08 11:38:41

Hey,

I removed from the other files, but something weird is happening (probably some tool I don't know), even setting the ovirtmgmt interface with the gateway, it keeps putting the interface eno1 as default route, if I "reconfigure manually" to put ovirtmgmt as default route (and remove the eno1 route) I can't ping the server or VMs from a machine on the network.

Another point, you know if the interface on the bridge (ovirtmgmt) is supposed to answer ping (when the default route is at eno1 and the pings are working)? I don't remember if it was answering before these problems started.

I also started having problems to access the VMs on oVirt with console (SPICE / Virt-Manager) from the network, get an error on Socket I/O Timeout, probably linked with all the network mess it is right now.

Code: Select all

[root@infraefika network-scripts]# cat ifcfg-eno1
TYPE=Ethernet
BOOTPROTO=static
DEFROUTE=no
IPV6INIT=no
NAME=eno1
UUID=36b5427c-ae83-4d90-8594-903497a6efb6
DEVICE=eno1
ONBOOT=yes
IPADDR=10.40.198.88
NETMASK=255.255.252.0
NM_CONTROLLED=no
[root@infraefika network-scripts]#
[root@infraefika network-scripts]# cat ifcfg-ovirtmgmt
# Generated by VDSM version 4.19.37-1.el7.centos
DEVICE=ovirtmgmt
TYPE=Bridge
DELAY=0
STP=off
ONBOOT=yes
BOOTPROTO=static
DEFROUTE=yes
NM_CONTROLLED=no
IPV6INIT=no
DNS1=10.41.24.17
DNS2=10.41.22.176
IPADDR=10.40.196.188
GATEWAY=10.40.196.1
NETMASK=255.255.252.0
MTU=1500
[root@infraefika network-scripts]#
[root@infraefika network-scripts]# route
Tabela de Roteamento IP do Kernel
Destino         Roteador        MáscaraGen.    Opções Métrica Ref   Uso Iface
default         gateway         0.0.0.0         UG    0      0        0 eno1
10.40.196.0     0.0.0.0         255.255.252.0   U     0      0        0 eno1
10.40.196.0     0.0.0.0         255.255.252.0   U     0      0        0 eno2
10.40.196.0     0.0.0.0         255.255.252.0   U     0      0        0 ovirtmgmt
10.40.196.0     0.0.0.0         255.255.252.0   U     0      0        0 ovirtmgmt-eno3
link-local      0.0.0.0         255.255.0.0     U     1002   0        0 eno1
link-local      0.0.0.0         255.255.0.0     U     1003   0        0 eno2
link-local      0.0.0.0         255.255.0.0     U     1021   0        0 ovirtmgmt-eno3
link-local      0.0.0.0         255.255.0.0     U     1023   0        0 ovirtmgmt
[root@infraefika network-scripts]#

User avatar
TrevorH
Forum Moderator
Posts: 21162
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Tg3 Network Link Unstable

Postby TrevorH » 2017/12/08 12:32:52

That looks like you're trying to put an ip address on the physical interface as well as on the bridge. I can see that causing this problem and it's not how I've ever done this (though I only use kvm not ovirt). The physical interfaces I configure have no ip address - that's on the bridge on top. You're also missing BRIDGE= from your physical interface file.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke