system unresponsive with massive tg3 errors

Issues related to configuring your network
Post Reply
shanghaiscott
Posts: 1
Joined: 2017/09/05 22:53:51

system unresponsive with massive tg3 errors

Post by shanghaiscott » 2017/09/07 03:37:04

A few days ago, a fully updated CentOS 7 server just started spewing stuff like this on the console:

[ 1406.543906] tg3 0000:03:00.0 enp3s0: 0x00000ce0: 0xcf8faa02, 0x00000000, 0x0000005c, 0x00041028
~[ 1406.543911] tg3 0000:03:00.0 enp3s0: 0x00000cf0: 0x00000000, 0x5000003c, 0x00000000, 0x00000000
[ 1406.543916] tg3 0000:03:00.0 enp3s0: 0x00001000: 0x00000002, 0x00000000, 0xa00007e0, 0x00000000
[ 1406.543921] tg3 0000:03:00.0 enp3s0: 0x00001010: 0x003c03c0, 0x000007e0, 0x00000000, 0x00000000
[ 1406.543926] tg3 0000:03:00.0 enp3s0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[[ 1406.543931] tg3 0000:03:00.0 enp3s0: 0x00001440: 0x0000003c, 0x0000003c, 0x0000003c, 0x0000003c
[ 1406.543936] tg3 0000:03:00.0 enp3s0: 0x00001450: 0x0000003c, 0x0000003c, 0x0000003c, 0x0000003c
[ 1406.543940] tg3 0000:03:00.0 enp3s0: 0x00001460: 0x0000003c, 0x0000003c, 0x0000003c, 0x0000003c
[ 1406.543945] tg3 0000:03:00.0 enp3s0: 0x00001470: 0x0000003c, 0x0000003c, 0x0000003c, 0x0000003c
3[ 1406.543950] tg3 0000:03:00.0 enp3s0: 0x00001480: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[ 1406.543955] tg3 0000:03:00.0 enp3s0: 0x00001800: 0x00000016, 0x00000000, 0x0000003c, 0x00000000
[ 1408.547427] tg3 0000:02:00.0 enp2s0: 0x00000890: 0x00000000, 0x0000000e, 0x00000000, 0x00000000
[ 1408.547431] tg3 0000:02:00.0 enp2s0: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000
2[ 1408.547436] tg3 0000:02:00.0 enp2s0: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[ 1408.547441] tg3 0000:02:00.0 enp2s0: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x004a0000
[ 1408.547446] tg3 0000:02:00.0 enp2s0: 0x00000ce0: 0xcf8198c6, 0x00000000, 0x0000004a, 0x00041028
[ 1408.547451] tg3 0000:02:00.0 enp2s0: 0x00000cf0: 0x00000000, 0x5000002a, 0x00000000, 0x00000000
~[ 1408.547456] tg3 0000:02:00.0 enp2s0: 0x00001000: 0x00000002, 0x00000000, 0xa0000db6, 0x00000000
[ 1408.547461] tg3 0000:02:00.0 enp2s0: 0x00001010: 0x002a02a0, 0x00000db6, 0x00000000, 0x00000000
[ 1408.547466] tg3 0000:02:00.0 enp2s0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[[ 1408.547471] tg3 0000:02:00.0 enp2s0: 0x00001440: 0x0000002a, 0x0000002a, 0x0000002a, 0x0000002a
[ 1408.547476] tg3 0000:02:00.0 enp2s0: 0x00001450: 0x0000002a, 0x0000002a, 0x0000002a, 0x0000002a
[ 1408.547480] tg3 0000:02:00.0 enp2s0: 0x00001460: 0x0000002a, 0x0000002a, 0x0000002a, 0x0000002a
3[ 1408.547485] tg3 0000:02:00.0 enp2s0: 0x00001470: 0x0000002a, 0x0000002a, 0x0000002a, 0x0000002a
[ 1408.547490] tg3 0000:02:00.0 enp2s0: 0x00001480: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[ 1408.547495] tg3 0000:02:00.0 enp2s0: 0x00001800: 0x00000016, 0x00000000, 0x0000002a, 0x00000000
2[ 1408.547500] tg3 0000:02:00.0 enp2s0: 0x00001840: 0x00000000, 0x00000000, 0x00000200, 0x00000010
[ 1408.547505] tg3 0000:02:00.0 enp2s0: 0x00001850: 0xd1c46290, 0x00000000, 0x00004290, 0x002a002a
~[ 1408.547509] tg3 0000:02:00.0 enp2s0: 0x00001860: 0x01000000, 0x00000000, 0x00000000, 0x00000000
[ 1408.547514] tg3 0000:02:00.0 enp2s0: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 1408.547520] tg3 0000:02:00.0 enp2s0: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[[ 1408.547524] tg3 0000:02:00.0 enp2s0: 0x00002010: 0x00000181, 0x00000001, 0x00790807, 0x00000000
[ 1408.547529] tg3 0000:02:00.0 enp2s0: 0x00002100: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1408.547534] tg3 0000:02:00.0 enp2s0: 0x00002110: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1408.547539] tg3 0000:02:00.0 enp2s0: 0x00002120: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
3[ 1408.547544] tg3 0000:02:00.0 enp2s0: 0x00002130: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1408.547549] tg3 0000:02:00.0 enp2s0: 0x00002140: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1408.547554] tg3 0000:02:00.0 enp2s0: 0x00002150: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
2[ 1408.547559] tg3 0000:02:00.0 enp2s0: 0x00002160: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1408.547564] tg3 0000:02:00.0 enp2s0: 0x00002170: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1408.547569] tg3 0000:02:00.0 enp2s0: 0x00002180: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1408.547573] tg3 0000:02:00.0 enp2s0: 0x00002190: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
~[ 1408.547578] tg3 0000:02:00.0 enp2s0: 0x000021a0: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1408.547583] tg3 0000:02:00.0 enp2s0: 0x000021b0: 0x00088241, 0x000c562b, 0x00000193, 0x00000000
[ 1413.567382] tg3 0000:03:00.0 enp3s0: 0x000000b0: 0x00000000, 0x00000001, 0x00000000, 0x00000001

I am unable to login or interact with the server. I am connecting to a terminal server which is connected to the serial console of the server. From here I can just watch this type of output scroll bye infinitely. I have tried rebooting after a power cycle with older kernels. The same thing occurs. I have to bug my co-lo provider for hands on and this is getting tedious.

Only when I boot via the "rescue mode" grub2 menu option do I get a system I can login to.

Can anyone provide a clue as to what is going on?

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: system unresponsive with massive tg3 errors

Post by hunter86_bg » 2017/09/12 03:24:03

As per google 'tg3' is a module responsible for one of your network cards (Broadcom based).
Have you done any testing with ethtool to verify that the NIC is OK?
When you boot from older kernel, do you experience the same?

Post Reply