[RESOLVED] stack trace error on ethernet during shutdown/reboot

General support questions
bakcompat
Posts: 29
Joined: 2006/09/08 21:22:44

[RESOLVED] stack trace error on ethernet during shutdown/reb

Post by bakcompat » 2012/07/26 18:39:01

Hi, I've got some Dell Poweredge 2650s running CentOS5. I wiped one to load CentOS 6.3 (32 bit) and make it a DHCP server. I did an install of: Base, Gnome Desktop, & X Windows. Basically, the bare minimum to get a desktop UI and install everything from there. Assign a static IP and yum update, no problem. On shutdown however, it locks up when it's killing ethernet, giving me stack trace errors with different hex code data every time. This photo shows 'irqbalance' at issue. That's not there every time. NetWorkManager does seem to bomb every time though.

This box has been running linux since Red Hat 7.2 and was last running CentOS 5.7.... so what can I do to resolve this issue?

Searches so far have yielded little information. Other people have encountered this error from a wide gamut of processes. One person suggested booting the kernel with acpi=off, which I did, to no change on reboot/shutdown. So, what to look at next?


[img]http://i73.photobucket.com/albums/i222/BakCompat/Linux/stacktraceerror.jpg[/img]

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: stack trace error on ethernet during shutdown/reboot

Post by pschaff » 2012/08/01 13:59:48

You have not given us much to go on, such as what hardware is present. Please [url=http://www.centos.org/modules/newbb/viewtopic.php?topic_id=28723&forum=54]provide more information about your system[/url] by running "./getinfo.sh" and showing us the output file.

bakcompat
Posts: 29
Joined: 2006/09/08 21:22:44

Re: stack trace error on ethernet during shutdown/reboot

Post by bakcompat » 2012/08/06 21:28:45

FC17 installed on one of these boxes just fine. I wiped one and loaded up from the minimal ISO. Did a yum update and rebooted. Locked again when trying to shut down the network.
Here are the results of running the getinfo script. I don't see wrong...

Information for general problems.
[code]
== BEGIN uname -rmi ==
2.6.32-279.2.1.el6.i686 i686 i386
== END uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
elrepo-release-6-4.el6.elrepo.noarch
centos-release-6-3.el6.centos.9.i686
== END rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
CentOS release 6.3 (Final)
== END cat /etc/redhat-release ==

== BEGIN getenforce ==
Enforcing
== END getenforce ==

== BEGIN free -m ==
total used free shared buffers cached
Mem: 1893 174 1719 0 35 76
-/+ buffers/cache: 62 1831
Swap: 4063 0 4063
== END free -m ==

== BEGIN rpm -qa yum\* rpm-\* python | sort ==
python-2.6.6-29.el6_2.2.i686
rpm-libs-4.8.0-27.el6.i686
rpm-python-4.8.0-27.el6.i686
yum-3.2.29-30.el6.centos.noarch
yum-metadata-parser-1.1.2-16.el6.i686
yum-plugin-fastestmirror-1.1.30-14.el6.noarch
yum-presto-0.6.2-1.el6.noarch
== END rpm -qa yum\* rpm-\* python | sort ==

== BEGIN ls /etc/yum.repos.d ==
CentOS-Base.repo
CentOS-Debuginfo.repo
CentOS-Media.repo
CentOS-Vault.repo
elrepo.repo
== END ls /etc/yum.repos.d ==

== BEGIN cat /etc/yum.conf ==
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=16&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release

# This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
# It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d
== END cat /etc/yum.conf ==

== BEGIN yum repolist all ==
Loaded plugins: fastestmirror, presto
Loading mirror speeds from cached hostfile
* base: centos.mirror.lstn.net
* elrepo: mirror.symnds.com
* extras: dallas.tx.mirror.xygenhosting.com
* updates: centos.mirror.netriplex.com
repo id repo name status
C6.0-base CentOS-6.0 - Base disabled
C6.0-centosplus CentOS-6.0 - CentOSPlus disabled
C6.0-contrib CentOS-6.0 - Contrib disabled
C6.0-extras CentOS-6.0 - Extras disabled
C6.0-updates CentOS-6.0 - Updates disabled
C6.1-base CentOS-6.1 - Base disabled
C6.1-centosplus CentOS-6.1 - CentOSPlus disabled
C6.1-contrib CentOS-6.1 - Contrib disabled
C6.1-extras CentOS-6.1 - Extras disabled
C6.1-updates CentOS-6.1 - Updates disabled
C6.2-base CentOS-6.2 - Base disabled
C6.2-centosplus CentOS-6.2 - CentOSPlus disabled
C6.2-contrib CentOS-6.2 - Contrib disabled
C6.2-extras CentOS-6.2 - Extras disabled
C6.2-updates CentOS-6.2 - Updates disabled
base CentOS-6 - Base enabled: 4,776
c6-media CentOS-6 - Media disabled
centosplus CentOS-6 - Plus disabled
contrib CentOS-6 - Contrib disabled
debug CentOS-6 - Debuginfo disabled
elrepo ELRepo.org Community Enterprise Linux Repository enabled: 179
elrepo-extras ELRepo.org Community Enterprise Linux Repository disabled
elrepo-kernel ELRepo.org Community Enterprise Linux Kernel Repo disabled
elrepo-testing ELRepo.org Community Enterprise Linux Testing Rep disabled
extras CentOS-6 - Extras enabled: 4
updates CentOS-6 - Updates enabled: 346
repolist: 5,305
== END yum repolist all ==

== BEGIN egrep 'include|exclude' /etc/yum.repos.d/*.repo ==
== END egrep 'include|exclude' /etc/yum.repos.d/*.repo ==

== BEGIN sed -n -e "/^\[/h; /priority *=/{ G; s/\n/ /; s/ity=/ity = /; p }" /etc/yum.repos.d/*.repo | sort -k3n ==
== END sed -n -e "/^\[/h; /priority *=/{ G; s/\n/ /; s/ity=/ity = /; p }" /etc/yum.repos.d/*.repo | sort -k3n ==

== BEGIN cat /etc/fstab ==

#
# /etc/fstab
# Created by anaconda on Mon Aug 6 15:32:05 2012
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_taurus-lv_root / ext4 defaults 1 1
UUID=6768bdde-5b2b-4627-b77c-f8a1b0da961a /boot ext4 defaults 1 2
/dev/mapper/vg_taurus-lv_swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
== END cat /etc/fstab ==

== BEGIN df -h ==
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_taurus-lv_root
29G 759M 27G 3% /
tmpfs 947M 0 947M 0% /dev/shm
/dev/sda1 485M 65M 395M 15% /boot
== END df -h ==

== BEGIN fdisk -l ==

Disk /dev/sda: 36.4 GB, 36397645824 bytes
255 heads, 63 sectors/track, 4425 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00002f20

Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 4426 35031040 8e Linux LVM

Disk /dev/mapper/vg_taurus-lv_root: 31.6 GB, 31608274944 bytes
255 heads, 63 sectors/track, 3842 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_taurus-lv_swap: 4261 MB, 4261412864 bytes
255 heads, 63 sectors/track, 518 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

== END fdisk -l ==

== BEGIN parted -l ==
./getinfo.sh: line 89: parted: command not found
== END parted -l ==

== BEGIN blkid ==
/dev/sda1: UUID="6768bdde-5b2b-4627-b77c-f8a1b0da961a" TYPE="ext4"
/dev/sda2: UUID="IVBh3X-Fr0I-sGpW-5MUV-uvom-eM62-QJ2PSz" TYPE="LVM2_member"
/dev/mapper/vg_taurus-lv_root: UUID="db3fb3b5-8562-4da2-88f0-1bfaebc1f1e6" TYPE="ext4"
/dev/mapper/vg_taurus-lv_swap: UUID="86050b85-1da5-4be2-a50d-cf219a6e5199" TYPE="swap"
== END blkid ==

== BEGIN cat /proc/mdstat ==
Personalities :
unused devices: <none>
== END cat /proc/mdstat ==

== BEGIN pvs ==
PV VG Fmt Attr PSize PFree
/dev/sda2 vg_taurus lvm2 a-- 33.41g 0
== END pvs ==

== BEGIN vgs ==
VG #PV #LV #SN Attr VSize VFree
vg_taurus 1 2 0 wz--n- 33.41g 0
== END vgs ==

== BEGIN lvs ==
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
lv_root vg_taurus -wi-ao-- 29.44g
lv_swap vg_taurus -wi-ao-- 3.97g
== END lvs ==

== BEGIN rpm -qa kernel\* | sort ==
kernel-2.6.32-279.2.1.el6.i686
kernel-2.6.32-279.el6.i686
kernel-firmware-2.6.32-279.2.1.el6.noarch
== END rpm -qa kernel\* | sort ==

== BEGIN lspci -nn ==
./getinfo.sh: line 89: lspci: command not found
== END lspci -nn ==

== BEGIN lsusb ==
./getinfo.sh: line 89: lsusb: command not found
== END lsusb ==

== BEGIN rpm -qa kmod\* kmdl\* ==
== END rpm -qa kmod\* kmdl\* ==

== BEGIN ifconfig -a ==
eth0 Link encap:Ethernet HWaddr 00:06:5B:F3:2B:E4
inet addr:69.XX.XXX.54 Bcast:69.XX.XXX.255 Mask:255.255.255.0
inet6 addr: fe80::206:5bff:fef3:2be4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1651 errors:0 dropped:0 overruns:0 frame:0
TX packets:646 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:608504 (594.2 KiB) TX bytes:49317 (48.1 KiB)
Interrupt:28

eth1 Link encap:Ethernet HWaddr 00:06:5B:F3:2B:E5
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Interrupt:29

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:880 (880.0 b) TX bytes:880 (880.0 b)

== END ifconfig -a ==

== BEGIN brctl show ==
bridge name bridge id STP enabled interfaces
== END brctl show ==

== BEGIN route -n ==
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
69.XX.XXX.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
0.0.0.0 69.XX.XXX.1 0.0.0.0 UG 0 0 0 eth0
== END route -n ==

== BEGIN cat /etc/resolv.conf ==
# Generated by NetworkManager
search rioplexwireless.com
nameserver 69.XX.XXX.40
nameserver 69.XX.XXX.43
== END cat /etc/resolv.conf ==

== BEGIN grep net /etc/nsswitch.conf ==
#networks: nisplus [NOTFOUND=return] files
#netmasks: nisplus [NOTFOUND=return] files
netmasks: files
networks: files
netgroup: nisplus
== END grep net /etc/nsswitch.conf ==

== BEGIN chkconfig --list | grep -Ei 'network|wpa' ==
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
== END chkconfig --list | grep -Ei 'network|wpa' ==

[/code]

bakcompat
Posts: 29
Joined: 2006/09/08 21:22:44

Re: stack trace error on ethernet during shutdown/reboot

Post by bakcompat » 2012/08/06 21:44:24

Here are the results of [code]lshw -sanitize -short[/code]
I am thinking the driver for the Broadcom BCM5701 nic is the culprit.

[code]H/W path Device Class Description
====================================================
system PowerEdge 2650
/0 bus 0K0710
/0/0 memory 64KiB BIOS
/0/400 processor Intel(R) Xeon(TM) CPU 2.40GHz
/0/400/700 memory 8KiB L1 cache
/0/400/701 memory 512KiB L2 cache
/0/400/0.1 processor Logical CPU
/0/400/0.2 processor Logical CPU
/0/401 processor Intel(R) Xeon(TM) CPU 2.40GHz
/0/401/703 memory 8KiB L1 cache
/0/401/704 memory 512KiB L2 cache
/0/401/6.1 processor Logical CPU
/0/401/6.2 processor Logical CPU
/0/1000 memory 2GiB System Memory
/0/1000/0 memory 512MiB DIMM DDR Synchronous 200 MHz (5.0 ns)
/0/1000/1 memory 512MiB DIMM DDR Synchronous 200 MHz (5.0 ns)
/0/1000/2 memory 512MiB DIMM DDR Synchronous 200 MHz (5.0 ns)
/0/1000/3 memory 512MiB DIMM DDR Synchronous 200 MHz (5.0 ns)
/0/1000/4 memory DIMM DDR Synchronous 200 MHz (5.0 ns) [empty]
/0/1000/5 memory DIMM DDR Synchronous 200 MHz (5.0 ns) [empty]
/0/100 bridge CMIC-WS Host Bridge (GC-LE chipset)
/0/100/4 generic Embedded Remote Access or ERA/O
/0/100/4.1 generic Remote Access Card III
/0/100/4.2 generic Embedded Remote Access: BMC/SMIC device
/0/100/e display Rage XL
/0/100/f.1 scsi0 storage CSB5 IDE Controller
/0/100/f.1/0.0.0 /dev/cdrom disk SCSI CD-ROM
/0/100/f.2 bus OSB4/CSB5 OHCI USB Controller
/0/100/f.2/1 usb1 bus OHCI Host Controller
/0/100/f.3 bridge CSB5 LPC bridge
/0/101 bridge CMIC-WS Host Bridge (GC-LE chipset)
/0/102 bridge CMIC-LE
/0/103 bridge CSB5 South Bridge
/0/104 bridge CIOB-X2 PCI-X I/O Bridge
/0/105 bridge CIOB-X2 PCI-X I/O Bridge
/0/106 bridge CIOB-X2 PCI-X I/O Bridge
/0/107 bridge CIOB-X2 PCI-X I/O Bridge
/0/6 eth0 network NetXtreme BCM5701 Gigabit Ethernet
/0/1 eth1 network NetXtreme BCM5701 Gigabit Ethernet
/0/108 bridge 80303 I/O Processor PCI-to-PCI Bridge
/0/8.1 scsi2 storage PowerEdge Expandable RAID Controller 3/Di
/0/8.1/0.0.0 /dev/sda disk 36GB SCSI Disk
/0/8.1/0.0.0/1 /dev/sda1 volume 500MiB EXT4 volume
/0/8.1/0.0.0/2 /dev/sda2 volume 33GiB Linux LVM Physical Volume partition
/0/8.1/1.0.0 disk 19GB MAM3184MC
/0/8.1/1.1.0 disk 19GB MAM3184MC
/0/8.1/1.2.0 disk SCSI Disk
/0/8.1/1.6.0 processor 1x5 SCSI BP
[/code]

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: stack trace error on ethernet during shutdown/reboot

Post by pschaff » 2012/08/06 21:55:55

[quote]
bakcompat wrote:
Here are the results of [code]lshw -sanitize -short[/code]
I am thinking the driver for the Broadcom BCM5701 nic is the culprit.
[/quote]
That may well be, but as you have not got [b]lspci[/b] (or [b]lsusb[/b]) installed the desired "[b]lspci -nn[/b]" results are missing from the script output.

bakcompat
Posts: 29
Joined: 2006/09/08 21:22:44

Re: stack trace error on ethernet during shutdown/reboot

Post by bakcompat » 2012/08/10 17:34:15

didn't notice that. Here is the corrected info.

[code]

. . .

== BEGIN lspci -nn ==
00:00.0 Host bridge [0600]: Broadcom CMIC-WS Host Bridge (GC-LE chipset) [1166:0012] (rev 13)
00:00.1 Host bridge [0600]: Broadcom CMIC-WS Host Bridge (GC-LE chipset) [1166:0012]
00:00.2 Host bridge [0600]: Broadcom CMIC-LE [1166:0000]
00:04.0 Unassigned class [ff00]: Dell Embedded Remote Access or ERA/O [1028:000c]
00:04.1 Unassigned class [ff00]: Dell Remote Access Card III [1028:0008]
00:04.2 Unassigned class [ff00]: Dell Embedded Remote Access: BMC/SMIC device [1028:000d]
00:0e.0 VGA compatible controller [0300]: Advanced Micro Devices [AMD] nee ATI Rage XL [1002:4752] (rev 27)
00:0f.0 Host bridge [0600]: Broadcom CSB5 South Bridge [1166:0201] (rev 93)
00:0f.1 IDE interface [0101]: Broadcom CSB5 IDE Controller [1166:0212] (rev 93)
00:0f.2 USB controller [0c03]: Broadcom OSB4/CSB5 OHCI USB Controller [1166:0220] (rev 05)
00:0f.3 ISA bridge [0601]: Broadcom CSB5 LPC bridge [1166:0225]
00:10.0 Host bridge [0600]: Broadcom CIOB-X2 PCI-X I/O Bridge [1166:0101] (rev 03)
00:10.2 Host bridge [0600]: Broadcom CIOB-X2 PCI-X I/O Bridge [1166:0101] (rev 03)
00:11.0 Host bridge [0600]: Broadcom CIOB-X2 PCI-X I/O Bridge [1166:0101] (rev 03)
00:11.2 Host bridge [0600]: Broadcom CIOB-X2 PCI-X I/O Bridge [1166:0101] (rev 03)
03:06.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet [14e4:1645] (rev 15)
03:08.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet [14e4:1645] (rev 15)
04:08.0 PCI bridge [0604]: Intel Corporation 80303 I/O Processor PCI-to-PCI Bridge [8086:0309] (rev 01)
04:08.1 RAID bus controller [0104]: Dell PowerEdge Expandable RAID Controller 3/Di [1028:000a] (rev 01)
== END lspci -nn ==

== BEGIN lsusb ==
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
== END lsusb ==

. . .

[/code]

[Moderator edited to delete the highly redundant information.]

bakcompat
Posts: 29
Joined: 2006/09/08 21:22:44

Re: stack trace error on ethernet during shutdown/reboot

Post by bakcompat » 2012/08/17 19:20:05

any thoughts on what I can do here? I'd really like to get C6.3 working.

User avatar
AlanBartlett
Forum Moderator
Posts: 9345
Joined: 2007/10/22 11:30:09
Location: ~/Earth/UK/England/Suffolk
Contact:

Re: stack trace error on ethernet during shutdown/reboot

Post by AlanBartlett » 2012/08/17 21:10:39

There are two steps I would now like you to try --

(1) Install the [url=http://elrepo.org/tiki/kmod-tg3]kmod-tg3[/url] package which is available from the [url=http://elrepo.org]ELRepo Project[/url]. That will provide you with an updated driver for the two [i]Broadcom[/i] NICs.

(2) Update the kernel to the latest distro release, [b]kernel-2.6.32-279.5.1.el6[/b].

bakcompat
Posts: 29
Joined: 2006/09/08 21:22:44

Re: stack trace error on ethernet during shutdown/reboot

Post by bakcompat » 2012/08/20 22:17:19

Thank you Alan. The kernel update was a rather minor one, but an update nonetheless. I updated it and loaded the kmod-tg3 package as suggested. Sure enough, that server now shuts down properly.

I read in the C5.8 release notes:
[code] There is a known issue with the tg3 kernel driver using VLANs (802.1q) and at least one Broadcom chipset (Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 21)). See CentOS Bug #0005572 . The current workaround (if you use VLAN interfaces) is to install the kmod-tg3-rhel5u7
package from CentOS updates repository. [/code]

I don't know if this bug caused my issue as well as I am not using VLANs. At any rate, it is resolved. Thanks for the recommendation.

User avatar
AlanBartlett
Forum Moderator
Posts: 9345
Joined: 2007/10/22 11:30:09
Location: ~/Earth/UK/England/Suffolk
Contact:

Re: [RESOLVED] stack trace error on ethernet during shutdown/reboot

Post by AlanBartlett » 2012/08/20 23:24:23

Thank you for reporting back. I am pleased to read that your problem is now resolved.

I had a "niggle" at the back of my mind that there has been a recent issue with the distro supplied [i]Broadcom[/i] [b]Tigon3[/b] ([b]tg3[/b]) driver and certain hardware chipsets.

On your behalf and for posterity, this thread is now marked [RESOLVED].

Post Reply