NetXtreme BCM5761 slow

sreilly · Post by **sreilly** » 2011/03/28 11:32:03

Hi,

I have a number of CentOS 5.5 machines on my office network, all fully patched.

I was testing the network performance of each machine using iperf when I noticed that three Dell machines all appear to have less than optimal network performance.
And that these all have NetXtreme BCM5761 Gigabit Ethernet PCIe cards and T5500 motherboards.

Over the LAN other machines machines average 110MBps but the three BCM5761 machines only give 20MBps max despite ethtools showing that;

Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Current message level: 0x000000ff (255)
Link detected: yes

I have tried using the tg3 Linux drivers from Broadcoms website (tg3-3.116j) but it made little difference.

Any ideas would be greatly appreciated.

TIA,

Steve

pschaff · Post by **pschaff** » 2011/03/28 14:13:28

Broadcom seems to attract a lot of negative comments. We might be better able to advise if you would provide the [i]Vendor:Device ID Pairing[/i] (the device [i]fingerprint[/i]).

Either run "./getrinfo.sh [b]network[/b]" as explained in [url=http://www.centos.org/modules/newbb/viewtopic.php?topic_id=25128&forum=47]How to provide information about your system[/url] or as a shorter alternative you could post the output returned by executing
[code]
[b]for BUSID in $(/sbin/lspci | awk '{ IGNORECASE=1 } /net/ { print $1 }'); do /sbin/lspci -s $BUSID -m; /sbin/lspci -s $BUSID -n; done[/b]
[/code]
Use of copy/paste is recommended to avoid typos.

sreilly · Post by **sreilly** » 2011/03/28 14:18:24

Phil,

Thanks for the reply.

Here is the info gleaned from that command;

06:00.0 "Ethernet controller" "Broadcom Corporation" "NetXtreme BCM5761 Gigabit Ethernet PCIe" -r10 "Dell" "Unknown device 026e"
06:00.0 0200: 14e4:1681 (rev 10)

Regards,

Steve

sreilly · Post by **sreilly** » 2011/03/28 14:34:32

From the getinfo.sh script;

Information for network problems.
[code]
== BEGIN uname -rmi ==
2.6.18-194.32.1.el5 x86_64 x86_64
== END uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
centos-release-notes-5.5-0
centos-release-5-5.el5.centos
== END rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
Red Hat Enterprise Linux Client release 5.5 (Tikanga)
== END cat /etc/redhat-release ==

== BEGIN getenforce ==
Disabled
== END getenforce ==

== BEGIN lspci ==
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 RAID bus controller: Intel Corporation 82801 SATA RAID Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 PCI bridge: Pericom Semiconductor PCI Express to PCI-XPI7C9X130 PCI-X Bridge (rev 04)
03:00.0 VGA compatible controller: nVidia Corporation Unknown device 06fd (rev a1)
04:00.0 VGA compatible controller: nVidia Corporation Unknown device 06fd (rev a1)
06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)
20:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
20:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
20:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
20:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
20:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
== END lspci ==

== BEGIN lspci -n ==
00:00.0 0600: 8086:3406 (rev 13)
00:01.0 0604: 8086:3408 (rev 13)
00:03.0 0604: 8086:340a (rev 13)
00:07.0 0604: 8086:340e (rev 13)
00:14.0 0800: 8086:342e (rev 13)
00:14.1 0800: 8086:3422 (rev 13)
00:14.2 0800: 8086:3423 (rev 13)
00:1a.0 0c03: 8086:3a37
00:1a.1 0c03: 8086:3a38
00:1a.2 0c03: 8086:3a39
00:1a.7 0c03: 8086:3a3c
00:1b.0 0403: 8086:3a3e
00:1c.0 0604: 8086:3a40
00:1c.5 0604: 8086:3a4a
00:1d.0 0c03: 8086:3a34
00:1d.1 0c03: 8086:3a35
00:1d.2 0c03: 8086:3a36
00:1d.7 0c03: 8086:3a3a
00:1e.0 0604: 8086:244e (rev 90)
00:1f.0 0601: 8086:3a16
00:1f.2 0104: 8086:2822
00:1f.3 0c05: 8086:3a30
01:00.0 0604: 12d8:e130 (rev 04)
03:00.0 0300: 10de:06fd (rev a1)
04:00.0 0300: 10de:06fd (rev a1)
06:00.0 0200: 14e4:1681 (rev 10)
20:07.0 0604: 8086:340e (rev 13)
20:09.0 0604: 8086:3410 (rev 13)
20:14.0 0800: 8086:342e (rev 13)
20:14.1 0800: 8086:3422 (rev 13)
20:14.2 0800: 8086:3423 (rev 13)
== END lspci -n ==

== BEGIN ifconfig -a ==
eth0 Link encap:Ethernet HWaddr 00:25:64:9C:90:F2
inet addr:xxx.xxx.xxx.xxx Bcast:xxx.xxx.xxx.255 Mask:255.255.255.0
inet6 addr: fe80::225:64ff:fe9c:90f2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6888807 errors:0 dropped:1620 overruns:0 frame:4953
TX packets:9053821 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3760271678 (3.5 GiB) TX bytes:10401785113 (9.6 GiB)
Interrupt:185

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2444 errors:0 dropped:0 overruns:0 frame:0
TX packets:2444 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:199572 (194.8 KiB) TX bytes:199572 (194.8 KiB)

sit0 Link encap:IPv6-in-IPv4
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

== END ifconfig -a ==

== BEGIN route -n ==
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
xxx.xxx.xxx.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
xxx.xxx.xxx.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 xxx.xxx.xxx.1 0.0.0.0 UG 0 0 0 eth0
== END route -n ==

== BEGIN cat /etc/resolv.conf ==
nameserver xxx.xxx.xxx.5
nameserver xxx.xxx.xxx.253
domain xxx.xxx.xxx
search xxx.xxx.xxx xxx.xxx
options timeout:1 rotate
== END cat /etc/resolv.conf ==

== BEGIN grep net /etc/nsswitch.conf ==
netmasks: files
networks: files
netgroup: files
== END grep net /etc/nsswitch.conf ==

== BEGIN chkconfig --list | grep -Ei 'network|wpa' ==
NetworkManager 0:off 1:off 2:off 3:off 4:off 5:off 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
wpa_supplicant 0:off 1:off 2:off 3:off 4:off 5:off 6:off
== END chkconfig --list | grep -Ei 'network|wpa' ==

[/code]

Thanks,

Steve

pschaff · Post by **pschaff** » 2011/03/28 14:55:24

Neither a [url=https://www.centos.org/search.php?query=14e4%3A1681&andor=AND&action=results&submit=Search]forum search[/url] nor a [url=http://www.google.com/search?q=14e4%3A1681+site%3Acentos.org&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a#q=14e4:1681+site:centos.org&hl=en&safe=active&client=firefox-a&hs=TtY&rls=org.mozilla:en-US:official&prmd=ivns&ei=F5qQTevSB_O90QGyibmmCw&start=10&sa=N&bav=on.2,or.r_gc.r_pw.&fp=888970748f66ece1]site search[/url] turn up much that looks interesting. Perhaps someone with that hardware will comment, now that it has been definitively identified.

AlanBartlett · Post by **AlanBartlett** » 2011/03/28 22:05:16

I'm posting the following information not because I think it will help but, as it relates to [i]RHEL 6.0[/i], wondering if it shows a symptom of the implementation of that hardware by [i]Dell[/i] ?

[code]
[ajb@Duo2 ~]$ uname -a
Linux Duo2 2.6.32-71.18.2.el6.x86_64 #1 SMP Wed Mar 2 14:17:40 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
[ajb@Duo2 ~]$ lspci -m | grep -i ether
09:00.0 "Ethernet controller" "Broadcom Corporation" "NetXtreme BCM5761e Gigabit Ethernet PCIe" -r10 "Dell" "Device 0263"
[ajb@Duo2 ~]$ lspci -n | grep ^09
09:00.0 0200: 14e4:1680 (rev 10)
[ajb@Duo2 ~]$ grep -i 14e4 /lib/modules/*/modules.alias | grep 1680
/lib/modules/2.6.32-71.18.2.el6.x86_64/modules.alias:alias pci:v000014E4d00001680sv*sd*bc*sc*i* tg3
/lib/modules/2.6.32-71.el6.x86_64/modules.alias:alias pci:v000014E4d00001680sv*sd*bc*sc*i* tg3
[ajb@Duo2 ~]$ modinfo -F version tg3
3.108
[ajb@Duo2 ~]$ sudo ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 100Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: g
Current message level: 0x000000ff (255)
Link detected: yes
[/code]
:-?

sreilly · Post by **sreilly** » 2011/04/01 12:51:11

I'm starting to form the opinion that it was rash of me to accuse the BCM5761.

I have tried a different network card (Realtek rtl8169sc chipset) and got slightly worse performance.

Could this be an issue with the m/b - T5500 ?

(BTW I have confirmed that the network cabling is sound.)

Steve

pschaff · Post by **pschaff** » 2011/04/01 13:37:37

Could be the MB IRQ handling or IO bandwidth, could possibly also be a switch or other external gear.

Post by **TrevorH** » 2011/04/01 14:58:37

[quote]
Over the LAN other machines machines average 110MBps but the three BCM5761 machines only give 20MBps max
[/quote]

Just to clarify, since a lot of people use the wrong abbreviations for these figures, are those megabits/s or megabytes/s?

You've tried different network ports on the switch? Eliminated duplicate IP addresses on the network as a possible cause? Duplicate MAC addresses?

sreilly · Post by **sreilly** » 2011/04/04 12:14:22

Hi TrevorH,

Unfortunately, the network is at a remote (12000 miles) site. I believe the switch ports have been changed as have intervening cables.

From my inventory monitoring tools there doesn't appear to be duplicate IPs or MAC on the three systems exhibiting this problem.

The iperf numbers are as stated. I start the server with "iperf -f M -s" and the clients with "iperf -f M -c "
Today the numbers are;
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 173 MBytes 17.3 MBytes/sec

Steve

CentOS