NetXtreme BCM5761 slow

Issues related to configuring your network

NetXtreme BCM5761 slow

Postby sreilly » 2011/03/28 11:32:03

Hi,

I have a number of CentOS 5.5 machines on my office network, all fully patched.

I was testing the network performance of each machine using iperf when I noticed that three Dell machines all appear to have less than optimal network performance.
And that these all have NetXtreme BCM5761 Gigabit Ethernet PCIe cards and T5500 motherboards.

Over the LAN other machines machines average 110MBps but the three BCM5761 machines only give 20MBps max despite ethtools showing that;

Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: g
Wake-on: g
Current message level: 0x000000ff (255)
Link detected: yes

I have tried using the tg3 Linux drivers from Broadcoms website (tg3-3.116j) but it made little difference.

Any ideas would be greatly appreciated.

TIA,

Steve
sreilly
 
Posts: 32
Joined: 2007/05/25 08:42:41
Location: Edinburgh, Scotland

NetXtreme BCM5761 slow

Postby pschaff » 2011/03/28 14:13:28

Broadcom seems to attract a lot of negative comments. We might be better able to advise if you would provide the Vendor:Device ID Pairing (the device fingerprint).

Either run "./getrinfo.sh network" as explained in How to provide information about your system or as a shorter alternative you could post the output returned by executing
Code: Select all
[b]for BUSID in $(/sbin/lspci | awk '{ IGNORECASE=1 } /net/ { print $1 }'); do /sbin/lspci -s $BUSID -m; /sbin/lspci -s $BUSID -n; done[/b]

Use of copy/paste is recommended to avoid typos.
pschaff
Retired Moderator
 
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America

Re: NetXtreme BCM5761 slow

Postby sreilly » 2011/03/28 14:18:24

Phil,

Thanks for the reply.

Here is the info gleaned from that command;

06:00.0 "Ethernet controller" "Broadcom Corporation" "NetXtreme BCM5761 Gigabit Ethernet PCIe" -r10 "Dell" "Unknown device 026e"
06:00.0 0200: 14e4:1681 (rev 10)

Regards,

Steve
sreilly
 
Posts: 32
Joined: 2007/05/25 08:42:41
Location: Edinburgh, Scotland

Re: NetXtreme BCM5761 slow

Postby sreilly » 2011/03/28 14:34:32

From the getinfo.sh script;

Information for network problems.
Code: Select all
== BEGIN uname -rmi ==
2.6.18-194.32.1.el5 x86_64 x86_64
== END   uname -rmi ==

== BEGIN rpm -qa \*-release\* ==
centos-release-notes-5.5-0
centos-release-5-5.el5.centos
== END   rpm -qa \*-release\* ==

== BEGIN cat /etc/redhat-release ==
Red Hat Enterprise Linux Client release 5.5 (Tikanga)
== END   cat /etc/redhat-release ==

== BEGIN getenforce ==
Disabled
== END   getenforce ==

== BEGIN lspci ==
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1a.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB Controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 RAID bus controller: Intel Corporation 82801 SATA RAID Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 PCI bridge: Pericom Semiconductor PCI Express to PCI-XPI7C9X130 PCI-X Bridge (rev 04)
03:00.0 VGA compatible controller: nVidia Corporation Unknown device 06fd (rev a1)
04:00.0 VGA compatible controller: nVidia Corporation Unknown device 06fd (rev a1)
06:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5761 Gigabit Ethernet PCIe (rev 10)
20:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
20:09.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
20:14.0 PIC: Intel Corporation 5520/5500/X58 I/O Hub System Management Registers (rev 13)
20:14.1 PIC: Intel Corporation 5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
20:14.2 PIC: Intel Corporation 5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
== END   lspci ==

== BEGIN lspci -n ==
00:00.0 0600: 8086:3406 (rev 13)
00:01.0 0604: 8086:3408 (rev 13)
00:03.0 0604: 8086:340a (rev 13)
00:07.0 0604: 8086:340e (rev 13)
00:14.0 0800: 8086:342e (rev 13)
00:14.1 0800: 8086:3422 (rev 13)
00:14.2 0800: 8086:3423 (rev 13)
00:1a.0 0c03: 8086:3a37
00:1a.1 0c03: 8086:3a38
00:1a.2 0c03: 8086:3a39
00:1a.7 0c03: 8086:3a3c
00:1b.0 0403: 8086:3a3e
00:1c.0 0604: 8086:3a40
00:1c.5 0604: 8086:3a4a
00:1d.0 0c03: 8086:3a34
00:1d.1 0c03: 8086:3a35
00:1d.2 0c03: 8086:3a36
00:1d.7 0c03: 8086:3a3a
00:1e.0 0604: 8086:244e (rev 90)
00:1f.0 0601: 8086:3a16
00:1f.2 0104: 8086:2822
00:1f.3 0c05: 8086:3a30
01:00.0 0604: 12d8:e130 (rev 04)
03:00.0 0300: 10de:06fd (rev a1)
04:00.0 0300: 10de:06fd (rev a1)
06:00.0 0200: 14e4:1681 (rev 10)
20:07.0 0604: 8086:340e (rev 13)
20:09.0 0604: 8086:3410 (rev 13)
20:14.0 0800: 8086:342e (rev 13)
20:14.1 0800: 8086:3422 (rev 13)
20:14.2 0800: 8086:3423 (rev 13)
== END   lspci -n ==

== BEGIN ifconfig -a ==
eth0      Link encap:Ethernet  HWaddr 00:25:64:9C:90:F2 
          inet addr:xxx.xxx.xxx.xxx  Bcast:xxx.xxx.xxx.255  Mask:255.255.255.0
          inet6 addr: fe80::225:64ff:fe9c:90f2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6888807 errors:0 dropped:1620 overruns:0 frame:4953
          TX packets:9053821 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3760271678 (3.5 GiB)  TX bytes:10401785113 (9.6 GiB)
          Interrupt:185

lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:2444 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2444 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:199572 (194.8 KiB)  TX bytes:199572 (194.8 KiB)

sit0      Link encap:IPv6-in-IPv4 
          NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

== END   ifconfig -a ==

== BEGIN route -n ==
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
xxx.xxx.xxx.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
xxx.xxx.xxx.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
0.0.0.0         xxx.xxx.xxx.1     0.0.0.0         UG    0      0        0 eth0
== END   route -n ==

== BEGIN cat /etc/resolv.conf ==
nameserver xxx.xxx.xxx.5
nameserver xxx.xxx.xxx.253
domain xxx.xxx.xxx
search xxx.xxx.xxx xxx.xxx
options timeout:1 rotate
== END   cat /etc/resolv.conf ==

== BEGIN grep net /etc/nsswitch.conf ==
netmasks:   files
networks:   files
netgroup:   files
== END   grep net /etc/nsswitch.conf ==

== BEGIN chkconfig --list | grep -Ei 'network|wpa' ==
NetworkManager    0:off   1:off   2:off   3:off   4:off   5:off   6:off
network           0:off   1:off   2:on   3:on   4:on   5:on   6:off
wpa_supplicant    0:off   1:off   2:off   3:off   4:off   5:off   6:off
== END   chkconfig --list | grep -Ei 'network|wpa' ==



Thanks,

Steve
sreilly
 
Posts: 32
Joined: 2007/05/25 08:42:41
Location: Edinburgh, Scotland

Re: NetXtreme BCM5761 slow

Postby pschaff » 2011/03/28 14:55:24

Neither a forum search nor a site search turn up much that looks interesting. Perhaps someone with that hardware will comment, now that it has been definitively identified.
pschaff
Retired Moderator
 
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America

Re: NetXtreme BCM5761 slow

Postby AlanBartlett » 2011/03/28 22:05:16

I'm posting the following information not because I think it will help but, as it relates to RHEL 6.0, wondering if it shows a symptom of the implementation of that hardware by Dell ?

Code: Select all
[ajb@Duo2 ~]$ uname -a
Linux Duo2 2.6.32-71.18.2.el6.x86_64 #1 SMP Wed Mar 2 14:17:40 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
[ajb@Duo2 ~]$ lspci -m | grep -i ether
09:00.0 "Ethernet controller" "Broadcom Corporation" "NetXtreme BCM5761e Gigabit Ethernet PCIe" -r10 "Dell" "Device 0263"
[ajb@Duo2 ~]$ lspci -n | grep ^09
09:00.0 0200: 14e4:1680 (rev 10)
[ajb@Duo2 ~]$ grep -i 14e4 /lib/modules/*/modules.alias | grep 1680
/lib/modules/2.6.32-71.18.2.el6.x86_64/modules.alias:alias pci:v000014E4d00001680sv*sd*bc*sc*i* tg3
/lib/modules/2.6.32-71.el6.x86_64/modules.alias:alias pci:v000014E4d00001680sv*sd*bc*sc*i* tg3
[ajb@Duo2 ~]$ modinfo -F version tg3
3.108
[ajb@Duo2 ~]$ sudo ethtool eth0
Settings for eth0:
   Supported ports: [ TP ]
   Supported link modes:   10baseT/Half 10baseT/Full
                           100baseT/Half 100baseT/Full
                           1000baseT/Half 1000baseT/Full
   Supports auto-negotiation: Yes
   Advertised link modes:  10baseT/Half 10baseT/Full
                           100baseT/Half 100baseT/Full
                           1000baseT/Half 1000baseT/Full
   Advertised pause frame use: No
   Advertised auto-negotiation: Yes
   Speed: 100Mb/s
   Duplex: Full
   Port: Twisted Pair
   PHYAD: 1
   Transceiver: internal
   Auto-negotiation: on
   MDI-X: Unknown
   Supports Wake-on: g
   Wake-on: g
   Current message level: 0x000000ff (255)
   Link detected: yes

:-?
User avatar
AlanBartlett
Forum Moderator
 
Posts: 9219
Joined: 2007/10/22 11:30:09
Location: ~/Earth/UK/England/Suffolk

Re: NetXtreme BCM5761 slow - or is it?

Postby sreilly » 2011/04/01 12:51:11

I'm starting to form the opinion that it was rash of me to accuse the BCM5761.

I have tried a different network card (Realtek rtl8169sc chipset) and got slightly worse performance.

Could this be an issue with the m/b - T5500 ?

(BTW I have confirmed that the network cabling is sound.)

Steve
sreilly
 
Posts: 32
Joined: 2007/05/25 08:42:41
Location: Edinburgh, Scotland

Re: NetXtreme BCM5761 slow - or is it?

Postby pschaff » 2011/04/01 13:37:37

Could be the MB IRQ handling or IO bandwidth, could possibly also be a switch or other external gear.
pschaff
Retired Moderator
 
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America

Re: NetXtreme BCM5761 slow - or is it?

Postby TrevorH » 2011/04/01 14:58:37

Over the LAN other machines machines average 110MBps but the three BCM5761 machines only give 20MBps max


Just to clarify, since a lot of people use the wrong abbreviations for these figures, are those megabits/s or megabytes/s?

You've tried different network ports on the switch? Eliminated duplicate IP addresses on the network as a possible cause? Duplicate MAC addresses?
User avatar
TrevorH
Forum Moderator
 
Posts: 13325
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: NetXtreme BCM5761 slow - or is it?

Postby sreilly » 2011/04/04 12:14:22

Hi TrevorH,

Unfortunately, the network is at a remote (12000 miles) site. I believe the switch ports have been changed as have intervening cables.

From my inventory monitoring tools there doesn't appear to be duplicate IPs or MAC on the three systems exhibiting this problem.

The iperf numbers are as stated. I start the server with "iperf -f M -s" and the clients with "iperf -f M -c "
Today the numbers are;
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 173 MBytes 17.3 MBytes/sec

Steve
sreilly
 
Posts: 32
Joined: 2007/05/25 08:42:41
Location: Edinburgh, Scotland

Next

Return to CentOS 5 - Networking Support

Who is online

Users browsing this forum: No registered users and 0 guests