802.3ad channel bonding in CentOS

Issues related to configuring your network
Post Reply
ZippySLC
Posts: 1
Joined: 2007/10/12 21:25:16

802.3ad channel bonding in CentOS

Post by ZippySLC » 2007/10/12 21:28:19

I have a Dell Poweredge 2900 with three Broadcom NetXtreme II nics. I'm running CentOS 4.4_ServerCD with kernel 2.6.9-55.0.9.plus.c4smp. The three nics are plugged into a Dell (yeah, I know) PowerConnect 5324 switch.

I have configured Linux bonding as per the [url=http://wiki.centos.org/TipsAndTricks/BondingInterfaces]CentOS Wiki[/url], as well as [url=http://linux-net.osdl.org/index.php/Bonding#MII_Monitor_Operation]this site[/url] which is a Wikified version of the kernel doc plus some extra information.

According to Dell support, my switch is configured for LAG correctly.

My problem is, when I bring up the bond0 network interface (ifup bond0) it comes up but brings up both slave interfaces (eth1 and eth2) and claims their links are down. If I do a ifdown eth1 and ifdown eth2 and then ifup both of them, they both come up and show their links as up, and bonding works as expected.

The question is, why do these cards show no link when brought up as part of the bond, but show a link when they are brought up separately?

Here's my configs. Logs are tailed from /var/log/messages.

[code]
[root@hudson /]# cat /etc/modprobe.conf
alias scsi_hostadapter megaraid_sas
alias usb-controller ehci-hcd
alias usb-controller1 uhci-hcd
alias eth0 bnx2
alias eth1 bnx2
alias eth2 bnx2

alias bond0 bonding
options bond0 mode=802.3ad miimon=100

[root@hudson /]# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=192.168.50.10
NETWORK=192.168.50.0
NETMASK=255.255.255.0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes

[root@hudson /]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none

[root@hudson /]# cat /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE=eth2
USERCTL=no
ONBOOT=yes
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
[/code]

and here's the logs when I bring up the bond by itself:

[code]
Oct 12 17:14:56 hudson kernel: ADDRCONF(NETDEV_UP): bond0: link is not ready
Oct 12 17:15:00 hudson kernel: bnx2: eth1: using MSI
Oct 12 17:15:00 hudson kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
Oct 12 17:15:00 hudson kernel: bonding: bond0: enslaving eth1 as a backup interface with a down link.
Oct 12 17:15:00 hudson kernel: bonding: Warning: No 802.3ad response from the link partner for any adapters in the bond
Oct 12 17:15:00 hudson kernel: bonding: Warning: No 802.3ad response from the link partner for any adapters in the bond
Oct 12 17:15:00 hudson kernel: bnx2: eth2: using MSI
Oct 12 17:15:01 hudson kernel: ADDRCONF(NETDEV_UP): eth2: link is not ready
Oct 12 17:15:01 hudson kernel: bonding: bond0: enslaving eth2 as a backup interface with a down link.
[/code]

and I get:

[code]
[root@hudson /]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 19
Number of ports: 1
Actor Key: 0
Partner Key: 1
Partner Mac Address: 00:00:00:00:00:00

Slave Interface: eth1
MII Status: down
Link Failure Count: 0
Permanent HW addr: 00:19:b9:dc:32:90
Aggregator ID: 19

Slave Interface: eth2
MII Status: down
Link Failure Count: 0
Permanent HW addr: 00:10:18:28:70:e6
[/code]


but then I do a:

[code]
[root@hudson /]# ifdown eth1
[root@hudson /]# ifdown eth2
[/code]

and then I do a:

[code]
[root@hudson /]# ifup eth1
[/code]

and get:

[code]
Oct 12 17:16:07 hudson kernel: bnx2: eth1: using MSI
Oct 12 17:16:07 hudson kernel: ADDRCONF(NETDEV_UP): eth1: link is not ready
Oct 12 17:16:07 hudson kernel: bonding: bond0: enslaving eth1 as a backup interface with a down link.
Oct 12 17:16:07 hudson kernel: bonding: Warning: No 802.3ad response from the link partner for any adapters in the bond
Oct 12 17:16:07 hudson kernel: bonding: Warning: No 802.3ad response from the link partner for any adapters in the bond
Oct 12 17:16:10 hudson kernel: bnx2: eth1 NIC Link is Up, 1000 Mbps full duplex
Oct 12 17:16:10 hudson kernel: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Oct 12 17:16:10 hudson kernel: bonding: bond0: link status definitely up for interface eth1.
Oct 12 17:16:11 hudson kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[/code]

and a

[code]
[root@hudson /]# ifup eth2
[/code]

and get:

[code]
Oct 12 17:17:11 hudson kernel: bnx2: eth2: using MSI
Oct 12 17:17:11 hudson kernel: ADDRCONF(NETDEV_UP): eth2: link is not ready
Oct 12 17:17:11 hudson kernel: bonding: bond0: enslaving eth2 as a backup interface with a down link.
Oct 12 17:17:14 hudson kernel: bnx2: eth2 NIC Link is Up, 1000 Mbps full duplex
Oct 12 17:17:14 hudson kernel: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
Oct 12 17:17:14 hudson kernel: bonding: bond0: link status definitely up for interface eth2.
[/code]

and the end result is:

[code]
[root@hudson /]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v2.6.3-rh (June 8, 2005)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Active Aggregator Info:
Aggregator ID: 17
Number of ports: 2
Actor Key: 17
Partner Key: 25
Partner Mac Address: 00:1c:23:63:ce:97

Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:19:b9:dc:32:90
Aggregator ID: 17

Slave Interface: eth2
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:10:18:28:70:e6
Aggregator ID: 17
[/code]

If anybody can help me out, I'd greatly appreciate it!

Post Reply

Return to “CentOS 4 - Networking Support”