KVM Bond & Bridge Does not work, But source bridge with bond does?

Issues related to configuring your network
Schorschi
Posts: 14
Joined: 2011/12/13 22:54:04

KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby Schorschi » 2017/11/18 03:12:10

A KVM Bond & Bridge Does not work, But VM with Source Bridge Mode with Bond does?

Found the following...

A Bond, with 2 NICs. Assignment of this works as expected with VM.
A Bond assigned to a Bridge, then assigned to VM does not work as expected.
Bridge and Bond created via virt-manager, NetworkManager is disabled.

Is this a know issue? I can provide whatever detail needed, just need to know what anyone wants to see? The Bond, does work is defined as follows...

# cat ifcfg-bond0
DEVICE="bond0"
ONBOOT="yes"
BONDING_OPTS=

# cat ifcfg-enp9s0f0
DEVICE="enp9s0f0"
ONBOOT="yes"
MASTER="bond0"
SLAVE="yes"

# cat ifcfg-enp9s0f1
DEVICE="enp9s0f1"
ONBOOT="yes"
MASTER="bond0"
SLAVE="yes"

VMs are configured to use source mode BRIDGE, and mapped to Bond0.

* Note... MAC addresses and IP addresses redacted *
bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
inet6 0000::000:0000:0000:0000 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 548 bytes 37506 (36.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3 bytes 242 (242.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp13s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 0.0.0.0 netmask 255.255.255.0 broadcast 0.0.0.255
inet6 0000::000:0000:0000:0000 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 1199 bytes 179059 (174.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 483 bytes 69100 (67.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp9s0f0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 274 bytes 18753 (18.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 74 (74.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 56 memory 0xfe3a0000-fe3c0000

enp9s0f1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 274 bytes 18753 (18.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2 bytes 168 (168.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 67 memory 0xfe340000-fe360000

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

If I add a Bridge above the Bond, then networking does not work...

VMs are configured to use bridge br0. VM boots, but NIC never gets reply from DHCP server, appears DHCP request is never sent? iPXE reports NIC is link up state, but no configuration methods succeeded.

# cat ifcfg-br0
DEVICE="br0"
ONBOOT="yes"
TYPE="Bridge"
STP="on"
DELAY="0.0"

# cat ifcfg-bond0
DEVICE="bond0"
ONBOOT="yes"
BONDING_OPTS=
BRIDGE="br0"

# cat ifcfg-enp9s0f1
DEVICE="enp9s0f1"
ONBOOT="yes"
MASTER="bond0"
SLAVE="yes"

# cat ifcfg-enp9s0f0
DEVICE="enp9s0f0"
ONBOOT="yes"
MASTER="bond0"
SLAVE="yes"

* Note... MAC addresses and IP addresses redacted *
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 0000::000:0000:0000:0000 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 7 bytes 400 (400.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3 bytes 258 (258.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
inet6 0000::000:0000:0000:0000 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 548 bytes 37506 (36.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3 bytes 242 (242.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp13s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 0.0.0.0 netmask 255.255.255.0 broadcast 0.0.0.255
inet6 0000::000:0000:0000:0000 prefixlen 64 scopeid 0x20<link>
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 1199 bytes 179059 (174.8 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 483 bytes 69100 (67.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp9s0f0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 274 bytes 18753 (18.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1 bytes 74 (74.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 56 memory 0xfe3a0000-fe3c0000

enp9s0f1: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500
ether 00:00:00:00:00:00 txqueuelen 1000 (Ethernet)
RX packets 274 bytes 18753 (18.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2 bytes 168 (168.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 67 memory 0xfe340000-fe360000

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Any help appreciated, thanks.

User avatar
TrevorH
Forum Moderator
Posts: 21206
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby TrevorH » 2017/11/18 10:06:41

What BONDING_OPTS= do you have specified?

You don't appear to have an ipv4 address assigned to your bridge (or your bond come to that).
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

User avatar
jlehtone
Posts: 1880
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby jlehtone » 2017/11/18 13:09:39

Schorschi wrote:A KVM Bond & Bridge Does not work?

Negative. Works for me.

Well, I have some extras on that, but they should not make things any easier.
1. On CentOS 7 I do use NetworkManager and no network.service at all.
2. On CentOS 6 I do use service network and no NetworkManager at all.
3. I have VLAN interface(s) between bond and bridge(s).

That said, I do have (LACP) bonds and persistent bridges on my hosts and VM's are happy with them.

The first thing I would test is to disable the STP.
IIRC, that is the default on libvirt-created bridges too.
The STP is a protection against subnet loops. You won't create loops, do you?

sidusnare
Posts: 10
Joined: 2017/11/15 18:19:49

Re: KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby sidusnare » 2017/11/18 13:50:32

When I first set this up, I was surprised that the IP had to be on the bridge interface, something to be aware of. I also make sure to have network.service setup the bonds, not let libvirtd do it.

Schorschi
Posts: 14
Joined: 2011/12/13 22:54:04

Re: KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby Schorschi » 2017/11/20 20:39:00

Ah! I am explicitly trying to NOT use an assigned IP address, because that should not be required for a bridged bond. I will allow the IP assignment and see if that works.

User avatar
jlehtone
Posts: 1880
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby jlehtone » 2017/11/21 13:37:03

IP address should not be necessary.

This is what I have on a CentOS 6 machine that has no NetworkManager packages:

Code: Select all

# cat ifcfg-eth2
DEVICE="eth2"
HWADDR="****"
ONBOOT="yes"
TYPE="Ethernet"
BOOTPROTO="none"
SLAVE="yes"
MASTER="bond0"

# cat ifcfg-bond0
DEVICE="bond0"
TYPE="Bond"
ONBOOT="yes"
BONDING_OPTS="mode=4 miimon=100"

# cat ifcfg-bond0.666
DEVICE="bond0.666"
BOOTPROTO="none"
ONBOOT="yes"
VLAN="yes"
BRIDGE="br666"

# cat ifcfg-br666
DEVICE="br666"
TYPE="Bridge"
BOOTPROTO="static"
ONBOOT="yes"
IPADDR=10.2.3.4
PREFIX=24
STP="off"
DELAY=0

On another (CentOS 7, NM-based config, there is no IP address).

Logically, this should do:

Code: Select all

# cat ifcfg-eth2
DEVICE="eth2"
HWADDR="****"
ONBOOT="yes"
TYPE="Ethernet"
BOOTPROTO="none"
SLAVE="yes"
MASTER="bond0"

# cat ifcfg-bond0
DEVICE="bond0"
TYPE="Bond"
ONBOOT="yes"
BOOTPROTO="none"
BONDING_OPTS="mode=4 miimon=100"
BRIDGE="br666"

# cat ifcfg-br666
DEVICE="br666"
TYPE="Bridge"
BOOTPROTO="none"
ONBOOT="yes"
STP="off"
DELAY=0

The BONDING_OPTS has to be set appropriately. You want to set it here and not within modprobe configuration.

VM's have (virsh dumpxml):

Code: Select all

    <interface type='bridge'>
      <mac address='****'/>
      <source bridge='br666'/>
      <model type='virtio'/>
    </interface>

hunter86_bg
Posts: 811
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby hunter86_bg » 2017/11/26 05:43:17

Also in later versions (bug report) bonding with bridge will work properly with dhcp.
Set the IP to the bridge ,check forwarding is on -and all should be good.

User avatar
jlehtone
Posts: 1880
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby jlehtone » 2017/11/27 08:09:02

hunter86_bg wrote:Set the IP to the bridge ,check forwarding is on -and all should be good.

Forwarding?

To forward is to act as a router. Routing is Layer 3 operation. A bridge is a network switch. Switch operates on Layer 2. A switch does not need an IP to operate. Unmanaged switches don't have IP at all. Managed switches (might) need a management IP. If the host has some other connection for management, then it needs no IP on the bridges that it provides for the VM's.

If VM guest accesses outside subnet via bridged connection, then the VM host does not need to route at all. Forwarding can be off.

If one creates a virtual subnet for VMs with libvirt -- a subnet that the host links to external subnet via routing -- the libvirt apparently enables ip_forward for us.

hunter86_bg
Posts: 811
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: KVM Bond & Bridge Does not work, But source bridge with bond does?

Postby hunter86_bg » 2017/11/27 20:32:05

That's true , but I doubt that plain linux bridge can do it as you wish. Most probably you have to use 'open vSwitch '.