2 x Interfaces - Bond - VLAN - Bridge on CentOS 7

[Ve0]

Hello!
I have simple config with bond interface (2 x eth - bond - bridge). But I want to put vlans between bond and bridge. I tried this config:
ifcfg-enp0s3

Code: Select all

NM_CONTROLLED=no
BOOTPROTO=none
DEVICE=enp0s3
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes
NAME=enp0s3

ifcfg-enp0s8

Code: Select all

NM_CONTROLLED=no
BOOTPROTO=none
DEVICE=enp0s8
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes
NAME=enp0s8

ifcfg-bond0

Code: Select all

DEVICE=bond0
NM_CONTROLLED=no
USERCTL=no
BOOTPROTO=none
BONDING_OPTS="mode=1 miimon=100"
TYPE=Ethernet
MTU=9000
NOZEROCONF=yes
IPADDR=10.0.1.111
PREFIX=24

ifcfg-bond0.1

Code: Select all

DEVICE=bond0.1
PHYSDEV=bond0
TYPE=Vlan
VLAN_ID=1
VLAN=yes
BOOTPROTO=none
NM_CONTROLLED=no
MTU=1500
ONBOOT=yes
USERCTL=no
ONPARENT=yes
BRIDGE=br0

ifcfg-bond0.2

Code: Select all

DEVICE=bond0.2
PHYSDEV=bond0
TYPE=Vlan
VLAN_ID=2
VLAN=yes
BOOTPROTO=none
NM_CONTROLLED=no
MTU=1500
ONBOOT=yes
USERCTL=no
ONPARENT=yes
BRIDGE=ovirtmgmt

ifcfg-br0

Code: Select all

TYPE=Bridge
NM_CONTROLLED=no
BOOTPROTO=none
DEVICE=br0
SLAVE=bond0.1
ONBOOT=yes
IPADDR=10.0.1.150
PREFIX=24
DEFROUTE=no
MTU=1500
NOZEROCONF=yes

ifcfg-ovirtmgmt

Code: Select all

TYPE=Bridge
NM_CONTROLLED=no
BOOTPROTO=none
DEVICE=ovirtmgmt
SLAVE=bond0.2
ONBOOT=yes
IPADDR=10.0.1.151
PREFIX=24
PEERDNS=no
GATEWAY=10.0.1.1
DEFROUTE=yes
MTU=9000
NOZEROCONF=yes

This config works just only if I put ip addr to bond0, without it doesn't work. I have no vlans in my network. Please, help! Where is the mistake?

[jlehtone]

Code: Select all

IPADDR=10.0.1.111
PREFIX=24

IPADDR=10.0.1.150
PREFIX=24

IPADDR=10.0.1.151
PREFIX=24

I have no vlans in my network. Please, help! Where is the mistake?

Do I read you right?
You don't have any cake, but trying to eat it fails?

The purpose of an VLAN-interface is to extract tagged incoming packages and send tagged
packages. Tagged with the ID that the interface has. There are tagged incoming packages
only if someone (beyond the bond0 wires) sends VLAN-tagged packets.

If the other end of the bond does not generate VLAN-tagged traffic, then it is futile to
expect tagged traffic. Furthermore, if you tag some traffic on the bond, then it is better
to tag all traffic on the bond.


Second issue. A VLAN is a LAN. A subnet. A broadcast domain. The VLAN technology simply
allows transferring more than one subnet via common hardware. You would need discrete
hardware for each subnet, if there were no VLAN-option.

Are the 10.0.1.150/24 and 10.0.1.151/24 in different subnets?
No, they are both in the same 10.0.1/24 subnet. That is an error.

You have to first decide the address range for each subnet.
For example:
Subnet "data": 10.0.1/24
Subnet "mgmt": 10.0.2/24

Then you have to configure the VLANs on the other side of the bond too.

[Ve0]

I try to use vlan like tunnels to connect bond0 and 2 bridges.

Thank you so much for your explanation. So I try to build something like in this manual: Official oVirt documentation: Bonding VLAN Bridge (RedHat Documentation: 4.3.3.5. Multiple Bridge, Multiple VLAN, and Bond Configuration)

This is LACP mode:

Code: Select all

# cat > /etc/sysconfig/network-scripts/ifcfg-bond0 << EOF
DEVICE=bond0
NM_CONTROLLED=no
USERCTL=no
BOOTPROTO=none
BONDING_OPTS="mode=4 miimon=100"
TYPE=Ethernet
MTU=9000
EOF

Then "enslave" the physical NICs to the bond:

Code: Select all

# cat > /etc/sysconfig/network-scripts/ifcfg-em1 << EOF
NM_CONTROLLED="no"
BOOTPROTO="none"
DEVICE="em1"
ONBOOT="yes"
USERCTL=no
MASTER=bond0
SLAVE=yes
EOF

# cat > /etc/sysconfig/network-scripts/ifcfg-em2 << EOF
NM_CONTROLLED="no"
BOOTPROTO="none"
DEVICE="em2"
ONBOOT="yes"
USERCTL=no
MASTER=bond0
SLAVE=yes
EOF

Then create VLAN interfaces ontop of the bond. In this example, I´m using VLAN ID 1 and 2:

Code: Select all

# cat > /etc/sysconfig/network-scripts/ifcfg-bond0.1 << EOF
DEVICE=bond0.1
VLAN=yes
BOOTPROTO=none
NM_CONTROLLED=no
BRIDGE=br0
MTU=1500
EOF

# cat > /etc/sysconfig/network-scripts/ifcfg-bond0.2 << EOF
DEVICE=bond0.2
VLAN=yes
BOOTPROTO=none
NM_CONTROLLED=no
BRIDGE=ovirtmgmt
MTU=9000
EOF

Lastly, create the bridges ontop of the VLAN interfaces. The names can be whatever you want, except one needs to be called "ovirtmgmt" of course:

Code: Select all

# cat > /etc/sysconfig/network-scripts/ifcfg-ovirtmgmt << EOF
TYPE=Bridge
NM_CONTROLLED="no"
BOOTPROTO="none"
DEVICE="ovirtmgmt"
ONBOOT="yes"
IPADDR=XXX.XXX.XXX.XXX
NETMASK=255.255.255.0
GATEWAY=XXX.XXX.XXX.XXX  # gateway goes into management network
EOF

# cat > /etc/sysconfig/network-scripts/ifcfg-br0 << EOF
TYPE=Bridge
NM_CONTROLLED="no"
BOOTPROTO="none"
DEVICE="br0"
ONBOOT="yes"
IPADDR=XXX.XXX.XXX.XXX
NETMASK=255.255.255.0
DEFROUTE=no
EOF

Last thing is to restart the network for the changes to take effect.

Fedora:

Code: Select all

# systemctl restart network

CentOS:

Code: Select all

# service network restart

This way, you can have almost(4096) as many interfaces as you want with only two physical NICs. I also gave an example on how you tune Jumbo Frames to be active on some interfaces, and have regular window size on the rest.

But this config doesn't work. It works just only if I put IP in bond interface... Again, I try to use VLAN like tunnel to connect bond and bridge interfaces. What I do wrong?

This is what I want:


RedHat Documentation:
4.3.3.5. Multiple Bridge, Multiple VLAN, and Bond Configuration

[jlehtone]

This is what I want:

Have you defined VLANs ID 1 and ID 2 on the Cisco and does it send those two VLANs, both tagged, to the trunk/bond/whatever_cisco_calls_it?

Does the Cisco route between subnets, or do you have a router elsewhere? If elsewhere, do the VLANs reach the router?

Do you now have separate subnet IP ranges for each subnet (i.e. VLAN)?

Do you have any other devices (like management PC) connected to the switch and on one of those subnets?


Bond+vlans+bridges+VM's is confirmed to function in CentOS 6 and CentOS 7, both with network.service and NetworkManager.service.

[Ve0]

Have you defined VLANs ID 1 and ID 2 on the Cisco and does it send those two VLANs, both tagged, to the trunk/bond/whatever_cisco_calls_it?

This pic from internet, I have Mikrotik router without VLANs.

Does the Cisco route between subnets, or do you have a router elsewhere? If elsewhere, do the VLANs reach the router?

I have no VLANs in my network.

Do you now have separate subnet IP ranges for each subnet (i.e. VLAN)?

No, I have no VLANs in my network.

Do you have any other devices (like management PC) connected to the switch and on one of those subnets?

No


Ok... I have the active backup, for example I have VLAN1 on mikrotik on ether1... On bond I have 2 interfaces... So if AB switch interfaces, bond will switch on Mikrotik ether2... but there is no VLAN1 on ether2 interface... And how it works!?

[Ve0]

Did you reed oVirt documents? Can somebody explained this too...

When adding more networks, start by adding them in all of your Hosts before defining the new "Logical Network" in Engine, it´ll get cranky:)

Code: Select all

# cp /etc/sysconfig/network-scripts/ifcfg-bond0.1 /etc/sysconfig/network-scripts/ifcfg-bond0.3
# sed -i -e 's/DEVICE=bond0.1/DEVICE=bond0.3/' -e 's/BRIDGE=br0/BRIDGE=NFS/' /etc/sysconfig/network-scripts/ifcfg-bond0.3
# cp /etc/sysconfig/network-scripts/ifcfg-br0 /etc/sysconfig/network-scripts/ifcfg-NFS
# sed -i -e 's/DEVICE="br0"/DEVICE="NFS"/' -e 's/IPADDR=XXX.XXX.XXX.XXX/IPADDR=YYY.YYY.YYY.YYY/' /etc/sysconfig/network-scripts/ifcfg-NFS

No need to restart the entire network, just ifup the new interfaces:

Code: Select all

# ifup bond0.3
# ifup NFS

Oh, and vdsmd needs to be restarted to pick it up. You can do this while Host is up, but again, Engine will get cranky. More elegant is to put Host in maintenance before.

Fedora:

Code: Select all

# systemctl restart vdsmd

CentOS:

Code: Select all

# service vdsmd restart

Please note! Jumbo Frames must only be active on interfaces that aren´t routed, since the default routing size is 1500.

[jlehtone]

This pic from internet, I have Mikrotik router without VLANs.
I have no VLANs in my network.
No, I have no VLANs in my network.

We do believe you. Why don't you believe?

There is no point in creating a VLAN interface if no tagged packets of that VLAN subnet are coming from the wire nor anyone on the other end of the wire is expecting tagged packets. Like you said, you have no VLANs and therefore VLAN interfaces cannot possibly work.


I must clarify my statement: VLAN's over LACP mode bond does definitely work with libvirt. CentOS installs libvirt by default. I have never had a need to look at oVirt.

Active backup .. I presume that if both Mikrotiks had a VLAN trunk port toward your host, then it might work.
https://wiki.mikrotik.com/wiki/Manual:Interface/VLAN
In that picture there is a trunk between two switches. In your case the switches are Mikrotik and host. The host you have already configured.

[Ve0]

I believe you. But I try to understand how it works...

I must clarify my statement: VLAN's over LACP mode bond does definitely work with libvirt. CentOS installs libvirt by default. I have never had a need to look at oVirt.

oVirt based on KVM. So it based on libvirt with qemu.

Code: Select all

# ps aux |grep libvirt
root      27239  0.0  0.0 1295792 20936 ?       Ssl  Jul28   0:48 /usr/sbin/libvirtd --listen

[jlehtone]

I try to understand how it works...

The description of active-backup bond mode says that exactly one interface of the bond is active. Your host will talk to exactly one Mikrotik at a time. From the Mikrotik's viewpoint there is no bond. It either sees your host, or does not. (If it does not, then the other Mikrotik should see your host.)

In other words, when setting up VLANs we can ignore that we have a bond.

As an example, lets say that we want to have two LANs:
1. Name "foo", ID 42, subnet 10.0.42/24
2. Name "bar", ID 69, subnet 10.0.69/24

On the Mikrotik we create VLAN's with ID's 42 and 69.
The Mikrotik can have addresses 10.0.42.1 and 10.0.69.1 in the respective networks.
We set the Mikrotik's port (that leads to host) to use and tag both VLANs.

Repeat for second Mikrotik, but set addresses 10.0.42.2 and 10.0.69.2. (this part I'm not 100% sure of)

On the host we create the same VLAN's with ID's 42 and 69 (similarly to what you have already done).
The host can have addresses 10.0.42.3 and 10.0.69.3 in the respective networks.


There are now two LAN's, foo and bar, each with three members: host and two Mikrotiks.
The Mikrotiks do not see each other in these LANs, unless there is some other connection,
for the bond uses only one of the wires.

If you want some else device to connect to the host, it is via Mikrotik, either routed or within
same LAN.

[Ve0]

Thank you so much for your patience. After our discussion here and your posts I start learn about VLAN and how it works in CentOS.

Finaly:

Screen Shot 2017-08-14 at 6.47.47 PM.png (66.67 KiB) Viewed 17357 times

Screen Shot 2017-08-14 at 7.08.49 PM.png (59.01 KiB) Viewed 17356 times

Thank you so much! You was totally right! Yes, I had to set up VLAN on Mikrotik.
But I did in different way, I put VLANs interfaces to Mikrotik bridge. I have small network, just only two servers and one UniFi controller behind Mikrotik. Everithing works perfectly) Again, THANKS!!!