KVM firewall network bridges

Issues related to configuring your network
itractus
Posts: 9
Joined: 2017/07/17 13:43:06

KVM firewall network bridges

Postby itractus » 2017/07/17 14:26:56

Hi guys,

So my VPS is up for a redesign. I've been trying to get network bridging right on many different servers, and it never really got it to work on CentOS.
But with this renewal I would love to step out of my comfort zone once again and really try out the limits of my knowledge to get a sweet setup going on my VPS.

THE PLAN

I want to step away from firewalld and move towards a fully fledged firewall solution. I've been experimenting with PFSense, and believe this would be a great addition to my server. So I would love to set it up virtually under QEMU and have network bridges to specific docker containers.

Well, this sounds neat but I have no idea how I would go about doing this. Firstly I think I would need to hook up eth0 (Host ethernet) to the external ethernet connection of the VM. And presumably I would create atleast 2 bridges. 1 for the DMZ and thus any webservers, 1 for the applications and containers. Then I would go and set up rules on the internet acces on the PFSense VM.

Would the host require any special configuration? Is this a safe(ish) setup? What am I overlooking? Is this even a good idea?

User avatar
jlehtone
Posts: 1799
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM firewall network bridges

Postby jlehtone » 2017/07/18 06:44:42

It is best to first plan the (logical) network without considering the hardware.

Lets assume that every item is a separate box. You have a wire coming in. The WAN.
You have a box with three ports and PFSense. A firewall and router. Perhaps DHCP, DNS, and time too.
WAN wire obviously plugs in. The other two ports are connected to two switches, labeled LAN and DMZ.

The other boxes are plugged into either LAN or DMZ. The host would be best to connect to the LAN?

Decide subnets and IP addresses for involved parties.


If that looks right, then lets implement it. Create two bridges on the host: LAN and DMZ.

Code: Select all

man nmcli-examples
man nmcli

ipv6.method ignore
bridge.stp no
For DMZ: ipv4.method disabled
For LAN: ipv4.method manual, and add IP/prefix and PFsense's LAN-port IP as default route.

Create bridges on the host, not with libvirt.

For eth0:
ipv4.method disabled
ipv6.method ignore

Note: The host has no direct outbound connectivity. For initial setup you might need it to use the eth0.

Create a VM and add both LAN and DMZ bridges to it (virtio network devices). Once the PFSense is up
(without WAN port), add the eth0 to it as a passthrough device. The virt-manager seems to offer at least
a 'macvtap', but there should be more than one passthrough mode. The PFSense VM will "own" the eth0
and host shall have no IP address on it.

Option 2: Create bridge "WAN". Enslave eth0 to it. Add the WAN to the PFSense VM.


A discrete firewall device has less attack vectors than a VM. Probably not an option in VPS.


A VPS with no console access? Then the juggle of "who has IP and who does not" is delicate, brittle,
or fascinating.

itractus
Posts: 9
Joined: 2017/07/17 13:43:06

Re: KVM firewall network bridges

Postby itractus » 2017/07/20 09:22:59

Thanks a million, your post got me guided through most of the setup! Not only that, it also makes sense to me now :D
I've got pfsense installed and ready to route my network!

Now I seem to be having an issue with qemu and KVM. I get the following error when I add any bridged network connection. No matter the setting:
Image

If I run the VM without network adapters, it works fine. But that's pretty useless for a firewall :D

Edit:
I found the solution on the ubuntu forums after some digging.
I added the <rom bar='off'/> option to both interface settings of the VM. Now I can start configuring! Check out https://askubuntu.com/questions/190929/how-do-i-disable-unwanted-ipxe-boot-attempt-in-libvirt-qemu-kvm

itractus
Posts: 9
Joined: 2017/07/17 13:43:06

Re: KVM firewall network bridges

Postby itractus » 2017/07/20 12:41:43

One last issue that I seem to have: nmcli

So I've now finally have a pfsense instance with 3 connections:
1 WAN linked to eth0 with a passthrough,
1 LAN adapter linked to the LAN bridge,
1 DMZ adapter linked to the DMZ bridge.

Now I need to create a new adapter on the host and connect it to the LAN bridge before I can use the web interface of pfsense. My VPS is cli only, would I still be able to set this up and configure the firewall? And how would I do this in nmcli?
edit:
Image

User avatar
jlehtone
Posts: 1799
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM firewall network bridges

Postby jlehtone » 2017/07/20 14:49:01

Both DMZ and LAN are trying to get an IP address from DHCP in the picture.
The PFSense probably does not have DHCP server running (yet).

Plan B, manual address.

Lets say that LAN is subnet 192.168.13.0/24, the PFSense has 192.168.13.1, and the host could have 192.168.13.2.

Code: Select all

nmcli con down LAN
nmcli con mod LAN ipv4.method manual ipv4.addresses 192.168.13.2/24
nmcli con up LAN


There is also 'nmtui'.

itractus
Posts: 9
Joined: 2017/07/17 13:43:06

Re: KVM firewall network bridges

Postby itractus » 2017/07/21 10:03:40

So the final issue is as follows:

I can't seem to connect to the internet with my eth0 on passthrough in qemu.
I've followed the troubleshoot guide on pfsense (https://doc.pfsense.org/index.php/Conne ... leshooting) but all settings seem to be correct.
If i do an "ip a s" I can see that the eth0 does recieve the correct IP address and subnet.
If I try a ping test from the firewall to say google (8.8.8.8) I get 100% dropped packages.

Is there a way to troubleshoot the qemu passthrough?

edit:
Would it be a good solution to create a new bridge on the host and attach the VM to that bridge, set a static IP and be done with it?

itractus
Posts: 9
Joined: 2017/07/17 13:43:06

Re: KVM firewall network bridges

Postby itractus » 2017/07/24 13:04:01

So I think I found the solution. The passthrough would require hardware virtualisation extentions, which my server doesn't.
source:http://wiki.libvirt.org/page/Networking#PCI_Passthrough_of_host_network_devices

User avatar
jlehtone
Posts: 1799
Joined: 2007/12/11 08:17:33
Location: Finland

Re: KVM firewall network bridges

Postby jlehtone » 2017/07/26 06:37:08

There are more than one "passthrough" mode in libvirt and not all of them require hardware.
IIRC, enabling PCI passtrough "removes" the device from the host, i.e. the host OS will not
see the "eth0 device" at all, while it is handed to a guest.


You do have a "virbr0" interface. That is the "Default network" that the libvirt installation
defines and libvirtd activates. It is a bridge interface. The host has an IP on the "virbr0
subnet" and runs dnsmasq process as DHCP and DNS for that subnet. The host is also the
router between "virbr0 subnet" and other networks. A router with NAT. The libvirt configures
and starts all that.

For a simple setup that is easy. One simply creates VM's and adds them to "virbr0 subnet".
The VM's get network configuration from the dnsmasq. All set.

One can create more networks with libvirt. route+NAT like the "Default", route sans NAT,
no routing, bridged, (and some other mode). The bridged subnet does not have dnsmasq.

However, if one does not start the libvirt services, then there is no virbr0 & co. The networking
of the host should not thus depend on the "networks" that the libvirt "manages".


One scenario is to create bridge on the host.
Enslave the eth0 to it.
Add WAN-port of the PFSense VM to the bridge.
The host may, but does not have to, have an address on the bridge. The outside DHCP
will think that there are max two computers behind the wire that leads to eth0 of the host.
The bridge is created whether PFSense starts or not.

The DMZ and LAN could be created with libvirt, rather than manually. They are less useful
unless libvirt is up. VM's on them could thus get config from dnsmasq processes that the
libvirt starts. I do see two issues there:
1. Host probably should not have IP on both. Not sure whether libvirt has "only DHCP" option.
2. That dnsmasq offers the host IP as gateway or no gateway at all. You want PFSense as
gateway. There are ways to tune the libvirt's dnsmasq config though.


One can set dnsmasq.service on the host to listen only bridges defined on the host for DHCP.


There is always the dilemma of on which subnets the host has IP.
If it has one outside of PFSense, then it is exposed, relying on its netfilter.
If it does not have one outside of PFSense and the PFSense fails, then you cannot reach
the host to fix the PFSense.


EDIT:
My VPS is cli only,

If you do connect with ssh and you connect from a machine that runs X11 server,
then you can run X11 clients on the VPS. All you need is some X-libs there.
The ssh can tunnel the X11 to your machine. App in VPS, buts its window on your
screen.

If you do connect with ssh from Linux machine, then you can run virt-manager (a python
app) on your machine and make it contact libvirtd of the VPS via ssh.

itractus
Posts: 9
Joined: 2017/07/17 13:43:06

Re: KVM firewall network bridges

Postby itractus » 2017/07/27 13:30:37

So I followed all the steps of creating a bridge, attaching my eth0 as a slave, setting it as my WAN connection in PFSense, and still I don’t get an IP over the bridge. Might Centos be bocking traffic? This would be strange, as I disabled firewalld to try to get this to work.
So I enslave the eth0 to the brWAN bridge:
Image
The WAN-port of the PFSense VM, linked to the bridge, doesn’t receive the DHCP config my ISP sends me:
Image
The bridge:
Image
The bridge does seem to be online:
Image
Is it possible that Centos 7 is blocking traffic on the bridge? I know the host isn’t connected to the bridge so it won’t get an IP, but the eth0 still seems to be active on the host since I can ping 8.8.8.8 with the host even though ICMP traffic is blocked on LAN. So for some reason my host is not releasing eth0 as an adapter.

itractus
Posts: 9
Joined: 2017/07/17 13:43:06

Re: KVM firewall network bridges

Postby itractus » 2017/08/10 11:11:46

After a reinstallation and reconfigure I keep bumping into the same issue.
if I connect the host to the bridge, I get an IP, and can surf the web.
If I connect the VM to the bridge, I don't get an IP, and if I set the correct IP settings and gateway settings in the VM, I still can't connect.