firewalld borked after boot, but systemctl restart fixes it

Support for security such as Firewalls and securing linux
Post Reply
jimj
Posts: 93
Joined: 2014/10/01 05:34:57

firewalld borked after boot, but systemctl restart fixes it

Post by jimj » 2015/03/28 04:52:37

I'm running CentOS 7 on my desktop. I've noticed recently that after booting my firewalld comes up in a borked state. Most firewalld commands don't work and I can't SSH into my box. However, simply restarting firewalld fixes everything. Where do I start troubleshooting something like this?

Code: Select all

[root@black ~]# systemctl status firewalld -l
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Fri 2015-03-27 23:45:35 CDT; 2min 25s ago
 Main PID: 788 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─788 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Mar 27 23:45:35 black systemd[1]: Started firewalld - dynamic firewall daemon.
Mar 27 23:45:37 black firewalld[788]: 2015-03-27 23:45:37 ERROR: INVALID_ZONE

[root@black ~]# systemctl restart firewalld

[root@black ~]# systemctl status firewalld -l
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Fri 2015-03-27 23:48:20 CDT; 2s ago
 Main PID: 3869 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─3869 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
Mar 27 23:48:20 black systemd[1]: Started firewalld - dynamic firewall daemon.

aks
Posts: 3073
Joined: 2014/09/20 11:22:14

Re: firewalld borked after boot, but systemctl restart fixes

Post by aks » 2015/03/28 13:21:53

What does the journal say for the firewall after boot?
The only clue here is ERROR: INVALID_ZONE - perhaps you have an invalid zone set at boot or the interfaces have not transitioned to the zone when firewalld starts up?

jimj
Posts: 93
Joined: 2014/10/01 05:34:57

Re: firewalld borked after boot, but systemctl restart fixes

Post by jimj » 2015/03/28 16:46:40

aks wrote:What does the journal say for the firewall after boot?
The only clue here is ERROR: INVALID_ZONE - perhaps you have an invalid zone set at boot or the interfaces have not transitioned to the zone when firewalld starts up?
Journal basically says the same thing. You can see both the boot messages with the warning and then my successful restart a few minutes later. The GUI says my default zone is "public".

Code: Select all

[root@black ~]# journalctl |ack firewall
Mar 27 23:45:34 black systemd[1]: Starting firewalld - dynamic firewall daemon...
Mar 27 23:45:35 black kernel: Bridge firewalling registered
Mar 27 23:45:35 black systemd[1]: Started firewalld - dynamic firewall daemon.
Mar 27 23:45:37 black firewalld[788]: 2015-03-27 23:45:37 ERROR: INVALID_ZONE
Mar 27 23:45:37 black NetworkManager[1000]: <warn> (enp3s0) firewall zone add/change failed [1]: (32) INVALID_ZONE
Mar 27 23:48:18 black systemd[1]: Stopping firewalld - dynamic firewall daemon...
Mar 27 23:48:20 black systemd[1]: Starting firewalld - dynamic firewall daemon...
Mar 27 23:48:20 black systemd[1]: Started firewalld - dynamic firewall daemon.
The only related log entry from /var/log/firewalld was:
2015-03-27 23:45:37 ERROR: INVALID_ZONE

User avatar
jyoung
Posts: 102
Joined: 2014/09/22 13:40:31
Location: Nashville, TN, USA

Re: firewalld borked after boot, but systemctl restart fixes

Post by jyoung » 2015/03/28 18:07:19

jimj wrote:
aks wrote:What does the journal say for the firewall after boot?
The only clue here is ERROR: INVALID_ZONE - perhaps you have an invalid zone set at boot or the interfaces have not transitioned to the zone when firewalld starts up?
Journal basically says the same thing. You can see both the boot messages with the warning and then my successful restart a few minutes later. The GUI says my default zone is "public".

Code: Select all

[root@black ~]# journalctl |ack firewall
Mar 27 23:45:34 black systemd[1]: Starting firewalld - dynamic firewall daemon...
Mar 27 23:45:35 black kernel: Bridge firewalling registered
Mar 27 23:45:35 black systemd[1]: Started firewalld - dynamic firewall daemon.
Mar 27 23:45:37 black firewalld[788]: 2015-03-27 23:45:37 ERROR: INVALID_ZONE
Mar 27 23:45:37 black NetworkManager[1000]: <warn> (enp3s0) firewall zone add/change failed [1]: (32) INVALID_ZONE
Mar 27 23:48:18 black systemd[1]: Stopping firewalld - dynamic firewall daemon...
Mar 27 23:48:20 black systemd[1]: Starting firewalld - dynamic firewall daemon...
Mar 27 23:48:20 black systemd[1]: Started firewalld - dynamic firewall daemon.
The only related log entry from /var/log/firewalld was:
2015-03-27 23:45:37 ERROR: INVALID_ZONE


[*]Are you running Docker?
[*]Are there any SELinux denials after booting your computer?
-- Jeremy --

jimj
Posts: 93
Joined: 2014/10/01 05:34:57

Re: firewalld borked after boot, but systemctl restart fixes

Post by jimj » 2015/03/28 18:52:10

jyoung wrote:[*]Are you running Docker?
[*]Are there any SELinux denials after booting your computer?
  1. No (unless docker is configured to do something by default, I've never messed with it)
  2. No, my SELinux is disabled

User avatar
jyoung
Posts: 102
Joined: 2014/09/22 13:40:31
Location: Nashville, TN, USA

Re: firewalld borked after boot, but systemctl restart fixes

Post by jyoung » 2015/03/28 19:25:03

What do these produce?

Code: Select all

echo -e "My default zone is $( firewall-cmd --get-default-zone )\n\nMy firewall is configured as follows:\n$( for ZONE in $( firewall-cmd --get-active-zones | egrep "^[A-Z]|^[a-z]" ); do firewall-cmd --zone=$ZONE --list-all; done )"

Code: Select all

nmcli con show | gawk '{print $1}' | grep -v NAME | while read CONN; do echo $CONN; nmcli con show $CONN | grep -i zone; echo; done

Code: Select all

journalctl -xln1000 --unit=firewalld
-- Jeremy --

jimj
Posts: 93
Joined: 2014/10/01 05:34:57

Re: firewalld borked after boot, but systemctl restart fixes

Post by jimj » 2015/03/28 19:52:22

jyoung wrote:What do these produce?
Thanks for those detailed instructions. I ran them under my current working state and have pasted those results below. Next I rebooted and ran the commands again so I could show you what it looks like when my system is in a borked state (for example I know commands like 'firewall-cmd --get-default-zone' fail). Well, after being able to reliably reproduce this problem for several weeks I can no longer reproduce it. I.e. after rebooting my PC my firewall now starts normally.

So I'm happy that the problem appears to be gone, but I'm also disappointed I don't know what caused or fixed it. I just hope it doesn't rear its ugly head again someday when I'm away and need remote SSH access to my PC.

In case you can spot anything amiss from your commands' output I've pasted it below. The only error is from the boot up last night. My active NIC is "enp3s0".

Code: Select all

echo -e "My default zone is $( firewall-cmd --get-default-zone )\n\nMy firewall is configured as follows:\n$( for ZONE in $( firewall-cmd --get-active-zones | egrep "^[A-Z]|^[a-z]" ); do firewall-cmd --zone=$ZONE --list-all; done )"
My default zone is public

My firewall is configured as follows:
public (default, active)
  interfaces: enp3s0
  sources: 
  services: dhcpv6-client ssh
  ports: 
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules: 


nmcli con show | gawk '{print $1}' | grep -v NAME | while read CONN; do echo $CONN; nmcli con show $CONN | grep -i zone; echo; done
enp4s0
connection.zone:                        --

enp3s0
connection.zone:                        --
GENERAL.ZONE:                           --




journalctl -xln1000 --unit=firewalld
-- Logs begin at Fri 2015-03-27 18:45:32 CDT, end at Sat 2015-03-28 14:33:48 CDT
Mar 27 23:45:34 black systemd[1]: Starting firewalld - dynamic firewall daemon..
-- Subject: Unit firewalld.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit firewalld.service has begun starting up.
Mar 27 23:45:35 black systemd[1]: Started firewalld - dynamic firewall daemon.
-- Subject: Unit firewalld.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit firewalld.service has finished starting up.
-- 
-- The start-up result is done.
Mar 27 23:45:37 black firewalld[788]: 2015-03-27 23:45:37 ERROR: INVALID_ZONE
Mar 27 23:48:18 black systemd[1]: Stopping firewalld - dynamic firewall daemon..
-- Subject: Unit firewalld.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit firewalld.service has begun shutting down.
Mar 27 23:48:20 black systemd[1]: Starting firewalld - dynamic firewall daemon..
-- Subject: Unit firewalld.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit firewalld.service has begun starting up.
Mar 27 23:48:20 black systemd[1]: Started firewalld - dynamic firewall daemon.
-- Subject: Unit firewalld.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit firewalld.service has finished starting up.
-- 
-- The start-up result is done.

User avatar
jyoung
Posts: 102
Joined: 2014/09/22 13:40:31
Location: Nashville, TN, USA

Re: firewalld borked after boot, but systemctl restart fixes

Post by jyoung » 2015/03/28 20:13:57

I'm glad that your problem is gone. Nothing sticks out from the output of those commands, so maybe a patch that you had installed at some point addressed an underlying issue.
-- Jeremy --

jimj
Posts: 93
Joined: 2014/10/01 05:34:57

Re: firewalld borked after boot, but systemctl restart fixes

Post by jimj » 2015/03/29 03:01:54

No patches installed since yesterday when it was 100% reproducible. Maybe a service start up timing issue? Time will tell if it's really "fixed", I'll keep my fingers crossed. Thanks for your help.

johnds
Posts: 2
Joined: 2015/07/31 07:54:27

Re: firewalld borked after boot, but systemctl restart fixes it

Post by johnds » 2017/07/05 07:24:57

I recently had a similar problem on my Fedora 25 system , where I had to manually start firewalld after a reboot, because it was started and immediately stopped again.
The problem turned out to be caused by the NetworkManager dispatcher script /etc/NetworkManager/dispatcher.d/30-iptables.
After I removed the script firewalld now stays running after a reboot.

Post Reply