Mysterious 12 minute pause in boot

[ktrampe]

I've been hammering and hammering on this one and I'm finally at the end of my rope.

When my OS boots, it sits at the "Welcome to CentOS Linux 7 (Core)!" screen for 12 minutes and 3 seconds. After that, the system boots "normally" and I'm at a login within a few seconds.

Code: Select all

systems-analyze blame

shows:

Code: Select all

         30.962s network.service
         11.582s mariadb.service
          5.378s lvm2-monitor.service
          4.568s httpd.service
          2.575s kdump.service
          2.153s boot.mount
          1.769s lvm2-pvscan@202:2.service
          1.555s tuned.service
           575ms systemd-cryptsetup@swap.service
           469ms rsyslog.service
           ... etc

Not super helpful.

Code: Select all

systems-analyze critical-chain

reveals:

Code: Select all

The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

rescue.target @12min 10.403s
`-rescue.service @12min 10.402s
  `-sysinit.target @12min 10.389s
    `-systemd-update-utmp.service @12min 10.337s +51ms
      `-systemd-tmpfiles-setup.service @12min 10.272s +62ms
        `-rhel-import-state.service @12min 10.145s +125ms
          `-local-fs.target @12min 10.137s
            `-boot.mount @12min 7.983s +2.153s
              `-local-fs-pre.target @12min 7.979s
                `-lvm2-monitor.service @12min 2.599s +5.378s
                  `-lvm2-lvmetad.service @12min 2.911s
                    `-lvm2-lvmetad.socket @12min 2.598s
                      `--.mount
                        `-system.slice
                          `--.slice

I've even done the "plot" and the SVG shows a 12 minute delay as soon as systemd starts, then a flurry of booting-goodness.
I am using LUKS for encryption, could that be hanging or doing something? Wouldn't that show up in the systems-analyze research? The XenServer that this is running on shows little to no CPU, RAM, and Disk activity during the 12 minute wait. This machine has all static IP addresses.

Does anyone have any advice to help me get rid of this 12 minute delay?

I sincerely appreciate any advice!
Kerry

[TrevorH]

Edit the kernel command line at the grub prompt when you reboot and remove rhgb quiet from it then hit ctrl-x to continue. A 12 minute wait ought to be fairly visible so you can see what it was waiting for at that point.

[hunter86_bg]

I had a similar experience when I had a 'resume=SOME_DEVICE' in the kernel parameter, but I did't use hibernation at all.

[desertcat]

I've been hammering and hammering on this one and I'm finally at the end of my rope.

When my OS boots, it sits at the "Welcome to CentOS Linux 7 (Core)!" screen for 12 minutes and 3 seconds. After that, the system boots "normally" and I'm at a login within a few seconds.

Code: Select all

systems-analyze blame

shows:

Code: Select all

         30.962s network.service
         11.582s mariadb.service
          5.378s lvm2-monitor.service
          4.568s httpd.service
          2.575s kdump.service
          2.153s boot.mount
          1.769s lvm2-pvscan@202:2.service
          1.555s tuned.service
           575ms systemd-cryptsetup@swap.service
           469ms rsyslog.service
           ... etc

Not super helpful.

Code: Select all

systems-analyze critical-chain

reveals:

Code: Select all

The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

rescue.target @12min 10.403s
`-rescue.service @12min 10.402s
  `-sysinit.target @12min 10.389s
    `-systemd-update-utmp.service @12min 10.337s +51ms
      `-systemd-tmpfiles-setup.service @12min 10.272s +62ms
        `-rhel-import-state.service @12min 10.145s +125ms
          `-local-fs.target @12min 10.137s
            `-boot.mount @12min 7.983s +2.153s
              `-local-fs-pre.target @12min 7.979s
                `-lvm2-monitor.service @12min 2.599s +5.378s
                  `-lvm2-lvmetad.service @12min 2.911s
                    `-lvm2-lvmetad.socket @12min 2.598s
                      `--.mount
                        `-system.slice
                          `--.slice

I've even done the "plot" and the SVG shows a 12 minute delay as soon as systemd starts, then a flurry of booting-goodness.
I am using LUKS for encryption, could that be hanging or doing something? Wouldn't that show up in the systems-analyze research? The XenServer that this is running on shows little to no CPU, RAM, and Disk activity during the 12 minute wait. This machine has all static IP addresses.

Does anyone have any advice to help me get rid of this 12 minute delay?

I sincerely appreciate any advice!
Kerry

Do what TrevorH says. I have a guess as I have run into this problem before. The delay is caused waiting for some service to "time out". The most frequent villain is ntpd.

Start up a konsole --> su --> [Enter Secret Sauce Password] --> issue command: systemctl stop ntpd --> reboot the machine See if it now boots faster. To re-enable ntpd issue the command: systemctl restart ntpd

[tunk]

Shouldn't that be systemctl disable ntpd, and systemctl enable ntpd plus systemctl start ntpd?

[TrevorH]

That's just guesswork. Read the boot log, get some facts then decide...

[lightman47]

you can also press the right-arrow key [==>] as it's booting to see the output. I know other keys do the same thing, but I don't know which they are.

[hunter86_bg]

Trevor is right.You need some logs.
You can check the

Code: Select all

journalctl -k

[ktrampe]

Wow! Thanks to everyone who replied so quickly! Here's where we're at:

Edit the kernel command line at the grub prompt when you reboot and remove rhgb quiet from it then hit ctrl-x to continue...

Great idea; I thought this was going to provide the info I was after, but unfortunately, It did not. It does the exact same thing it was doing before. It just hangs at the "Welcome to CentOS Linux 7 (Core)!" screen for 12 minutes and then proceeds to boot.

Do what TrevorH says. I have a guess as I have run into this problem before. The delay is caused waiting for some service to "time out". The most frequent villain is ntpd.

Start up a konsole --> su --> [Enter Secret Sauce Password] --> issue command: systemctl stop ntpd --> reboot the machine See if it now boots faster...

Sadly, ntpd is already disabled.

I had a similar experience when I had a 'resume=SOME_DEVICE' in the kernel parameter, but I did't use hibernation at all.

I was unable to find anything like this.

you can also press the right-arrow key [==>] as it's booting to see the output. I know other keys do the same thing, but I don't know which they are.

I did try pressing the right arrow, actually, all the arrow keys, LOL, and it "refreshed" what was already on the screen.
Interestingly, the screen blanks out for a bit before proceeding, but I was able to do a screen recording of the boot sequence. The next lines I can see after the "Welcome" message and 12 min pause are:

Code: Select all

[   OK   ] Stopped Switch Root.
[   OK   ] Stopped target Switch Root.
[   OK   ] Stopped target Initrd File Systems.

Any ideas here?
Thanks so much all,
Kerry

[TrevorH]

It does the exact same thing it was doing before. It just hangs at the "Welcome to CentOS Linux 7 (Core)!" screen for 12 minutes and then proceeds to boot.

Then you did it incorrectly as removing rhgb quiet from the kernel command line should disable that splash screen completely and just leave you looking at thousands of kernel messages as they scroll past.