Mysterious 12 minute pause in boot

General support questions
ktrampe
Posts: 21
Joined: 2018/02/20 22:39:56

Mysterious 12 minute pause in boot

Post by ktrampe » 2018/02/20 23:06:01

I've been hammering and hammering on this one and I'm finally at the end of my rope. :)

When my OS boots, it sits at the "Welcome to CentOS Linux 7 (Core)!" screen for 12 minutes and 3 seconds. After that, the system boots "normally" and I'm at a login within a few seconds.

Code: Select all

systems-analyze blame
shows:

Code: Select all

         30.962s network.service
         11.582s mariadb.service
          5.378s lvm2-monitor.service
          4.568s httpd.service
          2.575s kdump.service
          2.153s boot.mount
          1.769s lvm2-pvscan@202:2.service
          1.555s tuned.service
           575ms systemd-cryptsetup@swap.service
           469ms rsyslog.service
           ... etc
Not super helpful.

Code: Select all

systems-analyze critical-chain
reveals:

Code: Select all

The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

rescue.target @12min 10.403s
`-rescue.service @12min 10.402s
  `-sysinit.target @12min 10.389s
    `-systemd-update-utmp.service @12min 10.337s +51ms
      `-systemd-tmpfiles-setup.service @12min 10.272s +62ms
        `-rhel-import-state.service @12min 10.145s +125ms
          `-local-fs.target @12min 10.137s
            `-boot.mount @12min 7.983s +2.153s
              `-local-fs-pre.target @12min 7.979s
                `-lvm2-monitor.service @12min 2.599s +5.378s
                  `-lvm2-lvmetad.service @12min 2.911s
                    `-lvm2-lvmetad.socket @12min 2.598s
                      `--.mount
                        `-system.slice
                          `--.slice

I've even done the "plot" and the SVG shows a 12 minute delay as soon as systemd starts, then a flurry of booting-goodness.
I am using LUKS for encryption, could that be hanging or doing something? Wouldn't that show up in the systems-analyze research? The XenServer that this is running on shows little to no CPU, RAM, and Disk activity during the 12 minute wait. This machine has all static IP addresses.

Does anyone have any advice to help me get rid of this 12 minute delay?

I sincerely appreciate any advice!
Kerry

User avatar
TrevorH
Forum Moderator
Posts: 24098
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Mysterious 12 minute pause in boot

Post by TrevorH » 2018/02/21 00:49:44

Edit the kernel command line at the grub prompt when you reboot and remove rhgb quiet from it then hit ctrl-x to continue. A 12 minute wait ought to be fairly visible so you can see what it was waiting for at that point.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

hunter86_bg
Posts: 1451
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Mysterious 12 minute pause in boot

Post by hunter86_bg » 2018/02/21 05:03:58

I had a similar experience when I had a 'resume=SOME_DEVICE' in the kernel parameter, but I did't use hibernation at all.

desertcat
Posts: 482
Joined: 2014/08/07 02:17:29
Location: Tucson, AZ

Re: Mysterious 12 minute pause in boot

Post by desertcat » 2018/02/21 09:37:40

ktrampe wrote:I've been hammering and hammering on this one and I'm finally at the end of my rope. :)

When my OS boots, it sits at the "Welcome to CentOS Linux 7 (Core)!" screen for 12 minutes and 3 seconds. After that, the system boots "normally" and I'm at a login within a few seconds.

Code: Select all

systems-analyze blame
shows:

Code: Select all

         30.962s network.service
         11.582s mariadb.service
          5.378s lvm2-monitor.service
          4.568s httpd.service
          2.575s kdump.service
          2.153s boot.mount
          1.769s lvm2-pvscan@202:2.service
          1.555s tuned.service
           575ms systemd-cryptsetup@swap.service
           469ms rsyslog.service
           ... etc
Not super helpful.

Code: Select all

systems-analyze critical-chain
reveals:

Code: Select all

The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

rescue.target @12min 10.403s
`-rescue.service @12min 10.402s
  `-sysinit.target @12min 10.389s
    `-systemd-update-utmp.service @12min 10.337s +51ms
      `-systemd-tmpfiles-setup.service @12min 10.272s +62ms
        `-rhel-import-state.service @12min 10.145s +125ms
          `-local-fs.target @12min 10.137s
            `-boot.mount @12min 7.983s +2.153s
              `-local-fs-pre.target @12min 7.979s
                `-lvm2-monitor.service @12min 2.599s +5.378s
                  `-lvm2-lvmetad.service @12min 2.911s
                    `-lvm2-lvmetad.socket @12min 2.598s
                      `--.mount
                        `-system.slice
                          `--.slice

I've even done the "plot" and the SVG shows a 12 minute delay as soon as systemd starts, then a flurry of booting-goodness.
I am using LUKS for encryption, could that be hanging or doing something? Wouldn't that show up in the systems-analyze research? The XenServer that this is running on shows little to no CPU, RAM, and Disk activity during the 12 minute wait. This machine has all static IP addresses.

Does anyone have any advice to help me get rid of this 12 minute delay?

I sincerely appreciate any advice!
Kerry
Do what TrevorH says. I have a guess as I have run into this problem before. The delay is caused waiting for some service to "time out". The most frequent villain is ntpd.

Start up a konsole --> su --> [Enter Secret Sauce Password] --> issue command: systemctl stop ntpd --> reboot the machine See if it now boots faster. To re-enable ntpd issue the command: systemctl restart ntpd

tunk
Posts: 462
Joined: 2017/02/22 15:08:17

Re: Mysterious 12 minute pause in boot

Post by tunk » 2018/02/21 10:32:49

Shouldn't that be systemctl disable ntpd, and systemctl enable ntpd plus systemctl start ntpd?

User avatar
TrevorH
Forum Moderator
Posts: 24098
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Mysterious 12 minute pause in boot

Post by TrevorH » 2018/02/21 11:24:40

That's just guesswork. Read the boot log, get some facts then decide...
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

lightman47
Posts: 770
Joined: 2014/05/21 20:16:00
Location: Central New York, USA

Re: Mysterious 12 minute pause in boot

Post by lightman47 » 2018/02/21 12:46:46

you can also press the right-arrow key [==>] as it's booting to see the output. I know other keys do the same thing, but I don't know which they are.
"There's NO detailed answer to a question lacking details."

hunter86_bg
Posts: 1451
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Mysterious 12 minute pause in boot

Post by hunter86_bg » 2018/02/21 15:58:55

Trevor is right.You need some logs.
You can check the

Code: Select all

journalctl -k

ktrampe
Posts: 21
Joined: 2018/02/20 22:39:56

Re: Mysterious 12 minute pause in boot

Post by ktrampe » 2018/02/21 16:47:38

Wow! Thanks to everyone who replied so quickly! Here's where we're at:
TrevorH wrote:Edit the kernel command line at the grub prompt when you reboot and remove rhgb quiet from it then hit ctrl-x to continue...
Great idea; I thought this was going to provide the info I was after, but unfortunately, It did not. It does the exact same thing it was doing before. It just hangs at the "Welcome to CentOS Linux 7 (Core)!" screen for 12 minutes and then proceeds to boot.
desertcat wrote:Do what TrevorH says. I have a guess as I have run into this problem before. The delay is caused waiting for some service to "time out". The most frequent villain is ntpd.

Start up a konsole --> su --> [Enter Secret Sauce Password] --> issue command: systemctl stop ntpd --> reboot the machine See if it now boots faster...
Sadly, ntpd is already disabled.
hunter86_bg wrote:I had a similar experience when I had a 'resume=SOME_DEVICE' in the kernel parameter, but I did't use hibernation at all.
I was unable to find anything like this. :(

lightman47 wrote:you can also press the right-arrow key [==>] as it's booting to see the output. I know other keys do the same thing, but I don't know which they are.
I did try pressing the right arrow, actually, all the arrow keys, LOL, and it "refreshed" what was already on the screen.
Interestingly, the screen blanks out for a bit before proceeding, but I was able to do a screen recording of the boot sequence. The next lines I can see after the "Welcome" message and 12 min pause are:

Code: Select all

[   OK   ] Stopped Switch Root.
[   OK   ] Stopped target Switch Root.
[   OK   ] Stopped target Initrd File Systems.
Any ideas here?
Thanks so much all,
Kerry

User avatar
TrevorH
Forum Moderator
Posts: 24098
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Mysterious 12 minute pause in boot

Post by TrevorH » 2018/02/21 17:36:17

It does the exact same thing it was doing before. It just hangs at the "Welcome to CentOS Linux 7 (Core)!" screen for 12 minutes and then proceeds to boot.
Then you did it incorrectly as removing rhgb quiet from the kernel command line should disable that splash screen completely and just leave you looking at thousands of kernel messages as they scroll past.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Post Reply