Mysterious 12 minute pause in boot

[ktrampe]

Could the machine in question be trying to mount a local partition that no longer exists? I've had some long boot delays when a local share has suddenly disappeared, usually due to my own missteps. More like a 5 minute delay for me, never 12 minutes.

This could be possible. I quadruple-checked my /etc/fstab and /etc/crypttab, but they didn't reveal any mystery partitions. Are there other places I should check?

Have you compared the UUID in those files to the actual UUID's.

Yessir, I have. Does this make sense?

[root@hostname ~]# blkid /dev/xvda1
/dev/xvda1: UUID="00dd0b48-70a0-4217-8696-fb75ad8bd70e" TYPE="xfs"
[root@hostname ~]# cat /etc/fstab
# /etc/fstab
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info

/dev/mapper/centos_web--test-root / xfs defaults 0 0
UUID=00dd0b48-70a0-4217-8696-fb75ad8bd70e /boot xfs defaults 0 0

/dev/mapper/swap none swap defaults 0 0

[TrevorH]

Still waiting for that rd.debug output...

[ktrampe]

Still waiting for that rd.debug output...

Ha ha, sry, still in sanitation. It is a bit interesting though... The times jump around a bit...

[TrevorH]

I suspect the only place you will get the answer is from that output. Quicker than guesswork anyway!

[ktrampe]

Use pastebin.centos.org to show the log when you boot with rd.debug added to the kernel command line. Once it's booted you can see the output in journalctl -ab

I pasted it with a 6 hr expiry: http://pastebin.centos.org/575701/52035951/
PW: LittleFluffyDoggyBitesHard031

It was interesting to me that the clock jumped around a bit. Also, you can see the 12 minute pause reflected in the timestamps.

[TrevorH]

I see a number of errors in that that I don't like the look of.

Mar 05 11:05:38 hostname kernel: Linux version 3.10.0-229.11.1.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Aug 6 01:06:18 UTC 2015
Mar 05 11:05:46 hostname systemd[1]: Failed to configure loopback device: Permission denied
Mar 05 11:17:50 hostname systemd-cryptsetup[688]: Key file /dev/urandom is world-readable. This is not a good idea!
Mar 05 11:17:56 hostname systemd-tmpfiles[777]: stat(/tmp) failed: No such file or directory
Mar 05 11:17:56 hostname systemd-tmpfiles[777]: Failed to create directory /tmp/.X11-unix: No such file or directory
Mar 05 11:17:56 hostname systemd-tmpfiles[777]: Failed to create directory /tmp/.ICE-unix: No such file or directory
Mar 05 11:17:56 hostname systemd-tmpfiles[777]: Failed to create directory /tmp/.XIM-unix: No such file or directory
Mar 05 11:17:56 hostname systemd-tmpfiles[777]: Failed to create directory /tmp/.font-unix: No such file or directory
Mar 05 11:17:56 hostname systemd-tmpfiles[777]: Failed to create directory /tmp/.Test-unix: No such file or directory

First, you seem to be running a 2.5 year old kernel and I suspect the rest of your system is similar. Update.

Second one appears to be due to ipv6, have you tried to disable it? Is there a /etc/sysctl.d/disable_ipv6.conf?

Third one I mention since it probably needs fixing.

The rest all look related. What is your /tmp?

[ktrampe]

I see a number of errors in that that I don't like the look of.

First, you seem to be running a 2.5 year old kernel and I suspect the rest of your system is similar. Update.

Second one appears to be due to ipv6, have you tried to disable it? Is there a /etc/sysctl.d/disable_ipv6.conf?

Third one I mention since it probably needs fixing.

The rest all look related. What is your /tmp?

Thanks for the reply TrevorH. We are in the process of evaluating options which would, in turn, update the kernel or allow us to update the kernel.

/etc/sysctl.d/disable_ipv6.conf does not exist, however, in /etc/sysctl.d/99-sysctl.conf we have:

Code: Select all

net.ipv4.conf.all.arp_notify = 1
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

Will that suffice?

Not sure about your /tmp question, so I beg your pardon for my novice answer. When running in multi-user target, /tmp lives on the encrypted volume. I'm not sure where (or if) it lives before the multi-user target is reached. The logs do seem to indicate that there is something funny with /tmp. You may be on to something...

[TrevorH]

Try setting all those ipv6 parameters to 0 so ipv6 is enabled. That may solve your issue but I think that in order for that to take effect you may need to rebuild the initramfs for the current kernel - use dracut -f to do that for the currently running one.

When running in multi-user target, /tmp lives on the encrypted volume.

How do you have that set up?

[hunter86_bg]

Also, you can onsider setting /tmp as tmpfs (there is a systemd service already).

[TrevorH]

I would suggest fixing ipv6 and /tmp separately though so if either fixes the 12 minute pause, you know which one it was.