[SOLVED -> workaround] CentOS: very slow system boot / sometimes hangs entirely
[SOLVED -> workaround] CentOS: very slow system boot / sometimes hangs entirely
Hi,
I have CentOS installed on a GigaByte N3150-D3V Board.
For some reason I can't figure out, the system boots *very* slowly and sometimes even hangs. Click here for a demonstration video of the slow boot.
My gut tells me it has something to do with the video framebuffer, so I tried to disable it by various setting (blacklist i915 in /etc/modprobe.d/blacklist.conf), following this Guide or this one for ubuntu.
But nothing helps neither as far as the slow boot is concerned nor as far as disabling framebuffer and the pinguins.
I ran a systemd-analyze plot > heBacula-boot-analysis.svg and apparently kernel (with 33 sec) and initrd (47 sec) take the most time to complete.
There is also the output of dmesg which I examined but there's also nothing that looks suspicious to me.
Anyone got any pointers as to what the cause of the problem could be?
Any help would be greatly appreciated!
Thanks in advance,
socrates
I have CentOS installed on a GigaByte N3150-D3V Board.
For some reason I can't figure out, the system boots *very* slowly and sometimes even hangs. Click here for a demonstration video of the slow boot.
My gut tells me it has something to do with the video framebuffer, so I tried to disable it by various setting (blacklist i915 in /etc/modprobe.d/blacklist.conf), following this Guide or this one for ubuntu.
But nothing helps neither as far as the slow boot is concerned nor as far as disabling framebuffer and the pinguins.
I ran a systemd-analyze plot > heBacula-boot-analysis.svg and apparently kernel (with 33 sec) and initrd (47 sec) take the most time to complete.
There is also the output of dmesg which I examined but there's also nothing that looks suspicious to me.
Anyone got any pointers as to what the cause of the problem could be?
Any help would be greatly appreciated!
Thanks in advance,
socrates
Last edited by socrates on 2018/03/15 08:23:55, edited 1 time in total.
Re: CentOS: very slow system boot / sometimes hangs entirely
Almost all the big gaps in timestamps in your dmesg output seem to be to do with USB devices.
Your svg file tells me nothing. What's the text output from systemd-analyze blame instead?
Your svg file tells me nothing. What's the text output from systemd-analyze blame instead?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Re: CentOS: very slow system boot / sometimes hangs entirely
Well, the only thing I found interesting in the SVG file was in the upper left corner:TrevorH wrote: Your svg file tells me nothing.
Code: Select all
Startup finished in 16.682s (firmwre) + 2.831s (loader) + 33.105s (kernel) + 47.164s (initrd) + 16.892s (userspace) = 1 min 56.677s
I didn't include the ouptut from systemd-analyze blame because the "heavy" part is not included here:
TrevorH wrote: What's the text output from systemd-analyze blame instead?
Code: Select all
9.289s systemd-vconsole-setup.service
5.676s NetworkManager-wait-online.service
3.669s mariadb.service
3.598s kdump.service
1.617s postfix.service
1.424s network.service
1.104s lvm2-monitor.service
1.103s dev-mapper-vg_os\x2droot.device
933ms firewalld.service
672ms lvm2-pvscan@9:124.service
9.289s systemd-vconsole-setup.service
5.676s NetworkManager-wait-online.service
3.669s mariadb.service
3.598s kdump.service
1.617s postfix.service
1.424s network.service
1.104s lvm2-monitor.service
1.103s dev-mapper-vg_os\x2droot.device
933ms firewalld.service
672ms lvm2-pvscan@9:124.service
619ms tuned.service
494ms srv.mount
429ms boot.mount
373ms lvm2-pvscan@9:127.service
319ms var.mount
304ms auditd.service
258ms systemd-logind.service
222ms NetworkManager.service
217ms tmp.mount
213ms var-log.mount
196ms systemd-journald.service
179ms proc-fs-nfsd.mount
178ms kmod-static-nodes.service
175ms dev-hugepages.mount
161ms systemd-tmpfiles-clean.service
143ms polkit.service
141ms systemd-udev-trigger.service
140ms dev-mqueue.mount
132ms sshd.service
126ms systemd-tmpfiles-setup.service
125ms rhel-import-state.service
121ms var-log-audit.mount
110ms home.mount
99ms systemd-tmpfiles-setup-dev.service
99ms systemd-udevd.service
98ms rhel-readonly.service
95ms yum-cron.service
94ms mdmonitor.service
90ms microcode.service
88ms ntpd.service
85ms gssproxy.service
85ms rhel-dmesg.service
83ms rpc-statd-notify.service
83ms bacula-sd.service
82ms systemd-readahead-done.service
77ms dev-disk-by\x2duuid-82a895cc\x2d2acb\x2d45a3\x2d90df\x2d7be08a242de0.swap
67ms bacula-fd.service
59ms plymouth-start.service
58ms bacula-dir.service
57ms boot-efi_disk_SN_WCC4N7JCCFY9.mount
54ms systemd-readahead-collect.service
53ms systemd-readahead-replay.service
53ms rsyslog.service
50ms boot-efi_disk_SN_WCC4N7JCCZC9.mount
50ms boot-efi.mount
47ms sys-kernel-debug.mount
47ms systemd-sysctl.service
47ms boot-efi_disk_SN_WCC4N2ATJA1C.mount
44ms systemd-remount-fs.service
Right. The kernel even claims to have found a Logitech mouse:TrevorH wrote:Almost all the big gaps in timestamps in your dmesg output seem to be to do with USB devices.
Code: Select all
[ 74.156209] input: PS/2 Logitech Mouse as /devices/platform/i8042/serio1/input/input5
-
- Posts: 2019
- Joined: 2015/02/17 15:14:33
- Location: Bulgaria
- Contact:
Re: CentOS: very slow system boot / sometimes hangs entirely
Can you try with the rescue kernel and check the time needed to boot ?
Did you notice this behaviour in earlier kernels or it is observed with the latest one ?
Did you notice this behaviour in earlier kernels or it is observed with the latest one ?
Re: CentOS: very slow system boot / sometimes hangs entirely / entirely disabling graphical boot?
TL;DR: How do I completely disable the graphical boot and boot into old fashioned text mode?
Hi,
sorry for the delay in replying to this issue. Reason was that the server booted very slowly but it always reached the login -- so I could live with that.
Now it starts to hang on boot again. So I need to investigate further.
To answer the question: The phenomena is the same, regardless of using normal or rescue mode.
My gut tells me it has something to do with the graphical boot.
Can someone tell me how to disable this? I would like to have pure text mode login.
No penguins, no framebuffer, nothing at all.
I already tried to remove the "rhgb quiet" from the kernel line in grub - no success.
I tried #plymouth-set-default-theme to text or details but the graphical boot remains.
Thanks in advance,
soc
Hi,
sorry for the delay in replying to this issue. Reason was that the server booted very slowly but it always reached the login -- so I could live with that.
Now it starts to hang on boot again. So I need to investigate further.
To answer the question: The phenomena is the same, regardless of using normal or rescue mode.
My gut tells me it has something to do with the graphical boot.
Can someone tell me how to disable this? I would like to have pure text mode login.
No penguins, no framebuffer, nothing at all.
I already tried to remove the "rhgb quiet" from the kernel line in grub - no success.
I tried #plymouth-set-default-theme to text or details but the graphical boot remains.
Thanks in advance,
soc
Re: CentOS: very slow system boot / sometimes hangs entirely
Apparently you didn't remove rhgb quiet when you thought you did.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Re: CentOS: very slow system boot / sometimes hangs entirely
mmh... that would be strange
Here's my /boot/efi/EFI/centos/grub.cfg
And here is the bootline from grub I boot via Ctrl+X. https://pasteboard.co/HbTYEm8.jpg
I'm sorry but I don't see anywhere "rhgb quite". What am I missing??
Here's my /boot/efi/EFI/centos/grub.cfg
Code: Select all
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
### BEGIN /etc/grub.d/00_header ###
set pager=1
if [ -s $prefix/grubenv ]; then
load_env
fi
if [ "${next_entry}" ] ; then
set default="${next_entry}"
set next_entry=
save_env next_entry
set boot_once=true
else
set default="${saved_entry}"
fi
if [ x"${feature_menuentry_id}" = xy ]; then
menuentry_id_option="--id"
else
menuentry_id_option=""
fi
export menuentry_id_option
if [ "${prev_saved_entry}" ]; then
set saved_entry="${prev_saved_entry}"
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi
function savedefault {
if [ -z "${boot_once}" ]; then
saved_entry="${chosen}"
save_env saved_entry
fi
}
function load_video {
if [ x$feature_all_video_module = xy ]; then
insmod all_video
else
insmod efi_gop
insmod efi_uga
insmod ieee1275_fb
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus
fi
}
terminal_output console
if [ x$feature_timeout_style = xy ] ; then
set timeout_style=menu
set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
set timeout=5
fi
### END /etc/grub.d/00_header ###
### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
set tuned_initrd=""
### END /etc/grub.d/00_tuned ###
### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
source ${prefix}/user.cfg
if [ -n "${GRUB2_PASSWORD}" ]; then
set superusers="root"
export superusers
password_pbkdf2 root ${GRUB2_PASSWORD}
fi
fi
### END /etc/grub.d/01_users ###
### BEGIN /etc/grub.d/10_linux ###
menuentry 'CentOS Linux (3.10.0-693.21.1.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.11.6.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod diskfilter
insmod mdraid1x
insmod xfs
set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718' 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
else
search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
fi
linuxefi /vmlinuz-3.10.0-693.21.1.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1 LANG=en_GB.UTF-8
initrdefi /initramfs-3.10.0-693.21.1.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-693.17.1.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.11.6.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod diskfilter
insmod mdraid1x
insmod xfs
set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718' 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
else
search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
fi
linuxefi /vmlinuz-3.10.0-693.17.1.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1 LANG=en_GB.UTF-8
initrdefi /initramfs-3.10.0-693.17.1.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-693.11.6.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.11.6.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod diskfilter
insmod mdraid1x
insmod xfs
set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718' 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
else
search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
fi
linuxefi /vmlinuz-3.10.0-693.11.6.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1
initrdefi /initramfs-3.10.0-693.11.6.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-693.5.2.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.5.2.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod diskfilter
insmod mdraid1x
insmod xfs
set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718' 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
else
search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
fi
linuxefi /vmlinuz-3.10.0-693.5.2.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1
initrdefi /initramfs-3.10.0-693.5.2.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-514.21.2.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-514.21.2.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod diskfilter
insmod mdraid1x
insmod xfs
set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718' 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
else
search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
fi
linuxefi /vmlinuz-3.10.0-514.21.2.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1
initrdefi /initramfs-3.10.0-514.21.2.el7.x86_64.img
}
menuentry 'CentOS Linux (0-rescue-290b6d3e8ddb4718bc34ccbb76080149) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0-rescue-290b6d3e8ddb4718bc34ccbb76080149-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
load_video
insmod gzio
insmod part_gpt
insmod diskfilter
insmod mdraid1x
insmod xfs
set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718' 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
else
search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
fi
linuxefi /vmlinuz-0-rescue-290b6d3e8ddb4718bc34ccbb76080149 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1
initrdefi /initramfs-0-rescue-290b6d3e8ddb4718bc34ccbb76080149.img
}
if [ "x$default" = 'CentOS Linux (3.10.0-693.11.6.el7.x86_64) 7 (Core)' ]; then default='Advanced options for CentOS Linux>CentOS Linux (3.10.0-693.11.6.el7.x86_64) 7 (Core)'; fi;
### END /etc/grub.d/10_linux ###
### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###
### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###
### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###
### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###
### BEGIN /etc/grub.d/41_custom ###
if [ -f ${config_directory}/custom.cfg ]; then
source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f $prefix/custom.cfg ]; then
source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
I'm sorry but I don't see anywhere "rhgb quite". What am I missing??
Re: CentOS: very slow system boot / sometimes hangs entirely
So, when you boot do you see thousands of messages flood past? Or nothing much at all? If it boots then look in /proc/cmdline for the kernel command line that it actually used to boot with.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke
Re: CentOS: very slow system boot / sometimes hangs entirely
Boot still looks like the video I linked in my initial post: https://videobin.org/+paj/ttu.html
Yes, I see the boot messages but I'm still in some kind of framebuffer mode...
Maybe I'm not clear on my goal: I see the boot messages, so no splash screen or some Windows like dancing around stuff. What I would like is plain old (is it 80x25?) textmode without any bells and whistles.
Yes, I see the boot messages but I'm still in some kind of framebuffer mode...
Code: Select all
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-693.21.1.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1 LANG=en_GB.UTF-8
Re: CentOS: very slow system boot / sometimes hangs entirely
The slow graphics update at boot may be a red herring. I would concentrate on those USB delays -- it really shouldn't take that long to scan for the USB devices. Try unplugging all your USB devices (even if it's only your keyboard) and booting, see if there is a difference. Disabling USB 3.0 from your BIOS or setting it into some sort of a legacy mode may also help. There's also a nice lsusb command that lists all the USB devices/hubs.