[SOLVED -> workaround] CentOS: very slow system boot / sometimes hangs entirely

General support questions
socrates
Posts: 17
Joined: 2017/10/24 11:28:06

[SOLVED -> workaround] CentOS: very slow system boot / sometimes hangs entirely

Post by socrates » 2018/01/19 13:44:40

Hi,

I have CentOS installed on a GigaByte N3150-D3V Board.

For some reason I can't figure out, the system boots *very* slowly and sometimes even hangs. Click here for a demonstration video of the slow boot.

My gut tells me it has something to do with the video framebuffer, so I tried to disable it by various setting (blacklist i915 in /etc/modprobe.d/blacklist.conf), following this Guide or this one for ubuntu.

But nothing helps neither as far as the slow boot is concerned nor as far as disabling framebuffer and the pinguins.

I ran a systemd-analyze plot > heBacula-boot-analysis.svg and apparently kernel (with 33 sec) and initrd (47 sec) take the most time to complete.

There is also the output of dmesg which I examined but there's also nothing that looks suspicious to me.


Anyone got any pointers as to what the cause of the problem could be?

Any help would be greatly appreciated!

Thanks in advance,

socrates
Last edited by socrates on 2018/03/15 08:23:55, edited 1 time in total.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS: very slow system boot / sometimes hangs entirely

Post by TrevorH » 2018/01/19 13:56:50

Almost all the big gaps in timestamps in your dmesg output seem to be to do with USB devices.

Your svg file tells me nothing. What's the text output from systemd-analyze blame instead?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

socrates
Posts: 17
Joined: 2017/10/24 11:28:06

Re: CentOS: very slow system boot / sometimes hangs entirely

Post by socrates » 2018/01/19 15:13:27

TrevorH wrote: Your svg file tells me nothing.
Well, the only thing I found interesting in the SVG file was in the upper left corner:

Code: Select all

Startup finished in 16.682s (firmwre) + 2.831s (loader) + 33.105s (kernel) + 47.164s (initrd) + 16.892s (userspace) = 1 min 56.677s
The rest is pretty pointless, I agree.

I didn't include the ouptut from systemd-analyze blame because the "heavy" part is not included here:

TrevorH wrote: What's the text output from systemd-analyze blame instead?

Code: Select all

9.289s systemd-vconsole-setup.service
          5.676s NetworkManager-wait-online.service
          3.669s mariadb.service
          3.598s kdump.service
          1.617s postfix.service
          1.424s network.service
          1.104s lvm2-monitor.service
          1.103s dev-mapper-vg_os\x2droot.device
           933ms firewalld.service
           672ms lvm2-pvscan@9:124.service
          9.289s systemd-vconsole-setup.service
          5.676s NetworkManager-wait-online.service
          3.669s mariadb.service
          3.598s kdump.service
          1.617s postfix.service
          1.424s network.service
          1.104s lvm2-monitor.service
          1.103s dev-mapper-vg_os\x2droot.device
           933ms firewalld.service
           672ms lvm2-pvscan@9:124.service
           619ms tuned.service
           494ms srv.mount
           429ms boot.mount
           373ms lvm2-pvscan@9:127.service
           319ms var.mount
           304ms auditd.service
           258ms systemd-logind.service
           222ms NetworkManager.service
           217ms tmp.mount
           213ms var-log.mount
           196ms systemd-journald.service
           179ms proc-fs-nfsd.mount
           178ms kmod-static-nodes.service
           175ms dev-hugepages.mount
           161ms systemd-tmpfiles-clean.service
           143ms polkit.service
           141ms systemd-udev-trigger.service
           140ms dev-mqueue.mount
           132ms sshd.service
           126ms systemd-tmpfiles-setup.service
           125ms rhel-import-state.service
           121ms var-log-audit.mount
           110ms home.mount
            99ms systemd-tmpfiles-setup-dev.service
            99ms systemd-udevd.service
            98ms rhel-readonly.service
            95ms yum-cron.service
            94ms mdmonitor.service
            90ms microcode.service
            88ms ntpd.service
            85ms gssproxy.service
            85ms rhel-dmesg.service
            83ms rpc-statd-notify.service
            83ms bacula-sd.service
            82ms systemd-readahead-done.service
            77ms dev-disk-by\x2duuid-82a895cc\x2d2acb\x2d45a3\x2d90df\x2d7be08a242de0.swap
            67ms bacula-fd.service
            59ms plymouth-start.service
            58ms bacula-dir.service
            57ms boot-efi_disk_SN_WCC4N7JCCFY9.mount
            54ms systemd-readahead-collect.service
            53ms systemd-readahead-replay.service
            53ms rsyslog.service
            50ms boot-efi_disk_SN_WCC4N7JCCZC9.mount
            50ms boot-efi.mount
            47ms sys-kernel-debug.mount
            47ms systemd-sysctl.service
            47ms boot-efi_disk_SN_WCC4N2ATJA1C.mount
            44ms systemd-remount-fs.service

TrevorH wrote:Almost all the big gaps in timestamps in your dmesg output seem to be to do with USB devices.
Right. The kernel even claims to have found a Logitech mouse:

Code: Select all

[   74.156209] input: PS/2 Logitech Mouse as /devices/platform/i8042/serio1/input/input5
But the only thing attached to the server are network, monitor and (cherry) keyboard.

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: CentOS: very slow system boot / sometimes hangs entirely

Post by hunter86_bg » 2018/01/21 20:22:25

Can you try with the rescue kernel and check the time needed to boot ?
Did you notice this behaviour in earlier kernels or it is observed with the latest one ?

socrates
Posts: 17
Joined: 2017/10/24 11:28:06

Re: CentOS: very slow system boot / sometimes hangs entirely / entirely disabling graphical boot?

Post by socrates » 2018/03/14 13:30:20

TL;DR: How do I completely disable the graphical boot and boot into old fashioned text mode?


Hi,
sorry for the delay in replying to this issue. Reason was that the server booted very slowly but it always reached the login -- so I could live with that.

Now it starts to hang on boot again. So I need to investigate further.

To answer the question: The phenomena is the same, regardless of using normal or rescue mode.

My gut tells me it has something to do with the graphical boot.

Can someone tell me how to disable this? I would like to have pure text mode login.
No penguins, no framebuffer, nothing at all.

I already tried to remove the "rhgb quiet" from the kernel line in grub - no success.
I tried #plymouth-set-default-theme to text or details but the graphical boot remains.

Thanks in advance,

soc

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS: very slow system boot / sometimes hangs entirely

Post by TrevorH » 2018/03/14 15:34:04

Apparently you didn't remove rhgb quiet when you thought you did.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

socrates
Posts: 17
Joined: 2017/10/24 11:28:06

Re: CentOS: very slow system boot / sometimes hangs entirely

Post by socrates » 2018/03/14 19:21:45

mmh... that would be strange ;)

Here's my /boot/efi/EFI/centos/grub.cfg

Code: Select all

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
set tuned_initrd=""
### END /etc/grub.d/00_tuned ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'CentOS Linux (3.10.0-693.21.1.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.11.6.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod xfs
        set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718'  99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        else
          search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        fi
        linuxefi /vmlinuz-3.10.0-693.21.1.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1 LANG=en_GB.UTF-8
        initrdefi /initramfs-3.10.0-693.21.1.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-693.17.1.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.11.6.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod xfs
        set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718'  99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        else
          search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        fi
        linuxefi /vmlinuz-3.10.0-693.17.1.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1 LANG=en_GB.UTF-8
        initrdefi /initramfs-3.10.0-693.17.1.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-693.11.6.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.11.6.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod xfs
        set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718'  99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        else
          search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        fi
        linuxefi /vmlinuz-3.10.0-693.11.6.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1
        initrdefi /initramfs-3.10.0-693.11.6.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-693.5.2.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.5.2.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod xfs
        set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718'  99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        else
          search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        fi
        linuxefi /vmlinuz-3.10.0-693.5.2.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1
        initrdefi /initramfs-3.10.0-693.5.2.el7.x86_64.img
}
menuentry 'CentOS Linux (3.10.0-514.21.2.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-514.21.2.el7.x86_64-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod xfs
        set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718'  99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        else
          search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        fi
        linuxefi /vmlinuz-3.10.0-514.21.2.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1
        initrdefi /initramfs-3.10.0-514.21.2.el7.x86_64.img
}
menuentry 'CentOS Linux (0-rescue-290b6d3e8ddb4718bc34ccbb76080149) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-0-rescue-290b6d3e8ddb4718bc34ccbb76080149-advanced-683b8e00-b582-4afc-9617-48d7badbd870' {
        load_video
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod xfs
        set root='mduuid/e5f2c39cb90d962ddb4deecdd4480718'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/e5f2c39cb90d962ddb4deecdd4480718'  99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        else
          search --no-floppy --fs-uuid --set=root 99fa8e01-6c95-4e49-b9fd-5cfd462bbd2b
        fi
        linuxefi /vmlinuz-0-rescue-290b6d3e8ddb4718bc34ccbb76080149 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1
        initrdefi /initramfs-0-rescue-290b6d3e8ddb4718bc34ccbb76080149.img
}
if [ "x$default" = 'CentOS Linux (3.10.0-693.11.6.el7.x86_64) 7 (Core)' ]; then default='Advanced options for CentOS Linux>CentOS Linux (3.10.0-693.11.6.el7.x86_64) 7 (Core)'; fi;
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
And here is the bootline from grub I boot via Ctrl+X. https://pasteboard.co/HbTYEm8.jpg

I'm sorry but I don't see anywhere "rhgb quite". What am I missing??

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: CentOS: very slow system boot / sometimes hangs entirely

Post by TrevorH » 2018/03/14 19:24:44

So, when you boot do you see thousands of messages flood past? Or nothing much at all? If it boots then look in /proc/cmdline for the kernel command line that it actually used to boot with.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

socrates
Posts: 17
Joined: 2017/10/24 11:28:06

Re: CentOS: very slow system boot / sometimes hangs entirely

Post by socrates » 2018/03/14 19:34:29

Boot still looks like the video I linked in my initial post: https://videobin.org/+paj/ttu.html

Yes, I see the boot messages but I'm still in some kind of framebuffer mode...

Code: Select all

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.10.0-693.21.1.el7.x86_64 root=/dev/mapper/vg_os-root ro crashkernel=auto rd.md.uuid=9d6aa368:eda8a315:5b4eecc2:e96a5c96 rd.lvm.lv=vg_os/root rd.md.uuid=e5f2c39c:b90d962d:db4deecd:d4480718 rd.md.uuid=0bba1c28:d8823172:509ac593:c6853088 audit=1 LANG=en_GB.UTF-8
Maybe I'm not clear on my goal: I see the boot messages, so no splash screen or some Windows like dancing around stuff. What I would like is plain old (is it 80x25?) textmode without any bells and whistles.

User avatar
avij
Retired Moderator
Posts: 3046
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: CentOS: very slow system boot / sometimes hangs entirely

Post by avij » 2018/03/14 21:18:58

The slow graphics update at boot may be a red herring. I would concentrate on those USB delays -- it really shouldn't take that long to scan for the USB devices. Try unplugging all your USB devices (even if it's only your keyboard) and booting, see if there is a difference. Disabling USB 3.0 from your BIOS or setting it into some sort of a legacy mode may also help. There's also a nice lsusb command that lists all the USB devices/hubs.

Post Reply