Slow disk IO after reboot

General support questions
Post Reply
qw42
Posts: 4
Joined: 2018/07/07 20:30:07

Slow disk IO after reboot

Post by qw42 » 2018/07/07 20:52:54

My desktop is vanilla Centos installed on regualr HDD (no raid). It is updated regularly via "yum update".

Problem symptoms: right after reboot system becomes horribly slow. It started happening recently. For example launching browser could take more than a minute. I suspect it is due to slow disk IO. For some strange reason everything goes to normal after some time.

This happens if I start using computer right after reboot. If I login into desktop but not use it for some time the slowdown not observed.

How I do diagnostics for such a problem?


Results of disk benchmark:
Avg read rate 455 MB/s
Avg write rate: 46.2
Avg acces time: 0.12 msec

Memory: free -m
total used free shared buff/cache available
Mem: 23886 1819 20728 25 1339 21652
Swap: 12031 0 12031

User avatar
TrevorH
Site Admin
Posts: 33216
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Slow disk IO after reboot

Post by TrevorH » 2018/07/07 21:03:16

Avg read rate 455 MB/s
Unless that's an SSD, your benchmark is broken. There is no HDD currently on sale that will do 455MB/s.

Do you have any errors in /var/log/messages? What does hdparm -tT /dev/sda report? How about smartctl -a /dev/sda ?
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Slow disk IO after reboot

Post by hunter86_bg » 2018/07/08 05:39:25

In my opinion, most probably the issue is caused by gnome file indexing.
To verify that install iotop and test it before a reboot (there was an issue with the first 7.5 kernel).
Once you verify that iotop is working - then reboot and check with iotop.

qw42
Posts: 4
Joined: 2018/07/07 20:30:07

Re: Slow disk IO after reboot

Post by qw42 » 2018/07/08 16:46:29

TrevorH wrote:
2018/07/07 21:03:16
Avg read rate 455 MB/s
Unless that's an SSD, your benchmark is broken. There is no HDD currently on sale that will do 455MB/s.

Do you have any errors in /var/log/messages? What does hdparm -tT /dev/sda report? How about smartctl -a /dev/sda ?
Your are right, those benchmark numbers are wrong. There are 2 more drives in my workstation and one those is SSD. On second try I've got numbers consistent with HDD. As for SMART report everything looks OK.

qw42
Posts: 4
Joined: 2018/07/07 20:30:07

Re: Slow disk IO after reboot

Post by qw42 » 2018/07/08 17:02:46

hunter86_bg wrote:
2018/07/08 05:39:25
In my opinion, most probably the issue is caused by gnome file indexing.
To verify that install iotop and test it before a reboot (there was an issue with the first 7.5 kernel).
Once you verify that iotop is working - then reboot and check with iotop.
I believe slowdown started after 7.5 upgrade. After reboot there is HDD activity going on for several minutes. According to iotop it is tracker. But the utilization percentage is far from 100%. During that time opening any application takes very long time. I disables indexing from UI tool and removed mlocate as well. But the problem doesn't go away. Btw, my home directory size is over 10GB in 150K files. Moving files outside helps. Also new account are not affected by slowdown. Looks like it is indexing large number of files causes the slowdown. How do I pinpoint and fix it?

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Slow disk IO after reboot

Post by hunter86_bg » 2018/07/09 04:34:42

It seems that disabling the tracker might break some stuff.
So ,maybe an exclude of the whole home (or portions of it) will reduce the stress.
Create a file like described here and test again.
It should be something like:

Code: Select all

 touch ~/.trackerignore 
By the way, mlocate indexing happens during the night, so you might reconsidered the removal. If your home is so large , a find will be very slow.

desertcat
Posts: 843
Joined: 2014/08/07 02:17:29
Location: Tucson, AZ

Re: Slow disk IO after reboot

Post by desertcat » 2018/07/09 08:23:13

qw42 wrote:
2018/07/08 17:02:46
hunter86_bg wrote:
2018/07/08 05:39:25
In my opinion, most probably the issue is caused by gnome file indexing.
To verify that install iotop and test it before a reboot (there was an issue with the first 7.5 kernel).
Once you verify that iotop is working - then reboot and check with iotop.
I believe slowdown started after 7.5 upgrade. After reboot there is HDD activity going on for several minutes. According to iotop it is tracker. But the utilization percentage is far from 100%. During that time opening any application takes very long time. I disables indexing from UI tool and removed mlocate as well. But the problem doesn't go away. Btw, my home directory size is over 10GB in 150K files. Moving files outside helps. Also new account are not affected by slowdown. Looks like it is indexing large number of files causes the slowdown. How do I pinpoint and fix it?
Hummmmmm. You say this is a HDD and not a SSD? The last time I saw something like this was just before my HDD died. When it is cold it takes like forever to come up but after it is running for a while everything is normal. Now might be a very good time to backup the entire disk. I'd run some disk diagnostics to check the health of the drive. If I were a betting man I think you are seeing the early signs of disk failure. If the drive is over 2-3 years and doing a lot of I/O stuff the odds go up. If you could clone the drive to another drive then do a complete scrub of the drive and formatting it might show you bad sectors etc. Just a guess. Backing up the drive however is cheap insurance in the event I am right. If I'm wrong?? Still cheap insurance.

qw42
Posts: 4
Joined: 2018/07/07 20:30:07

Re: Slow disk IO after reboot

Post by qw42 » 2018/07/09 13:51:35

For some reason disabling tracker from UI tool doesn't help. I wen't ahead and disabled tracker startup by editing /etc/xdg/autostart/tracker-*.
After that everything went back to normal. So far nothing is broken, at least everything I need works. I can get on with my work.

Post Reply