Very slow io after update

General support questions
Post Reply
testsoft
Posts: 2
Joined: 2018/11/15 20:13:22

Very slow io after update

Post by testsoft » 2018/11/15 20:27:42

Hello, plz help :)

CentOS was updated and now it very very slow. (CentOS Linux release 7.5.1804 / 3.10.0-862.14.4.el7.x86_64)

VM system: 2012 R2 Hyper V / i3-2120 / 16 GB /HDD WD5000AAKX-001CA 2Tb

I make some tests:
[root@bitrixcrm ~]# dd if=/dev/zero of=tempfile bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 38.7399 s, 27.7 MB/s
[root@bitrixcrm ~]# dd if=tempfile of=/dev/null bs=1M count=1024 iflag=direct
1073741824 bytes (1.1 GB) copied, 10.8025 s, 99.4 MB/s
---
If I add «nobarrier» to /etc/fstab
LABEL=bxRoot / ext4 defaults,noatime,nobarrier 1 1
LABEL=bxBoot /boot ext4 defaults,nobarrier 1 2
LABEL=bxSwap swap swap defaults,nobarrier 0 0
---
Tests result:
[root@bitrixcrm ~]# dd if=/dev/zero of=tempfile bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 15.0595 s, 71.3 MB/s
[root@bitrixcrm ~]# dd if=tempfile of=/dev/null bs=1M count=1024 iflag=direct
1073741824 bytes (1.1 GB) copied, 10.8574 s, 98.9 MB/s

Then I make clean VM system on the same server and run test:
---
[root@localhost ~]# dd if=/dev/zero of=tempfile bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 1.17313 s, 915 MB/s
[root@localhost ~]# dd if=tempfile of=/dev/null bs=1M count=1024 iflag=direct
1073741824 bytes (1.1 GB) copied, 0.918117 s, 1.2 GB/s
---

Add Info:
[root@bitrixcrm ~]# sdparm /dev/sda
/dev/sda: Msft Virtual Disk 1.0
mode sense (10): transport: Host_status=0x07 [DID_ERROR]
Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]
mode sense (10): transport: Host_status=0x07 [DID_ERROR]
Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]
Caching (SBC) mode page:
WCE 1 [cha: n, def: 1]
RCD 0 [cha: n, def: 0]
mode sense (10): transport: Host_status=0x07 [DID_ERROR]
Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]
mode sense (10): transport: Host_status=0x07 [DID_ERROR]
Driver_status=0x00 [DRIVER_OK, SUGGEST_OK]
---
How can I fix io? Tnx :)

lknls
Posts: 9
Joined: 2018/11/12 17:26:13
Contact:

Re: Very slow io after update

Post by lknls » 2018/11/16 10:36:58

did you tried iotop which can give you an idea about disk usage?

see https://www.cyberciti.biz/hardware/linu ... o-monitor/

testsoft
Posts: 2
Joined: 2018/11/15 20:13:22

Re: Very slow io after update

Post by testsoft » 2018/11/16 11:14:23

lknls wrote:
2018/11/16 10:36:58
did you tried iotop which can give you an idea about disk usage?

see https://www.cyberciti.biz/hardware/linu ... o-monitor/
nothing criminal, i think...
Attachments
on the run
on the run
iotop2.jpg (145.31 KiB) Viewed 1762 times
iotop info
iotop info
iotop.jpg (160.26 KiB) Viewed 1764 times

hunter86_bg
Posts: 2019
Joined: 2015/02/17 15:14:33
Location: Bulgaria
Contact:

Re: Very slow io after update

Post by hunter86_bg » 2018/11/16 15:34:57

Usually 'no barrier' should give you no more than 1-2% benefit.
What profile are you using in the VM?
I guess your disk scheduler is the default one (deadline). This is not suitable for VM - change it to 'noop' and test again.

If your Hypervisor's storage has battery-backed write cache -> keep the nobarrier.

Edit: Just noticed you use ext4. If you want better I/O performance -> consider switching to XFS.

User avatar
TrevorH
Site Admin
Posts: 33218
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Very slow io after update

Post by TrevorH » 2018/11/16 16:03:59

Also, what did you update from? If you haven't updated all year then you're probably only now seeing the "benefits" of the meltdown/spectre changes.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply