XFS - blk_update_request: I/O error on Hardware RAID5

Issues related to applications and software problems
Post Reply
foolbar
Posts: 7
Joined: 2017/06/03 08:41:53

XFS - blk_update_request: I/O error on Hardware RAID5

Post by foolbar » 2017/06/03 08:54:08

Hey there,

so I've been running CentOS Servers for quite a while, mostly for personal use, and now I'm encountering a frequently reoccurring issue on my owncloud/nextcloud Virtual Machine.

I have a Hyper-V Host (yes, I know - no opportunity atm to change the Hypervisor) with a Hardware RAID 5 (HP Smart Array P410) running a load of Virtual Machines (about 20 or 30) and only one of them, namely my nextcloud VM is frequently stopping with the following error message:

Code: Select all

blk_update_request: I/O error, dev sda, sector 23600528
XFS (sda3): metadata I/O error: block 0x1187590 ("xlog_iodone") error 5 numblks 64
XFS (sda3): Log I/O Error Detected. Shutting down filesystem
XFS (sda3): Please unmount the filesystem and rectify the problem(s)
sda3 is the root partition. After a reboot, the VM works again like a charm until it stops again after about a day.
I already did filesystem checks but they didn't find anything. Also the RAID controller (newest firmware installed) doesn't mark any drive as defective.

One more important information: I'm doing backups using Virtual Machine Snapshots and it may be, that the VM is always crashing during backup. Either because it is then trying to read the problematic sector or because of some I/O stall caused by Snapshot creation/deletion.

Anyone got an idea regarding this? Might there be some I/O timeout which kicks in? I'm using Veeam for the backup and "Application-aware processing" is disabled for this VM, as this feature mostly doesn't play very nice with Unix machines (if even compatible).

Thanks!

edit: sorry, I forgot some spec details:

Code: Select all

[root@owncloud ~]# uname -a
Linux owncloud 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@owncloud ~]# cat /etc/centos-release
CentOS Linux release 7.3.1611 (Core)

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: XFS - blk_update_request: I/O error on Hardware RAID5

Post by TrevorH » 2017/06/03 12:12:44

Run yum update and make sure the latest kernel is installed and then reboot into it.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

foolbar
Posts: 7
Joined: 2017/06/03 08:41:53

Re: XFS - blk_update_request: I/O error on Hardware RAID5

Post by foolbar » 2017/06/03 12:51:16

Thanks, already did that two hours ago but then yum update overwrote my nextcloud with owncloud because I forgot to remove it from yum first, before I migrated from owncloud to nextcloud.

So I will have to clean that up first (already restored from backup) and then I'll do the yum update again.

Thanks again, I'll let you know if it helped.

foolbar
Posts: 7
Joined: 2017/06/03 08:41:53

Re: XFS - blk_update_request: I/O error on Hardware RAID5

Post by foolbar » 2017/06/22 19:48:44

update: this solved the issue! thanks!

Post Reply