Centos VMs with IO failure on root

Installing, Configuring, Troubleshooting server daemons such as Web and Mail
Post Reply
bobby320
Posts: 5
Joined: 2012/01/17 02:46:16

Centos VMs with IO failure on root

Post by bobby320 » 2012/01/18 22:16:47

Hello,

I am running into this issues, few VMWare VM's that are running with RHEL 5.6 and Centos 5.7, are having read write errors on the root file system. This is noticed on all vm's. After looking to log files this what I noticed.

#touch test.txt
touch: cannot touch `test.txt': Read-only file system.
#

after this we have reboot the server, during booting up, fsck is fixing the issue, in one case we had to rebuild whole VM as did keep on panic, it did'nt as prompt for fsck.

some of the log message error that I noticed.

Jan 18 13:25:01 Hostname1 kernel: EXT3-fs warning (device dm-0): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Jan 18 13:25:01 Hostname1 kernel: EXT3-fs warning (device dm-0): ext3_clear_journal_err: Marking fs in need of filesystem check.

Thanks,

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Centos VMs with IO failure on root

Post by pschaff » 2012/01/18 22:50:03

I don't understand how rebooting the server is going to fsck and fix guest disk filesystems, unless you are using raw devices, and also mounting them on the host - a big no-no.

bobby320
Posts: 5
Joined: 2012/01/17 02:46:16

Re: Centos VMs with IO failure on root

Post by bobby320 » 2012/01/20 16:36:44

Pschaff,

I mean to say, When I notice this read/write error message, rebooted the server. While server booting up, due to inconsistency on the root filesystem it prompted me to run the fsck manually on it. This is how I fixed the read/write error by running fsck. But not sure why is this causing this issue.

Thanks,

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: Centos VMs with IO failure on root

Post by pschaff » 2012/01/20 17:59:26

I'd be looking at the logs for errors on the disks, and/or checking SMART status.

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Centos VMs with IO failure on root

Post by TrevorH » 2012/01/20 18:57:51

[quote]
rebooted the server.
[/quote]

Do you actually mean you rebooted the server - i.e. the host? Or the guest?

bobby320
Posts: 5
Joined: 2012/01/17 02:46:16

Re: Centos VMs with IO failure on root

Post by bobby320 » 2012/01/20 21:17:46

Pschaff, It stopped updating the /var/log/messages file as the root file system became readonly.

Only error messages that I noticed in the log file, after rebooting is listed in my first post.


TrevorH, I rebooted the virtual machine, which is centos 5.6[server]

pschaff
Retired Moderator
Posts: 18276
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America
Contact:

Re: Centos VMs with IO failure on root

Post by pschaff » 2012/01/20 21:23:48

OK - thanks for clearing up that confusion. Note that 5.6 is an obsolete/unsupported release. CentOS-5.7 is the current release. See the [url=http://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.7]CentOS 5.7 Release Notes[/url] for details. By not updating you are implicitly accepting that you will live with numerous bugs and security issues (and associated known exploits) that have subsequently been fixed.

What VMware product and version are you using?

bobby320
Posts: 5
Joined: 2012/01/17 02:46:16

Re: Centos VMs with IO failure on root

Post by bobby320 » 2012/01/30 17:18:08

Phil,

ESX version is 4.1,

This seems to be Kernel bug, as per this vmware link
"http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=51306&sliceId=1&docTypeID=DT_KB_1_1&dialogID=283139327 &stateId=0%200%20283141315"

it is suggested to patch the server, which are made against kernel version 2.6.18-8.1.4.el5 from kernel-2.6.9-55.EL


Code:
# uname -r
2.6.18-274.3.1.el5
# more /etc/redhat-release
CentOS release 5.7 (Final)

I am currently CentOS 5.7, and kernel version is listed above, not sure weather this patch is included in part of installation. Can someone please suggest.

Thanks,

User avatar
TrevorH
Site Admin
Posts: 33202
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Centos VMs with IO failure on root

Post by TrevorH » 2012/01/30 20:53:21

Yes, the bug report you reference says "For RHEL5, upgrade to Update 1". You are on Update 7 so this is fixed.

Post Reply