Centos VMs with IO failure on root

Installing, Configuring, Troubleshooting server daemons such as Web and Mail

Centos VMs with IO failure on root

Postby bobby320 » 2012/01/18 22:16:47

Hello,

I am running into this issues, few VMWare VM's that are running with RHEL 5.6 and Centos 5.7, are having read write errors on the root file system. This is noticed on all vm's. After looking to log files this what I noticed.

#touch test.txt
touch: cannot touch `test.txt': Read-only file system.
#

after this we have reboot the server, during booting up, fsck is fixing the issue, in one case we had to rebuild whole VM as did keep on panic, it did'nt as prompt for fsck.

some of the log message error that I noticed.

Jan 18 13:25:01 Hostname1 kernel: EXT3-fs warning (device dm-0): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure
Jan 18 13:25:01 Hostname1 kernel: EXT3-fs warning (device dm-0): ext3_clear_journal_err: Marking fs in need of filesystem check.

Thanks,
bobby320
 
Posts: 5
Joined: 2012/01/17 02:46:16

Centos VMs with IO failure on root

Postby pschaff » 2012/01/18 22:50:03

I don't understand how rebooting the server is going to fsck and fix guest disk filesystems, unless you are using raw devices, and also mounting them on the host - a big no-no.
pschaff
Retired Moderator
 
Posts: 18277
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America

Re: Centos VMs with IO failure on root

Postby bobby320 » 2012/01/20 16:36:44

Pschaff,

I mean to say, When I notice this read/write error message, rebooted the server. While server booting up, due to inconsistency on the root filesystem it prompted me to run the fsck manually on it. This is how I fixed the read/write error by running fsck. But not sure why is this causing this issue.

Thanks,
bobby320
 
Posts: 5
Joined: 2012/01/17 02:46:16

Re: Centos VMs with IO failure on root

Postby pschaff » 2012/01/20 17:59:26

I'd be looking at the logs for errors on the disks, and/or checking SMART status.
pschaff
Retired Moderator
 
Posts: 18277
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America

Re: Centos VMs with IO failure on root

Postby TrevorH » 2012/01/20 18:57:51

rebooted the server.


Do you actually mean you rebooted the server - i.e. the host? Or the guest?
User avatar
TrevorH
Forum Moderator
 
Posts: 9103
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Centos VMs with IO failure on root

Postby bobby320 » 2012/01/20 21:17:46

Pschaff, It stopped updating the /var/log/messages file as the root file system became readonly.

Only error messages that I noticed in the log file, after rebooting is listed in my first post.


TrevorH, I rebooted the virtual machine, which is centos 5.6[server]
bobby320
 
Posts: 5
Joined: 2012/01/17 02:46:16

Re: Centos VMs with IO failure on root

Postby pschaff » 2012/01/20 21:23:48

OK - thanks for clearing up that confusion. Note that 5.6 is an obsolete/unsupported release. CentOS-5.7 is the current release. See the CentOS 5.7 Release Notes for details. By not updating you are implicitly accepting that you will live with numerous bugs and security issues (and associated known exploits) that have subsequently been fixed.

What VMware product and version are you using?
pschaff
Retired Moderator
 
Posts: 18277
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America

Re: Centos VMs with IO failure on root

Postby bobby320 » 2012/01/30 17:18:08

Phil,

ESX version is 4.1,

This seems to be Kernel bug, as per this vmware link
"http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=51306&sliceId=1&docTypeID=DT_KB_1_1&dialogID=283139327 &stateId=0%200%20283141315"

it is suggested to patch the server, which are made against kernel version 2.6.18-8.1.4.el5 from kernel-2.6.9-55.EL


Code:
# uname -r
2.6.18-274.3.1.el5
# more /etc/redhat-release
CentOS release 5.7 (Final)

I am currently CentOS 5.7, and kernel version is listed above, not sure weather this patch is included in part of installation. Can someone please suggest.

Thanks,
bobby320
 
Posts: 5
Joined: 2012/01/17 02:46:16

Re: Centos VMs with IO failure on root

Postby TrevorH » 2012/01/30 20:53:21

Yes, the bug report you reference says "For RHEL5, upgrade to Update 1". You are on Update 7 so this is fixed.
User avatar
TrevorH
Forum Moderator
 
Posts: 9103
Joined: 2009/09/24 10:40:56
Location: Brighton, UK


Return to CentOS 5 - Server Support

Who is online

Users browsing this forum: No registered users and 2 guests