Help on doing FSCK sucessfully

General support questions including new installations

Help on doing FSCK sucessfully

Postby mitchykoi » 2011/05/24 02:09:37

Gurus,

Got this problem when starting my db having 'Linux Error: 30 : Read only Filesystem'

Need help on doing FSCK, tried running it since yesterday afternoon but it seems to be hanging forever? Am I doing the correct method? Or this is just normal for a 1TB disk? Are there other ways to execute fsck? And also how do I see valuable results logs of FSCK command?

[root@Apmnc-DBServer ~]# uname -a
Linux Apmnc-DBServer 2.6.18-194.8.1.el5 #1 SMP Thu Jul 1 19:04:48 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@Apmnc-DBServer ~]# rpm -a kernel\* | sort
RPM version 4.4.2.3
Copyright (C) 1998-2002 - Red Hat, Inc.

[root@Apmnc-DBServer ~]# more /etc/fstab
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0


[root@Apmnc-DBServer ~]# fsck -yvf /dev/VolGroup00/LogVol00
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 183369735 has zero dtime. Fix? yes

Inodes that were part of a corrupted orphan linked list found. Fix? yes

Inode 183369737 was part of the orphaned inode list. FIXED.
Inode 183369738 was part of the orphaned inode list. FIXED.
Inode 183369742 was part of the orphaned inode list. FIXED.
Inode 183369745 was part of the orphaned inode list. FIXED.


Thanks.

Mitch
mitchykoi
 
Posts: 3
Joined: 2011/05/23 02:35:25

Re: Help on doing FSCK sucessfully

Postby grom-ph » 2011/05/24 05:35:36

It's a good practice to do the fsck on unmounted partition. Refer to this post [https://www.centos.org/modules/newbb/viewtopic.php?topic_id=31492&forum=37] for has been suggested. As to the fsck log, follow the alternative approach on this thread [https://www.centos.org/modules/newbb/viewtopic.php?viewmode=threaded&order=ASC&topic_id=22398&forum=37&move=next&topic_time=1254021382]
grom-ph
 
Posts: 5
Joined: 2011/05/23 08:41:30

Re: Help on doing FSCK sucessfully

Postby mitchykoi » 2011/05/24 08:22:35

Hi There.

If File system was on read-only mode, so that does still mean I should not FSCK? What would be the difference on running against unmounted vs read-only file systems?


[root@Apmnc-DBServer ~]# cd /

[root@Apmnc-DBServer /]# touch x.txt
touch: cannot touch `x.txt': Read-only file system

[root@Apmnc-DBServer /]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 ro,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
/etc/auto.misc /misc autofs rw,fd=7,pgrp=3539,timeout=300,minproto=5,maxproto=5,indirect 0 0
-hosts /net autofs rw,fd=13,pgrp=3539,timeout=300,minproto=5,maxproto=5,indirect 0 0
mitchykoi
 
Posts: 3
Joined: 2011/05/23 02:35:25

Help on doing FSCK sucessfully

Postby pschaff » 2011/05/25 20:47:51

Welcome to the CentOS fora. Reading FAQ & Readme First is recommended for new users.

Given that the root filesystem is already read-only you are probably not doing additional harm with the fsck, but that does indicate a serious problem. An fsck on a large filesystem may take a long time. Hopefully you have recent backups of your database and any other valuable files as you may have hardware problems with the disk. Checking the SMART status with smartctl would be good to do.

Your kernel version indicates that you may still be running CentOS 5.5 and are probably in need of an update.
pschaff
Retired Moderator
 
Posts: 18277
Joined: 2006/12/13 20:15:34
Location: Tidewater, Virginia, North America

Re: Help on doing FSCK sucessfully

Postby mitchykoi » 2011/05/26 02:21:27

Tried doing SMARTCTL, below is the result. Whats does it mean? Also, how would you do a kernel update?


[root@Apmnc-DBServer ~]# smartctl -d cciss,2 -a /dev/cciss/c0d0p2
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Serial number: 9QM4B00S
Device type: disk
Local Time is: Tue May 24 12:25:24 2011 CST
Device supports SMART and is Enabled
Temperature Warning Disabled or Not Supported
Log Sense failed, IE page [scsi response fails sanity test]
Read defect list: asked for grown list but didn't get it

Error Counter logging not supported
Device does not support Self Test logging
mitchykoi
 
Posts: 3
Joined: 2011/05/23 02:35:25

Re: Help on doing FSCK sucessfully

Postby r_hartman » 2011/05/26 05:27:05

Judging from the 'cciss'-bit it would appear your disks are on an HP SmartArray (just guessing, I only have HP boxes here), which begs the question whether you run that in RAID or not.
Not sure you can request SMART data from hardware raid, or even from a SmartArray. I had a go on one of my boxes but got 'A mandatory SMART command failed' all the time.

If my assessment that you're on an HP SMartArray (ProLiant server) is correct, I suggest you have a look at DMESG and search for any complaints about the cciss driver; you may need to update the SmartArray firmware. After upgrading a couple of problem-free boxes from RHEL 5.2 / 5.4 to RHEL 5.6 I had three of them failing (after some hours or days) on cciss I/O; writing to the disks timed out, causing infinite hang . One of these boxes had a line in DMESG
Code: Select all
cciss 0000:04:01.0: Controller reports max supported commands of 0, an obvious lie. Using 16. Ensure that firmware is up to date.
but the others did not. All of them were fixed by updating the SmartArray firmware to the latest level, though.

Somewhere during startup you must see messages as to why the filesystem is placed in read-only mode. Although your symptoms are not the same (and you're not on 5.6 but 5.5), you may be suffering from similar firmware issues, and it doesn't hurt to check/try. You may also get some indications when you go into the SmartArray's BIOS on boot, and see if it has any issues with drive statuses.

However, if you haven't done so yet, before anything else, my first line of action would be to backup any valuable data from the still readable disk, as it may still be the disk that's failing.

I found that HP SmartAray identifications are very confusing, and the same controller can be referred to under at least three different names. Updating the firmware from a running box (running the downloaded .scexe program) failed, so I had to boot from the SmartUpdate iso, through ILO Virtual Media, as getting physical access to the box is somewhat of a hassle. SmartUpdate ISOs (firmware cd's) can be downloaded from the HP support site.

Hope this helps.
r_hartman
 
Posts: 701
Joined: 2009/03/23 15:08:11
Location: Netherlands


Return to CentOS 5 - General Support

Who is online

Users browsing this forum: No registered users and 9 guests