server freezes with this error kernel: ata1.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action

[gurulakshman]

HI,
i am using centos 5.5 with raid configuration.
The server freezes suddenly after some time
with the following error on the messages file

kernel: ata1.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x0

The server only reboots when i push reset button


my hard disk details are


Model Number: ST500NM0011
Serial Number: Z1M02LT7
Firmware Revision: SN02


Let us know if any one knows the solution

[gerald_clark]

Welcome to CentOS.
New members need to read
http://www.centos.org/modules/newbb/viewforum.php?forum=47

It looks like a failing drive, but you have not given any useful information.
What kind of RAID are you using?

[TrevorH]

There are almost certainly other messages before and after the one that you quoted and they often contain useful information in diagnosing if the problem is software, hardware or cable.

[gurulakshman]

It has RAID 1 configuration with two segate hard disks

the model number of hard disks are

Model Number: ST500NM0011
Serial Number: Z1M02LT7
Firmware Revision: SN02


I am also attaching the message log file as attachment

I can run any command or i can give any more details for my server

[gurulakshman]

Attaching messages log file in zip format

update getting error message while attaching the file

[TrevorH]

If the log messages are large then please use somewhere like pastebin to hold them and post the URL here

[r_hartman]

[url=https://bugzilla.redhat.com/show_bug.cgi?id=404851]This bugzilla entry[/url] suggests turning off Native Command Queueing on the drive:
[code]# echo 1 > /sys/block/sdX/device/queue_depth[/code]
Seems to be related to WD drives; not sure if and how this would apply to your situation, as you mention Seagate.
While the bugzilla refers to RHEL5.3, nothing got patched as it was decided that [i]apparently Western Digital don't have a clue how to do NCQ[/i].

There's another buzilla [url=https://bugzilla.redhat.com/show_bug.cgi?id=462425]here[/url], for Fedora9, which I think RHEL5 (and thus CentOS5) is based on, related to software RAID5 with Seagate drives (on a Marvell chipset). It mentions that CentOS suffers this as well, but less frequent with newer kernels; [quote]I switched from Fedora to CentOS a couple of years ago because I needed GFS2
support on my cluster nodes, but got the same error frequently initially.
However, the frequency has gone down with every kernel update over the years,
and hardly ever occurs with the current CentOS kernel (2.6.18-238.9.1.el5), but
still does now and then (say once every two month's on one of the cluster
nodes).[/quote]
As 5.5 is obsolete, I strongly suggest you upgrade to 5.7 and the latest kernel (2.6.18-274.7.1.el5).

Finally, you may want to check if drive firmware is upgradable, and whether there is a later firmware.

[gurulakshman]

Please find the messages log at http://hyperwebenable.com/log/messages

also i am using hard ware RAID 1

smartctl logs are


[root@hyperwebenable ~]# smartctl --all /dev/sda
smartctl version 5.38 [x86_64-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model: ST500NM0011
Serial Number: Z1M02LT7
Firmware Version: SN02
User Capacity: 500,107,862,016 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Tue Nov 29 16:20:27 2011 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 609) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 77) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x10bd) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 070 063 044 Pre-fail Always - 11071375
3 Spin_Up_Time 0x0003 098 097 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 26
5 Reallocated_Sector_Ct 0x0033 056 056 036 Pre-fail Always - 1829
7 Seek_Error_Rate 0x000f 064 060 030 Pre-fail Always - 68762714066
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 896
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 26
184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Unknown_Attribute 0x0032 100 099 000 Old_age Always - 4295032833
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 067 050 045 Old_age Always - 33 (Lifetime Min/Max 28/34)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 2
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 21
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 40
194 Temperature_Celsius 0x0022 033 050 000 Old_age Always - 33 (0 22 0 0)
195 Hardware_ECC_Recovered 0x001a 106 099 000 Old_age Always - 11071375
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0

SMART Error Log Version: 1
ATA Error Count: 9 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 9 occurred at disk power-on lifetime: 809 hours (33 days + 17 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
10 51 00 ff ff ff 0f

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 13:08:43.290 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 13:08:43.221 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 13:08:43.182 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 13:08:43.163 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 13:08:43.139 WRITE FPDMA QUEUED

Error 8 occurred at disk power-on lifetime: 809 hours (33 days + 17 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
10 51 00 0d 3f bf 09

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 13:08:26.538 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 13:08:26.197 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 13:08:26.189 READ FPDMA QUEUED
60 00 10 ff ff ff 4f 00 13:08:26.148 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 13:08:26.080 READ FPDMA QUEUED

Error 7 occurred at disk power-on lifetime: 795 hours (33 days + 3 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
10 51 00 ff ff ff 0f

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 48 ff ff ff 4f 00 2d+13:15:47.175 WRITE FPDMA QUEUED
61 00 e8 ff ff ff 4f 00 2d+13:15:47.165 WRITE FPDMA QUEUED
61 00 c0 ff ff ff 4f 00 2d+13:15:46.834 WRITE FPDMA QUEUED
61 00 30 ff ff ff 4f 00 2d+13:15:46.722 WRITE FPDMA QUEUED
61 00 28 ff ff ff 4f 00 2d+13:15:45.020 WRITE FPDMA QUEUED

Error 6 occurred at disk power-on lifetime: 795 hours (33 days + 3 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
10 51 00 ff ff ff 0f

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 30 ff ff ff 4f 00 2d+13:15:29.801 WRITE FPDMA QUEUED
61 00 10 ff ff ff 4f 00 2d+13:15:29.612 WRITE FPDMA QUEUED
61 00 20 ff ff ff 4f 00 2d+13:15:29.162 WRITE FPDMA QUEUED
61 00 98 ff ff ff 4f 00 2d+13:15:26.841 WRITE FPDMA QUEUED
61 00 b0 ff ff ff 4f 00 2d+13:15:26.582 WRITE FPDMA QUEUED

Error 5 occurred at disk power-on lifetime: 733 hours (30 days + 13 hours)
When the command that caused the error occurred, the device was active or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
10 51 00 1d 34 8b 0a

Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 08:48:53.298 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 08:48:52.958 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 08:48:50.501 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 08:48:50.367 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 08:48:49.558 WRITE FPDMA QUEUED

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 896 -

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

[root@hyperwebenable ~]#

[gerald_clark]

Smells like fakeraid to me, not hardware RAID.
The clues: Marvel chipset and smartclt can see the drive.

[gurulakshman]

1. How can u check this is fake RAID
2. Also is it problem with rad disk or with sata cables

Please find the messages log at http://hyperwebenable.com/log/messages

i have run this command dmraid -r
[root@hyperwebenable ~]# dmraid -r
/dev/sda: ddf1, ".ddf1_disks", GROUP, ok, 976494592 sectors, data@ 0
/dev/sdb: ddf1, ".ddf1_disks", GROUP, ok, 976494592 sectors, data@ 0