kernel panic occurs once every 1-2 weeks

Support for the other architectures (X86_64, s390(x) and PowerPC)
Post Reply
cygnus-x1
Posts: 49
Joined: 2007/10/12 17:51:38

kernel panic occurs once every 1-2 weeks

Post by cygnus-x1 » 2007/10/12 17:59:21

We are running CentOS 4.4 (64 bit) kernel 2.6.9-34.ELsmp. There are 2 RAID arrays on the box 1 72 gig array root partition( / ) at RAID level 1 and a 1.8 Tera byte partition at RAID level 5.

The SCSI and RAID controllers in the box are:

SCSI
03:01.0 SCSI storage controller: Adaptec AIC-7892A U160/m (rev 02)
Subsystem: Adaptec 29160 Ultra160 SCSI
Controller

RAID
03:02.0 RAID bus controller: Adaptec AAC-RAID (Rocket) (rev 02)
Subsystem: Adaptec ASR-2820SA

Approximately every 1-2 weeks we get a kernel panic. Originally we were having drive issues but have since swapped out to a newer drives and according to the RAID controller the arrays are optimal now but the kernel panic keeps occurring.

The box does a lot of I/O as it is a production web backup machine.

I have not yet had the chance to setup a netdump server so I don't have the whole console but I do have a screen shot of the final lines. It appears to be a syncing issue within ext3. BTW LVM is in use if that is important.

I have attached an image of the console screen shot

I posted this in LinuxQuestions.org and someone suggested I look into a CentOS 64 bug with SATA in pre 2.6.10 kernels
Someone also suggested using sar and vmstat tools but I am pretty new at this and don't know how to use them effectively. vmstat looks fine but I can't look at it when the kernel has panicked.

Thanks

cygnus-x1
Posts: 49
Joined: 2007/10/12 17:51:38

Re: kernel panic occurs once every 1-2 weeks

Post by cygnus-x1 » 2007/10/12 18:03:00

I can't see my attachment. Here is a photobucket link:

http://i211.photobucket.com/albums/bb183/cygnus-x1_2007/capestor-console.jpg

User avatar
toracat
Forum Moderator
Posts: 7386
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

kernel panic occurs once every 1-2 weeks

Post by toracat » 2007/10/12 19:22:37

Do you have anything from /var/log/messages ? kernel 2.6.9-34 is very old. You should seriously consider updating to the latest version because there are security issues with older kernels. I also suggest you file a bug report at http://bugs.centos.org with all the details.

cygnus-x1
Posts: 49
Joined: 2007/10/12 17:51:38

Re: kernel panic occurs once every 1-2 weeks

Post by cygnus-x1 » 2007/10/12 19:44:07

/var/log/messages shows nothing. The messages just stop until the machine is restarted.

I think I will upgrade the kernel to see if that fixes it. I am a bit nervous about this stuff because I had to build some software on the box and I would hate for an update to bring down something that would undo my changes. Hopefully I can get away with just the kernel.

I have never upgraded the kernel on CentOS before. Can the upgrade take place while it is running and then reboot to start the new kernel?

If I were to file a bug report wouldn't they want the complete console dump? I need to get the netdump server setup (waiting on someone else).

Any info/tips would be helpful. I am not a true admin only doing it because I am the only one that knows Linux here.

thanks

Doug

pjwelsh
Posts: 2598
Joined: 2007/01/07 02:18:02
Location: Central IL USA

Re: kernel panic occurs once every 1-2 weeks

Post by pjwelsh » 2007/10/12 20:21:16

All of the CentOS 4.4 -> 4.5 updates that I have done were problem free. I have done various AMD Tyan + 3ware, Intel + 3ware and Dell +PERC (4/5) + EMC FC etc. I would expect your update to go without issue... but backup up the backup of coarse.

Heck even the 4.5 -> 5.0 DVD updates have only show a couple of easily solved issues (but most are fresh 5.0 installs).

cygnus-x1
Posts: 49
Joined: 2007/10/12 17:51:38

Re: kernel panic occurs once every 1-2 weeks

Post by cygnus-x1 » 2007/10/12 20:32:40

Okay now I understand a little better. I am a Slackware user at home and build most of my stuff usually from source and don't rely oin packages. The RPM/Yum stuff is new to me. I tried looking at what updates were available to me for 4.4 and it did not go higher than 2.6.9-55.0.9.EL so I did not see how to upgrade to 2.6.10 or higher. Now I see that you mean upgrade as in 4.4 -> 4.5. This I have never done before. Is there a doc/tutorial that covers an upgrade like this that you recommend ? I would be much appreciative.

I assume the upgrade will leave data and such in its place.

thanks

Doug

User avatar
toracat
Forum Moderator
Posts: 7386
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: kernel panic occurs once every 1-2 weeks

Post by toracat » 2007/10/12 21:22:55

Well, 4.4 -> 4.5 is not really an "upgrade", maybe an "update". If you have the daily yum update running, you would be running 4.5 by now. The latest kernel for CentOS 4 is indeed 2.6.9-55.0.9.EL at this moment. There will be no 2.6.10 in CentOS 4 during its life span. Kernel bugs will be backported without changing the release number (2.6.9).

Going from CentOS 4 to 5 will be an upgrade. Depending on what applications you are running, it can be trouble-free or it can be troublesome. You can see the procedure at:

http://wiki.centos.org/HowTos/MigrationGuide/ServerCD_4.4_to_5

cygnus-x1
Posts: 49
Joined: 2007/10/12 17:51:38

Re: kernel panic occurs once every 1-2 weeks

Post by cygnus-x1 » 2007/10/13 00:34:23

So does that mean that the bug I heard about for SATA and 2.6.9 on 64 bit may have be fixed and is available as a higher number 2.6.9 kernel? What I am asking is if the bug fixes present for the 2.6.10 + kernels are are also fixed in 2.6.9 ?

With the evidence I have supplied thus far (not a lot I know) what would be your recommendation ? Yum was never done because of a firewall believe it or not. Someone is a bit overly paranoid sometimes here I am affraid. I can't perform DNS queries from the box. =( I should be able to get it opened for a while but the hole will eventually close up again.

thanks

Doug

User avatar
toracat
Forum Moderator
Posts: 7386
Joined: 2006/09/03 16:37:24
Location: California, US
Contact:

Re: kernel panic occurs once every 1-2 weeks

Post by toracat » 2007/10/13 01:56:48

I can answer only in general terms...

CentOS aims at 100% binary compatible with upstream (RHEL). This means that any bugs in the upstream kernel will be in the CentOS kernel as well. This also means that, to get any fix in the CentOS kernel, the upstream kernel must be fixed first. As far as security patches are concerned, RedHat backports them to their kernel although the release version (2.6.9 for RHEL4 and 2.6.18 for RHEL5) remains the same. So, CentOS kernels get the same patches. When it comes to enhancements such as support for new hardware, not everything in newer kernels is implemented. I don't know exactly how they decide as to what gets included and what does not, but it could be based on popularity of the hardware, or severity of the problem, or simply the demand from users. One way of pushing the upstream is to file a request through their bugzilla.

One thing to note here is that, while CentOS strictly keeps its 100% compatibility policy, it does offer new features and updates. CentOSPlus is such an example. So, you can ask CentOS team to provide driver support for newer hardware that is not included in RHEL. Sending an e-mail to the centos-devel mailing list and asking about the possibility for such support may be one way to contact the developers. It is not rare at all that someone in the community offers help by supplying needed drivers.

Post Reply

Return to “CentOS 4 - X86_64,s390(x) and PowerPC Support”