kernel 3.10.0-862.6.3 hanging when loading raid

Issues related to applications and software problems
q7joey
Posts: 11
Joined: 2018/07/04 05:34:19

kernel 3.10.0-862.6.3 hanging when loading raid

Post by q7joey » 2018/07/04 05:39:52

just upgraded to the latest kernel and the server hung during reboot. the last thing on the console was trying to load the raid driver. this server has dual path sas to 12 drives and lots of lvm striped raid on top of that.

mainly posting to see if anyone else is seeing problems with the latest kernel hanging in a similar fashion.

User avatar
avij
Forum Moderator
Posts: 2682
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by avij » 2018/07/04 10:56:34

What kind of RAID controller? Do you use CentOS drivers for that, or manufacturer's drivers, or some other 3rd party drivers?

q7joey
Posts: 11
Joined: 2018/07/04 05:34:19

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by q7joey » 2018/07/04 14:08:55

sorry should have said this is all software raid. it hangs after loading the software raid module.

tomkep
Posts: 17
Joined: 2018/04/25 13:30:50

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by tomkep » 2018/07/04 14:24:31

I can see this problem as well on three DELL Precision systems (one T7400, the other two T7500). Software raid1, HW is different between them (some kind of SAS controller, I can dig the details if needed) but probably irrelevant. This does not reproduce on Lenovo laptop without SW raid.

Luckily I have a screenshot from one of them:

Image
Attachments
panic.png
panic.png (197.07 KiB) Viewed 455 times

q7joey
Posts: 11
Joined: 2018/07/04 05:34:19

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by q7joey » 2018/07/04 15:01:02

glad to see i'm not the only one. i don't get a panic, just a hang. rolled back to 862.3.3 and things are fine there. this is a production box, but just used for backups so i can probably do some more diagnostics after the 4th.

User avatar
avij
Forum Moderator
Posts: 2682
Joined: 2010/12/01 19:25:52
Location: Helsinki, Finland
Contact:

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by avij » 2018/07/04 20:48:20

Interesting. I'm unable to reproduce this bug myself, though. I set up a VM in VirtualBox with two disks, / and /boot as RAID1 devices (no LVM) and a small swap partition (non-RAID) on both devices. Works ok with 3.10.0-862.6.3. I wonder which specific configuration combination triggers this bug.

tomkep
Posts: 17
Joined: 2018/04/25 13:30:50

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by tomkep » 2018/07/05 08:06:28

In my case these are LVMs which are spread across two disks with RAID1 and /boot partition which is on regular, software, RAID1. To add more to the picture, some of the LVMs (including RAID1 LVMs) have luks on top of them.

User avatar
TrevorH
Forum Moderator
Posts: 23681
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by TrevorH » 2018/07/05 08:13:36

Given the stacktrace, what's underneath the mdadm array? Are they standard SATA disks? What are they attached to?
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

q7joey
Posts: 11
Joined: 2018/07/04 05:34:19

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by q7joey » 2018/07/05 15:35:03

looking at things a bit more today. it looks like it is the dm_raid module that is the problem. that looks to be the last kernel message before my system hangs. i have a bunch of raid10 volumes on top of multipath, so i'm guessing it is a resource issue of some sort. my /proc/partitions is 375 lines long.

i'll keep updating as i get more info.

User avatar
TrevorH
Forum Moderator
Posts: 23681
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: kernel 3.10.0-862.6.3 hanging when loading raid

Post by TrevorH » 2018/07/05 15:48:49

Are you using FakeRAID?
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

Post Reply