kernel 3.10.0-862.6.3 hanging when loading raid

[q7joey]

just upgraded to the latest kernel and the server hung during reboot. the last thing on the console was trying to load the raid driver. this server has dual path sas to 12 drives and lots of lvm striped raid on top of that.

mainly posting to see if anyone else is seeing problems with the latest kernel hanging in a similar fashion.

[avij]

What kind of RAID controller? Do you use CentOS drivers for that, or manufacturer's drivers, or some other 3rd party drivers?

[q7joey]

sorry should have said this is all software raid. it hangs after loading the software raid module.

[tomkep]

I can see this problem as well on three DELL Precision systems (one T7400, the other two T7500). Software raid1, HW is different between them (some kind of SAS controller, I can dig the details if needed) but probably irrelevant. This does not reproduce on Lenovo laptop without SW raid.

Luckily I have a screenshot from one of them:

[q7joey]

glad to see i'm not the only one. i don't get a panic, just a hang. rolled back to 862.3.3 and things are fine there. this is a production box, but just used for backups so i can probably do some more diagnostics after the 4th.

[avij]

Interesting. I'm unable to reproduce this bug myself, though. I set up a VM in VirtualBox with two disks, / and /boot as RAID1 devices (no LVM) and a small swap partition (non-RAID) on both devices. Works ok with 3.10.0-862.6.3. I wonder which specific configuration combination triggers this bug.

[tomkep]

In my case these are LVMs which are spread across two disks with RAID1 and /boot partition which is on regular, software, RAID1. To add more to the picture, some of the LVMs (including RAID1 LVMs) have luks on top of them.

[TrevorH]

Given the stacktrace, what's underneath the mdadm array? Are they standard SATA disks? What are they attached to?

[q7joey]

looking at things a bit more today. it looks like it is the dm_raid module that is the problem. that looks to be the last kernel message before my system hangs. i have a bunch of raid10 volumes on top of multipath, so i'm guessing it is a resource issue of some sort. my /proc/partitions is 375 lines long.

i'll keep updating as i get more info.

[TrevorH]

Are you using FakeRAID?