Critical Issues with Multipath losing Config on reboot

[h1tchiker]

Hi,

I was wondering if some one could help me please? I am of course new to multipath which is probably the issue I get the concepts and I think have it configured kind of. I am currently using FC connected disks and presenting these out via iSCSI using targetcli. all was going great apart from a couple of setup hickups. I now have our Companies backup going to the SAN's using VDP in VMware and some critical data.

Yesterday I had to reboot the CentOS 7 box and wow much to my suprise all of my disks disappeared, luckily I managed to get these back using the following commands:

kpartx -a -p p -v /dev/mapper/mpatha1
kpartx -a -p p -v /dev/mapper/mpathb1
kpartx -a -p p -v /dev/mapper/mpathc1

I had an issue with /dev/mapper/mpathd where I had to recreate the partition:

fdisk /dev/mapper/mpathd

Then I was able to use:

/dev/mapper/mpathd1

I will admit my partitions are in a real mess and I cannot afford to lose any data:

total 0
lrwxrwxrwx. 1 root root 8 Aug 16 13:20 cl-home -> ../dm-10
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 cl-root -> ../dm-0
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 cl-swap -> ../dm-1
crw-------. 1 root root 10, 236 Aug 16 13:20 control
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 mpatha -> ../dm-4
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 mpatha1 -> ../dm-6
lrwxrwxrwx. 1 root root 8 Aug 16 14:10 mpatha1p1 -> ../dm-11
lrwxrwxrwx. 1 root root 8 Aug 16 14:14 mpatha1p1p1 -> ../dm-15
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 mpathb -> ../dm-5
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 mpathb1 -> ../dm-8
lrwxrwxrwx. 1 root root 8 Aug 16 14:11 mpathb1p1 -> ../dm-12
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 mpathb2 -> ../dm-9
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 mpathc -> ../dm-3
lrwxrwxrwx. 1 root root 7 Aug 16 13:20 mpathc1 -> ../dm-7
lrwxrwxrwx. 1 root root 8 Aug 16 14:13 mpathc1p1 -> ../dm-14
lrwxrwxrwx. 1 root root 8 Aug 16 14:15 mpathc1p1p1 -> ../dm-17
lrwxrwxrwx. 1 root root 7 Aug 16 14:50 mpathd -> ../dm-2
lrwxrwxrwx. 1 root root 8 Aug 16 14:59 mpathdp1 -> ../dm-19
lrwxrwxrwx. 1 root root 8 Aug 16 15:00 mpathdp1p1 -> ../dm-20
lrwxrwxrwx. 1 root root 8 Aug 16 15:02 mpathdp1p1p1 -> ../dm-21
lrwxrwxrwx. 1 root root 8 Aug 16 14:11 vg_file01_vdp-lv_file01 -> ../dm-13
lrwxrwxrwx. 1 root root 8 Aug 16 14:15 vg_iscsi_storage-lv_iscsi_storage -> ../dm-18
lrwxrwxrwx. 1 root root 8 Aug 16 15:02 vg_iscsi_vdp-lv_vdp -> ../dm-22
lrwxrwxrwx. 1 root root 8 Aug 16 14:14 vg_iscsi_vms-lv_iscsi_vm -> ../dm-16

Now since the reboot the command:

multipath -v2

Returns absolutely nothing but was working fine until the system reboot, the command 'multipath -ll' returns:

# multipath -ll
mpathd (3600c0ff00012e031efe9655901000000) dm-2 HP ,P2000 G3 FC
size=13T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 0:0:1:0 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 0:0:2:0 sdd 8:48 active ready running
mpathc (3600c0ff00014ebf5c757635901000000) dm-3 HP ,P2000 G3 FC
size=9.1T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 0:0:0:0 sdb 8:16 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 0:0:3:0 sde 8:64 active ready running
mpathb (3600c0ff00012e03163ea655901000000) dm-5 HP ,P2000 G3 FC
size=7.3T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 4:0:1:0 sdg 8:96 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 4:0:2:0 sdh 8:112 active ready running
mpatha (3600c0ff000dadd07d959635901000000) dm-4 HP ,P2000 G3 FC
size=2.7T features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| `- 4:0:3:0 sdi 8:128 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
`- 4:0:0:0 sdf 8:80 active ready running

I read through the couple of pages at https://www.centos.org/docs/5/html/5.2/ ... edure.html for setting up multipath but I have obviously missed some thing quite major here and I am in need of some help, please please please............

How do I get the system stable? What do I have to do make sure that I do not lose the config on reboot?

My Multipath.conf is default, although I do remember adding in /dev/sda

Can some one please help?

[hunter86_bg]

This one could provide a more general overview:

Code: Select all

lsblk

Note: you can use the [ code] [/code] tags (just don't leave the space before the "code")

[h1tchiker]

Here it is:

# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 136.7G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 135.7G 0 part
├─cl-root 253:0 0 50G 0 lvm /
├─cl-swap 253:1 0 13.7G 0 lvm [SWAP]
└─cl-home 253:10 0 72G 0 lvm /home
sdb 8:16 0 9.1T 0 disk
├─sdb1 8:17 0 9.1T 0 part
└─mpathc 253:3 0 9.1T 0 mpath
└─mpathc1 253:7 0 9.1T 0 part
└─mpathc1p1 253:14 0 9.1T 0 part
└─mpathc1p1p1 253:17 0 9.1T 0 part
└─vg_iscsi_storage-lv_iscsi_storage 253:18 0 9.1T 0 lvm
sdc 8:32 0 12.8T 0 disk
├─sdc1 8:33 0 12.8T 0 part
└─mpathd 253:2 0 12.8T 0 mpath
└─mpathdp1 253:19 0 12.8T 0 part
└─mpathdp1p1 253:20 0 12.8T 0 part
└─mpathdp1p1p1 253:21 0 12.8T 0 part
└─vg_iscsi_vdp-lv_vdp 253:22 0 12.8T 0 lvm
sdd 8:48 0 12.8T 0 disk
├─sdd1 8:49 0 12.8T 0 part
└─mpathd 253:2 0 12.8T 0 mpath
└─mpathdp1 253:19 0 12.8T 0 part
└─mpathdp1p1 253:20 0 12.8T 0 part
└─mpathdp1p1p1 253:21 0 12.8T 0 part
└─vg_iscsi_vdp-lv_vdp 253:22 0 12.8T 0 lvm
sde 8:64 0 9.1T 0 disk
├─sde1 8:65 0 9.1T 0 part
└─mpathc 253:3 0 9.1T 0 mpath
└─mpathc1 253:7 0 9.1T 0 part
└─mpathc1p1 253:14 0 9.1T 0 part
└─mpathc1p1p1 253:17 0 9.1T 0 part
└─vg_iscsi_storage-lv_iscsi_storage 253:18 0 9.1T 0 lvm
sdf 8:80 0 2.7T 0 disk
├─sdf1 8:81 0 2.7T 0 part
└─mpatha 253:4 0 2.7T 0 mpath
└─mpatha1 253:6 0 2.7T 0 part
└─mpatha1p1 253:11 0 2.7T 0 part
└─mpatha1p1p1 253:15 0 2.7T 0 part
└─vg_iscsi_vms-lv_iscsi_vm 253:16 0 2.7T 0 lvm
sdg 8:96 0 7.3T 0 disk
├─sdg1 8:97 0 7.3T 0 part
├─sdg2 8:98 0 1007K 0 part
└─mpathb 253:5 0 7.3T 0 mpath
├─mpathb1 253:8 0 7.3T 0 part
│ └─mpathb1p1 253:12 0 7.3T 0 part
│ └─vg_file01_vdp-lv_file01 253:13 0 7.3T 0 lvm
└─mpathb2 253:9 0 1007K 0 part
sdh 8:112 0 7.3T 0 disk
├─sdh1 8:113 0 7.3T 0 part
├─sdh2 8:114 0 1007K 0 part
└─mpathb 253:5 0 7.3T 0 mpath
├─mpathb1 253:8 0 7.3T 0 part
│ └─mpathb1p1 253:12 0 7.3T 0 part
│ └─vg_file01_vdp-lv_file01 253:13 0 7.3T 0 lvm
└─mpathb2 253:9 0 1007K 0 part
sdi 8:128 0 2.7T 0 disk
├─sdi1 8:129 0 2.7T 0 part
└─mpatha 253:4 0 2.7T 0 mpath
└─mpatha1 253:6 0 2.7T 0 part
└─mpatha1p1 253:11 0 2.7T 0 part
└─mpatha1p1p1 253:15 0 2.7T 0 part
└─vg_iscsi_vms-lv_iscsi_vm 253:16 0 2.7T 0 lvm
sr0 11:0 1 1024M 0 rom


Thank You

[hunter86_bg]

Post the output of:

Code: Select all

pvs  && vgs && lvs

and your multipath config.I'm not 100% sure but you might have to regenerate your initramfs after every change in the multipath.
Have you checked this guide?

[h1tchiker]

Here you go:

# pvs && vgs && lvs
PV VG Fmt Attr PSize PFree
/dev/mapper/mpatha1p1p1 vg_iscsi_vms lvm2 a-- 2.73t 0
/dev/mapper/mpathb1p1 vg_file01_vdp lvm2 a-- 7.26t 0
/dev/mapper/mpathc1p1p1 vg_iscsi_storage lvm2 a-- 9.09t 0
/dev/mapper/mpathdp1p1p1 vg_iscsi_vdp lvm2 a-- 12.73t 0
/dev/sda2 cl lvm2 a-- 135.70g 0
VG #PV #LV #SN Attr VSize VFree
cl 1 3 0 wz--n- 135.70g 0
vg_file01_vdp 1 1 0 wz--n- 7.26t 0
vg_iscsi_storage 1 1 0 wz--n- 9.09t 0
vg_iscsi_vdp 1 1 0 wz--n- 12.73t 0
vg_iscsi_vms 1 1 0 wz--n- 2.73t 0
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home cl -wi-ao---- 72.02g
root cl -wi-ao---- 50.00g
swap cl -wi-ao---- 13.67g
lv_file01 vg_file01_vdp -wi-ao---- 7.26t
lv_iscsi_storage vg_iscsi_storage -wi-ao---- 9.09t
lv_vdp vg_iscsi_vdp -wi-ao---- 12.73t
lv_iscsi_vm vg_iscsi_vms -wi-ao---- 2.73t
#

Thank You

[h1tchiker]

Will updating the initramfs cause any harm to the current config?

I will check the guide in the morning and see if I have missed anything and get back to you ASAP.

[h1tchiker]

Hi,

Thanks for the information so far, I have taken a look at the guide that you sent the link for, although it is more informational it does not provide much more than the one I used originally:

https://www.centos.org/docs/5/html/5.2/ ... edure.html

How ever I have updated the initramfs using the command:

dracut --force --add multipath --include /etc/multipath

As suggeysted, but I am still very concerned here, the command:

multipath -v2

Still returns nothing, one thing I have noticed is when I checked to see if multipathd was active on boot i get the following:

systemctl status multipathd -l
● multipathd.service - Device-Mapper Multipath Device Controller
Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2017-08-16 13:20:03 BST; 6 days ago
Main PID: 688 (multipathd)
CGroup: /system.slice/multipathd.service
└─688 /sbin/multipathd

Aug 16 13:20:05 SERVERNAME multipathd[688]: sdi: add path (uevent)
Aug 16 13:20:05 SERVERNAME multipathd[688]: sdi: spurious uevent, path already in pathvec
Aug 16 13:20:05 SERVERNAME multipathd[688]: mpathd: load table [0 27343749888 multipath 1 queue_if_no_path 0 2 1 service-time 0 1 1 8:32 1 service-time 0 1 1 8:48 1]
Aug 16 13:20:05 SERVERNAME multipathd[688]: mpathc: load table [0 19515446400 multipath 1 queue_if_no_path 0 2 1 service-time 0 1 1 8:16 1 service-time 0 1 1 8:64 1]
Aug 16 13:20:05 SERVERNAME multipathd[688]: mpatha: load table [0 5854593280 multipath 1 queue_if_no_path 0 2 1 service-time 0 1 1 8:128 1 service-time 0 1 1 8:80 1]
Aug 16 13:20:05 SERVERNAME multipathd[688]: mpathb: load table [0 15590232192 multipath 1 queue_if_no_path 0 2 1 service-time 0 1 1 8:96 1 service-time 0 1 1 8:112 1]
Aug 16 13:20:05 SERVERNAME multipathd[688]: sda: add path (uevent)
Aug 16 13:20:05 SERVERNAME multipathd[688]: sda: spurious uevent, path already in pathvec
Aug 16 14:10:37 SERVERNAME multipathd[688]: dm-11: remove map (uevent)
Aug 16 14:12:53 SERVERNAME multipathd[688]: dm-14: remove map (uevent)

Other than this I cannot find any thing useful. Can some one help please? Is this a known bug in this release of multipath? I have found this:

https://access.redhat.com/solutions/31230

But I cannot access it as I do not have a RH subscription

[hunter86_bg]

This solution just mentions that it was fixed with an Erratum (aka updated package).
If the sda device is local , then you can blacklist it into the multipath config.Also, if you do not provide a logical volume as iscsi target block device - then you should add it to the global filter in lvm.conf

What is your version of the multipath ? Is there a newer one - if so , maybe it would be best to update it and restart the daemon. Maybe an update of the kernel should be the next step.

[h1tchiker]

OK, I have read similar, and every thing I have seen on the system leads me to believe the issue has been caused by this scenario:

1. Install CentOS

2. Update Multipath to latest version, 0.4.9 in my case

3. Configure multipath

What has happened here I believe is that because friendly names is now enabled by default, the initramfs uses the WWIDs during boot. I have not used multipath for my root partitions. Once you start the multipath config you tell multipath to use /etc/multipath.conf and /etc/multipath:

Code: Select all

mpathconf --enable --with_multipathd y

As can be seen here:

https://access.redhat.com/documentation ... _procedure

What this does is set up the bindings file for the friendly names and WWIDs maps, as such. Now due to the fact that this was setup after the event of the Kernel being aware of these settings, on boot the system is blissfully unaware that you have performed any config!!!!!!!!! Why is that, its because the initramfs has no idea about the friendly names mapping that was setup in /dev/mapper so literally deletes/drops the config.

If you follow the Official guide for CentOS 7 and multipath (which I did), I do not see any mention of updating initramfs, bug, perhaps, very poor documentation is a definite yes!

So, I am yet to get myself out of this hole and it does seem to me that there is not a great deal of knowledge out there where it comes to multipath (no offence hunter86_bg your input has been greatly appreciated), or it is more likely the case that it is just not that easy to trouble shoot. It has been around long enough now to surely warrant its own set of log files as it is of course quite important.

If I upgrade multipath; of which I haven't looked for a package yet, I would recommend that multipath should be upgraded before a Kernel upgrade, if you do not have a kernel upgrade to perform, I suspect you will need to update the initramfs other wise it is likely the system will spiral down the same issue. If you do upgrade multipath then you should update the initramfs:

dracut --force --add multipath --include /etc/multipath

I still have nothing in multipath -v2 output. Next week I will be able to do some testing. I believe that a Kernel upgrade or re-install is required to fix this issue to sort the multipath problem. I would certainly recommend that if you intend to use multipath make sure that you stay with the config you choose during the install of the OS.

I will let you know what happens in regard to the reboot after performing the initramfs update, but if there is a kernel update I am certainly going to do that first so I may never find out.

Thanks again hunter86_bg I will keep you posted on this one in case it may stop some one else getting into this mess

[h1tchiker]

Hi,

Well, in short, I updated the initramfs with no effect and I finally got a Kernel update installed and rebooted with no effect. As far as I am aware I have followed the RedHat guide to the letter for setting this up and it simply does not work!

I now have a documented procedure to fix the issue if the server reboots but this is far from ideal as it leaves Virtual machines in a bad state due to the fact that targetcli cannot see the LV's when the server reboots and drops the config (another bug that is supposed to be fixed), so not only do I have to manually bring the drives back online I also have to restore the targetcli config. I also believe that since the update the system is not multipathing anymore, which leaves me stuck with an OS that I cannot ditch through fear of not being able to get back to my backup .vmdk's if I change this.

I am however going to try and P2V this and attempt to replicate an FC SAN so I can find a way out of this hole