mdraid OS drive replacement unsuccessful

[pharthiphan]

I am trying to replace OS drive which is configured with mdraid but after removing and adding back drive to the raid, system won't come up after rebooting.

Code: Select all

# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Mon May 13 13:12:06 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=99bb66eb-7f11-4bf0-8668-c38aec73487a /                       ext4    defaults        1 1
UUID=da159c04-63c5-45c4-bec9-ce83ccadf52f swap                    swap    defaults        0 0
#

Code: Select all

#  blkid | egrep "md126|md127"
/dev/md126: UUID="99bb66eb-7f11-4bf0-8668-c38aec73487a" TYPE="ext4" 
/dev/md127: UUID="da159c04-63c5-45c4-bec9-ce83ccadf52f" TYPE="swap" 
#

Code: Select all

# cat /etc/mdadm.conf 
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md/LABELime-boot level=raid1 num-devices=2 UUID=ad9fb443:25747435:88919c76:bbdf9159
ARRAY /dev/md/LABELime-swap level=raid0 num-devices=2 UUID=aab47aa5:d1a284ab:2ae13ea7:1b828b5c
# 

Code: Select all

# cat /proc/mdstat 
Personalities : [raid1] [raid0] 
md126 : active raid1 sdb2[1] sda2[0]
      968231936 blocks super 1.2 [2/2] [UU]
      bitmap: 4/8 pages [16KB], 65536KB chunk

md127 : active raid0 sdb1[1] sda1[0]
      16775168 blocks super 1.2 512k chunks
      
unused devices: <none>
#

Code: Select all

# mdadm --detail /dev/md/LABELime-boot
/dev/md/LABELime-boot:
           Version : 1.2
     Creation Time : Mon May 13 09:11:47 2019
        Raid Level : raid1
        Array Size : 968231936 (923.38 GiB 991.47 GB)
     Used Dev Size : 968231936 (923.38 GiB 991.47 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Mon May 13 22:48:19 2019
             State : clean 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : LABELime-boot
              UUID : ad9fb443:25747435:88919c76:bbdf9159
            Events : 12140

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
# 

Code: Select all

# mdadm --detail /dev/md/LABELime-swap
/dev/md/LABELime-swap:
           Version : 1.2
     Creation Time : Mon May 13 09:11:59 2019
        Raid Level : raid0
        Array Size : 16775168 (16.00 GiB 17.18 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Mon May 13 09:11:59 2019
             State : clean 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

        Chunk Size : 512K

Consistency Policy : none

              Name : LABELime-swap
              UUID : aab47aa5:d1a284ab:2ae13ea7:1b828b5c
            Events : 0

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
# 

Code: Select all

# swapon -s
Filename				Type		Size	Used	Priority
/dev/md127                             	partition	16775164	0	-2
# swapoff -a
# swapon -s
# 

Code: Select all

# mdadm --manage /dev/md/LABELime-swap --stop
mdadm: stopped /dev/md/LABELime-swap
# 

Code: Select all

# mdadm --manage /dev/md/LABELime-boot --fail /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md/LABELime-boot
[root@sn19 ~]# mdadm --manage /dev/md/LABELime-boot --remove /dev/sdb2
mdadm: hot removed /dev/sdb2 from /dev/md/LABELime-boot
# 

Code: Select all

# mdadm --zero-superblock /dev/sdb2
# mdadm --zero-superblock /dev/sdb1

Code: Select all

# sfdisk -d /dev/sda | sfdisk --force /dev/sdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdb: 121601 cylinders, 255 heads, 63 sectors/track
Old situation:
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1          0       -       0          0    0  Empty
/dev/sdb2          0       -       0          0    0  Empty
/dev/sdb3          0       -       0          0    0  Empty
/dev/sdb4          0       -       0          0    0  Empty
New situation:
Units: sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1          2048  16795647   16793600  fd  Linux raid autodetect
/dev/sdb2   *  16795648 1953523711 1936728064  fd  Linux raid autodetect
/dev/sdb3             0         -          0   0  Empty
/dev/sdb4             0         -          0   0  Empty
Warning: partition 1 does not end at a cylinder boundary
Warning: partition 2 does not start at a cylinder boundary
Warning: partition 2 does not end at a cylinder boundary
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
#

Code: Select all

# mdadm -Cv /dev/md/LABELime-swap -l0 -n2 --name=LABELime-swap -R /dev/sda1 /dev/sdb1
mdadm: chunk size defaults to 512K
mdadm: /dev/sda1 appears to be part of a raid array:
       level=raid0 devices=2 ctime=Mon May 13 09:11:59 2019
mdadm: creation continuing despite oddities due to --run
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/LABELime-swap started.
# 

Code: Select all

# sed -i '/LABELime-swap/d' /etc/mdadm.conf
# mdadm --detail --scan | grep LABELime-swap >> /etc/mdadm.conf

Code: Select all

# mdadm --manage /dev/md/LABELime-boot --add /dev/sdb2
mdadm: added /dev/sdb2

Code: Select all

# cat /proc/mdstat 
Personalities : [raid1] [raid0] 
md127 : active raid0 sdb1[1] sda1[0]
      16775168 blocks super 1.2 512k chunks
      
md126 : active raid1 sdb2[2] sda2[0]
      968231936 blocks super 1.2 [2/1] [U_]
      [=>...................]  recovery =  9.4% (91276288/968231936) finish=151.5min speed=96443K/sec
      bitmap: 4/8 pages [16KB], 65536KB chunk

unused devices: <none>
#

After a successful rebuild, I am trying to reboot the server but it gets stuck saying

mdadm-boot-failure.png (221.98 KiB) Viewed 781 times

what am I doing wrong?

[pharthiphan]

I had to change UUID of swap (as it was recreated) /etc/default/grub and run grub2-mkconfig -o /boot/grub2/grub.cfg