[SOLVED] Unable to start my machine

[Chinostra]

Hello,

This is my first topic it's due to a big problem... after today's upgrade, a reboot was requested, and now, I'm unable to start my machine... after rebooting, it seems that my LVM volumes can't be correctly mounted:

First of all...

Code: Select all

[root@fak1r ~]# uname -a
Linux fak1r 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

And now...

dmesg:

Code: Select all

...
Setting up Logical Volume Management: device-mapper: table: 253:0: linear: dm-linear: Device lookup failed
device-mapper: ioctl: error adding target to table
device-mapper: table: 253:1: linear: dm-linear: Device lookup failed
device-mapper: ioctl: error adding target to table
  device-mapper:device-mapper: table: 253:2: linear: dm-linear: Device lookup failed
 reload ioctl ondevice-mapper: ioctl: error adding target to table
  failed: Invalid argument
  device-mapper: reload ioctl on  failed: Invalid argument
  device-mapper: reload ioctl on  failed: Invalid argument
  3 logical volume(s) in volume group "vg00" now active
[FAILED]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/md1
root: clean, 19023/262144 files, 262308/1048560 blocks
[/sbin/fsck.ext4 (1) -- /usr] fsck.ext4Give root password for maintenance
(or type Control-D to continue):
bash: tty: command not found
bash: warning: setlocale: LC_CTYPE: cannot change locale (es_ES.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (es_ES.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (es_ES.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (es_ES.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (es_ES.UTF-8): No such file or directory

dmsetup status:

Code: Select all

[root@fak1r ~]# dmsetup status
vg00-home:
vg00-usr:
vg00-var:
[root@fak1r ~]#

mount -a:

Code: Select all

[root@fak1r ~]# mount -a
mount: special device /dev/vg00/usr does not exist
mount: special device /dev/vg00/var does not exist
mount: devpts already mounted or /dev/pts busy

fstab content:

Code: Select all

[root@fak1r ~]# cat /etc/fstab
/dev/md1   /      ext3    defaults        1 1
/dev/sda2   none      swap    sw
/dev/sdb2   none      swap    sw
/dev/vg00/usr      /usr      ext4   defaults   0 2
/dev/vg00/var      /var      ext4   defaults,usrquota   0 2
devpts      /dev/pts   devpts  gid=5,mode=620  0 0
none      /proc      proc    defaults        0 0
none      /tmp   tmpfs   defaults        0 0

vgchange:

Code: Select all

[root@fak1r ~]# lvm vgchange -a y --ignorelockingfailure
  File-based locking initialisation failed.
device-mapper: table: 253:0: linear: dm-linear: Device lookup failed
device-mapper: ioctl: error adding target to table
  device-mapper:device-mapper: table: 253:1: linear: dm-linear: Device lookup failed
 reload ioctl ondevice-mapper: ioctl: error adding target to table
  failed: Invalidevice-mapper: table: 253:2: linear: dm-linear: Device lookup failed
d argument
  dedevice-mapper: ioctl: error adding target to table
vice-mapper: reload ioctl on  failed: Invalid argument
  device-mapper: reload ioctl on  failed: Invalid argument
  3 logical volume(s) in volume group "vg00" now active

Some tests:

Code: Select all

[root@fak1r ~]# pvscan --ignorelockingfailure
  File-based locking initialisation failed.
  PV /dev/md3   VG vg00   lvm2 [925.51 GiB / 25.51 GiB free]
  Total: 1 [925.51 GiB] / in use: 1 [925.51 GiB] / in no VG: 0 [0   ]

Code: Select all

[root@fak1r ~]# vgscan --ignorelockingfailure
  File-based locking initialisation failed.
  Reading all physical volumes.  This may take a while...
  Found volume group "vg00" using metadata type lvm2

Code: Select all

[root@fak1r ~]# vgdisplay -v --ignorelockingfailure
    Creating directory "/var/lock/lvm"
  File-based locking initialisation failed.
    Locking disabled - only read operations permitted.
    Finding all volume groups
    Finding volume group "vg00"
  --- Volume group ---
  VG Name               vg00
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  7
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               925.51 GiB
  PE Size               4.00 MiB
  Total PE              236931
  Alloc PE / Size       230400 / 900.00 GiB
  Free  PE / Size       6531 / 25.51 GiB
  VG UUID               F1whyb-qksn-5vvq-mPIG-6siB-zJFH-PryH1U
   
  --- Logical volume ---
  LV Path                /dev/vg00/usr
  LV Name                usr
  VG Name                vg00
  LV UUID                qBQ7Jk-RzGk-mkW0-WxWz-uKLp-CADz-G5RSpp
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 0
  LV Size                300.00 GiB
  Current LE             76800
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
   
  --- Logical volume ---
  LV Path                /dev/vg00/var
  LV Name                var
  VG Name                vg00
  LV UUID                FT7sY1-UelC-d9nD-2WNQ-py7c-ytRm-pgimQQ
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 0
  LV Size                300.00 GiB
  Current LE             76800
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
   
  --- Logical volume ---
  LV Path                /dev/vg00/home
  LV Name                home
  VG Name                vg00
  LV UUID                5eMxca-3l4L-M7LS-jwJC-JEq0-ffNu-tPCH7K
  LV Write Access        read/write
  LV Creation host, time ,
  LV Status              available
  # open                 0
  LV Size                300.00 GiB
  Current LE             76800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2
   
  --- Physical volumes ---
  PV Name               /dev/sda3     
  PV UUID               Mq4X37-KE2c-q9e2-wbaK-Vcse-F4tV-yYijBF
  PV Status             allocatable
  Total PE / Free PE    236931 / 6531

Code: Select all

[root@fak1r ~]# vgchange -ay --ignorelockingfailure
  File-based locking initialisation failed.
device-mapper: table: 253:0: linear: dm-linear: Device lookup failed
device-mapper: ioctl: error adding target to table
  device-mapper:device-mapper: table: 253:1: linear: dm-linear: Device lookup failed
 reload ioctl ondevice-mapper: ioctl: error adding target to table
  failed: Invalidevice-mapper: table: 253:2: linear: dm-linear: Device lookup failed
d argument
  dedevice-mapper: ioctl: error adding target to table
vice-mapper: reload ioctl on  failed: Invalid argument
  device-mapper: reload ioctl on  failed: Invalid argument
  3 logical volume(s) in volume group "vg00" now active

Code: Select all

[root@fak1r ~]# vgscan -v --ignorelockingfailure --mknodes -d
    Creating directory "/var/lock/lvm"
  File-based locking initialisation failed.
    Locking disabled - only read operations permitted.
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
    Finding all volume groups
    Finding volume group "vg00"
  Found volume group "vg00" using metadata type lvm2
    Finding all logical volumes

Can't figure out where the problem resides or what I need to do to solve it, I've tried several things and can't remember all the tests that I've done, if you think I have to do something more or any other tests or information you might need (even it sounds very stupid), please, tell me, because I need to have this working ASAP.

Many thanks in advance,

Pablo A.

[TrevorH]

Try rebooting and hitting Esc when you see the "Starting CentOS 2.6.32-431.23.3.el6 in X seconds" prompt. From there you can select the older kernel and see if that works. If it does then yum remove the newer one, rm the correct /boot/initramfs file and then yum update kernel and see if it works then.

[Chinostra]

Try rebooting and hitting Esc when you see the "Starting CentOS 2.6.32-431.23.3.el6 in X seconds" prompt. From there you can select the older kernel and see if that works. If it does then yum remove the newer one, rm the correct /boot/initramfs file and then yum update kernel and see if it works then.

Hi TrevorH, many thanks for your interest and answer, unfortunately, I've tried with all the available kernels that I've, and none of them works, the older ones throws different error than the new ones... like the 'Unable to read superblock', but I think that everything is related.

Do you want me to get the output from a different kernel to check?

Thanks again,

Pablo A.

[Chinostra]

Ok, finally I found a solution... this hosted machine has a weird configuration... I've the disks mounted as a RAID1 by software (mdadm), but both disks doesn't have a filesystem due to this are controlled by LVM.

Found this: http://forums.gentoo.org/viewtopic-t-623639-view-previous.html?sid=bafbb705cc9461aa442bef38446edf94

And it worked, deleting the /etc/lvm/cache/.cache hidden file and rebooting (after a complete resync of the RAID) made this boot normally.