Thin pool lvm mismatch

fvlasie · Post by **fvlasie** » 2018/03/21 17:25:11

Hello everyone,

I have an OpenStack install on CentOS 7 (via PackStack). Some of my lvm volumes have become unavailable.

I really need some help retrieving the data from these volumes.

Code: Select all

[root@plato ~]# lvdisplay
  --- Logical volume ---
  LV Name                cinder-volumes-pool
  VG Name                cinder-volumes
  LV UUID                PEkGKb-fhAc-CJD2-uDDA-k911-SIX9-1uyvFo
  LV Write Access        read/write
  LV Creation host, time plato, 2018-02-01 13:33:51 -0800
  LV Pool metadata       cinder-volumes-pool_tmeta
  LV Pool data           cinder-volumes-pool_tdata
  LV Status              NOT available
  LV Size                9.50 TiB
  Current LE             2490368
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/cinder-volumes/volume-8f4a5fff-749f-47fe-976f-6157f58a4d9e
  LV Name                volume-8f4a5fff-749f-47fe-976f-6157f58a4d9e
  VG Name                cinder-volumes
  LV UUID                C2o7UD-uqFp-3L3r-F0Ys-etjp-QBJr-idBhb0
  LV Write Access        read/write
  LV Creation host, time plato, 2018-02-02 10:18:41 -0800
  LV Pool name           cinder-volumes-pool
  LV Status              NOT available
  LV Size                1.00 GiB
  Current LE             256
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/cinder-volumes/volume-6ad82e98-c8e2-4837-bffd-079cf76afbe3
  LV Name                volume-6ad82e98-c8e2-4837-bffd-079cf76afbe3
  VG Name                cinder-volumes
  LV UUID                qisf80-j4XV-PpFy-f7yt-ZpJS-99v0-m03Ql4
  LV Write Access        read/write
  LV Creation host, time plato, 2018-02-02 10:26:46 -0800
  LV Pool name           cinder-volumes-pool
  LV Status              NOT available
  LV Size                1.00 GiB
  Current LE             256
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/cinder-volumes/volume-ee107488-2559-4116-aa7b-0da02fd5f693
  LV Name                volume-ee107488-2559-4116-aa7b-0da02fd5f693
  VG Name                cinder-volumes
  LV UUID                FS9Y2o-HYe2-HK03-yM0Z-P7GO-kAzD-cOYNTb
  LV Write Access        read/write
  LV Creation host, time plato.spots.onsite, 2018-02-12 10:28:57 -0800
  LV Pool name           cinder-volumes-pool
  LV Status              NOT available
  LV Size                40.00 GiB
  Current LE             10240
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/cinder-volumes/volume-d6f0260d-21b5-43e7-afe5-84e0502fa734
  LV Name                volume-d6f0260d-21b5-43e7-afe5-84e0502fa734
  VG Name                cinder-volumes
  LV UUID                b6pX01-mOEH-3j3K-32NJ-OHsz-UMQe-y10vSM
  LV Write Access        read/write
  LV Creation host, time plato.spots.onsite, 2018-02-14 14:24:41 -0800
  LV Pool name           cinder-volumes-pool
  LV Status              NOT available
  LV Size                40.00 GiB
  Current LE             10240
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/cinder-volumes/volume-a7bd0bc8-8cbc-4053-bdc2-2eb9bfb0f147
  LV Name                volume-a7bd0bc8-8cbc-4053-bdc2-2eb9bfb0f147
  VG Name                cinder-volumes
  LV UUID                T07JAE-3CNU-CpwN-BUEr-aAJG-VxP5-1qFYZz
  LV Write Access        read/write
  LV Creation host, time plato.spots.onsite, 2018-03-12 10:33:24 -0700
  LV Pool name           cinder-volumes-pool
  LV Status              NOT available
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/cinder-volumes/volume-29fa3b6d-1cbf-40db-82bb-1756c6fac9a5
  LV Name                volume-29fa3b6d-1cbf-40db-82bb-1756c6fac9a5
  VG Name                cinder-volumes
  LV UUID                IB0q1n-NnkR-tx5w-BbBu-LamG-jCbQ-mYXWyC
  LV Write Access        read/write
  LV Creation host, time plato.spots.onsite, 2018-03-14 09:52:14 -0700
  LV Pool name           cinder-volumes-pool
  LV Status              NOT available
  LV Size                40.00 GiB
  Current LE             10240
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
   
  --- Logical volume ---
  LV Path                /dev/centos/root
  LV Name                root
  VG Name                centos
  LV UUID                nawE4n-dOHs-VsNH-f9hL-te05-mvGC-WoFQzv
  LV Write Access        read/write
  LV Creation host, time localhost, 2018-01-22 09:50:38 -0800
  LV Status              available
  # open                 1
  LV Size                50.00 GiB
  Current LE             12800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0
   
  --- Logical volume ---
  LV Path                /dev/centos/swap
  LV Name                swap
  VG Name                centos
  LV UUID                Vvlni4-nwTl-ORwW-Gg8b-5y4h-kXJ5-T67cKU
  LV Write Access        read/write
  LV Creation host, time localhost, 2018-01-22 09:50:38 -0800
  LV Status              available
  # open                 2
  LV Size                8.12 GiB
  Current LE             2080
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:1
   
  --- Logical volume ---
  LV Path                /dev/centos/home
  LV Name                home
  VG Name                centos
  LV UUID                lCXJ7v-jeOC-DFKI-unXa-HUKx-9DXp-nmzSMg
  LV Write Access        read/write
  LV Creation host, time localhost, 2018-01-22 09:50:39 -0800
  LV Status              available
  # open                 1
  LV Size                964.67 GiB
  Current LE             246956
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2

Code: Select all

[root@plato ~]# lvchange -a y cinder-volumes
  Thin pool cinder--volumes-cinder--volumes--pool-tpool (253:5) transaction_id is 0, while expected 72.
  Thin pool cinder--volumes-cinder--volumes--pool-tpool (253:5) transaction_id is 0, while expected 72.
  Thin pool cinder--volumes-cinder--volumes--pool-tpool (253:5) transaction_id is 0, while expected 72.
  Thin pool cinder--volumes-cinder--volumes--pool-tpool (253:5) transaction_id is 0, while expected 72.
  Thin pool cinder--volumes-cinder--volumes--pool-tpool (253:5) transaction_id is 0, while expected 72.
  Thin pool cinder--volumes-cinder--volumes--pool-tpool (253:5) transaction_id is 0, while expected 72.
  Thin pool cinder--volumes-cinder--volumes--pool-tpool (253:5) transaction_id is 0, while expected 72.

All the volumes in the cinder-volumes VG cannot be activated.

Any help or suggestions would be much appreciated.

FV

wawayue · Post by **wawayue** » 2018/05/24 07:42:42

It is fairly clear that these MAY NOT be persistent.

first off you should find out WHY these volumes have become unavailable.
Did you overflow your thin pool?
did you just shut down OS without the correct procedure, you CANNOT just turn off / shutdown -h now , an openstack instance.

go check out the following logs for errors:
neutron
nova
cinder

and stop using a scatter gun
The fault is CLEARLY the thin-pool
"LVdisplay" is USELESS

moving on.....

lvs -a
or better still:

lvs -a cinder-volumes

post the result.......

fvlasie · Post by **fvlasie** » 2018/05/25 16:20:14

Hello,

Thank you for your reply!

Unfortunately, the OpenStack install in question is now but a distant memory.

I was not able to recover any of the data despite copious applications of scatter gun spray....

I had a big helping of humble pie and rebuilt the system.

The motto: Never be without a backup (pretty obvious I know)

FV

wawayue · Post by **wawayue** » 2018/05/26 08:39:50

A backup will not fully save you...... trust me....
There is a "flaw" in the packstack setup, the thin pool is a file, barely enough for a few systems of a few GB each.
this "file" gets mounted on the "loopback"
start throwing in "snapshots" and its game over very quickly.

The point I was trying to make is that to run such systems a user MUST be completely familiar with the underlying tech
in this case the "thin pool" or "quick death" as it should be called.

since i have my dev system on today... i can get a dump

notice below my use of "lvs -a cinder-volumes"

lvs -a cinder-volumes

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
cinder-volumes-pool cinder-volumes twi-aotz-- 200.00g 4.07 6.94
[cinder-volumes-pool_tdata] cinder-volumes Twi-ao---- 200.00g
[cinder-volumes-pool_tmeta] cinder-volumes ewi-ao---- 72.00m
[lvol0_pmspare] cinder-volumes ewi------- 20.00m
volume-f21049ee-26c4-4d57-8aa1-aebc3b6ee6b6 cinder-volumes Vwi-aotz-- 10.00g cinder-volumes-pool 81.46

this gives us insight INTO the thin pool, NOT the attached volumes LVdisplay (which tells us nothing)

using your error message you now can glean a clear idea of exactly what went wrong.
it was not "openstack" per se, but rather a thinpool malfunction.

The thin pool can corrupt at multiple levels,
1. thin pool tmeta
2. thin pool tdata
3. thin pool container "cinder-volumes-pool"
4. enclosed LVM volumes. " volume-f21049ee-26c4-4d57-8aa1-aebc3b6ee6b6" in this case
they can corrupt at any level and there is NOT a consolidated framework for reporting the errors.

so ... even with a backup if any of these gets corrupt, it's enough to make baby Jesus cry.......

fvlasie · Post by **fvlasie** » 2018/05/26 19:31:43

Very helpful information thank you!

What I meant by a backup was the data itself so that it could be copied on to a new system if the fragile lvm thin pool breaks.

I was actually running cinder with a lv group rather than the loopback, but I understand what you mean.

Thanks again for the additional info.

FV

CentOS

Thin pool lvm mismatch

Thin pool lvm mismatch

Re: Thin pool lvm mismatch

Re: Thin pool lvm mismatch

Re: Thin pool lvm mismatch

Re: Thin pool lvm mismatch