Centos 5 Upgrade - We Know We Need to Go to 7

A 5 star hangout for overworked and underpaid system admins.
Post Reply
snidleystash
Posts: 5
Joined: 2019/01/10 19:35:32

Centos 5 Upgrade - We Know We Need to Go to 7

Post by snidleystash » 2019/01/10 20:17:50

I have an environment I'm just walking into. Not my fault for why it is what it is. I just want to make the safest possible choice.

We have Centos 5.8. We have had a hard disk failure in what I believe is a RAID 1 configuration. The environment has not been backed up in an eternity, so there's nothing to fall back on. As a result, we're looking to install a backup agent for Acronis. The installation procedure requires kernel, kernel headers, and kernel sources (without explicitly calling for any particular package), as well as gcc, make, etc.

Looking at the packages available and installed, I find that these are installed:

kernel.x86_64 2.6.18-308.13.1.el5
kernel-headers.x86_64 2.6.18-308.13.1.el5

Available are:

kernel 2.6.18-419.el5 updates
kernel-headers.x86_64 2.6.18-419.el5 updates
kernel-devel.x86_64 2.6.18-419.el5 updates

I have been assuming that to have "kernel sources" I need to have kernel-devel.

The documentation for Acronis clearly says that all kernel packages should be at the same level. As such, I'm assuming I need to apply 2.6.18-419 for all three packages.

In most cases, when applying anything to the kernel, I like to reboot. But since we have an at-risk disk environment, with a disk already having failed, it may not be an option.

So, the procedure I'm currently considering, to minimize failure potential, is as follows:

1) Attach a USB external drive to the server.
2) Copy both file systems to the USB-attached drive.
3) Upgrade the kernel packages.
4) Install Acronis agent.
5) Back up the server using Acronis.
6) Replace failed hard drive with new one.
7) Initiate rebuild of hard drive from functional drive still in place.

I do not consider an upgrade to CentOS 6 or 7 to be feasible until after this process is done. We anticipate the update to a new major version of CentOS will take weeks, and we can't afford that at this time. And when we do that upgrade, it is likely to be to the cloud.

The question I have is, "Is this a reasonable process to follow that is most likely to lead to success?"

Do you have any suggestions on how to improve this process?

What utilities would you use for the backup to the USB drive? There are two LVs involved. One is the "/" file system, and is fairly small - under 15G. The other is the data volume and is about 275GB.

Thanks in advance.

SnidleyStash

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Centos 5 Upgrade - We Know We Need to Go to 7

Post by TrevorH » 2019/01/10 20:33:12

You say you have a hard disk failure and that it might be on a RAID 1 setup. Is that hardware RAID or linux software RAID? Your first step should be to determine which of those two you have - if any RAID at all. If it's linux software RAID then the "file" /proc/mdstat will tell you about the current status of the RAID array. If it's hardware RAID then it will show up if you run lspci -nn | grep -i raid and that will help you to identify who makes the hardware so that you can download and install the necessary manufacturer utility to determine the current status.

The reason for that is so that you can find out exactly how risky an operation you are about to perform. You need to know if there is RAID at all and if so, how many disks remain in the array in a working condition.

If it is RAID 1 and you've only lost a single disk then you're not at huge risk and you can replace the failed disk and let it rebuild. It'll also tell you if it's safe to reboot.

You could also just attach an external USB disk and copy the contents of the data LV onto it. There's really not much point in backing up the system volume as you have to reinstall the system to get a working one. Back up your configuration files perhaps, but backing up the o/s itself is fairly pointless.

So first step, gather some data about the current health of the system. Your next steps will depend on the results of that.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

snidleystash
Posts: 5
Joined: 2019/01/10 19:35:32

Re: Centos 5 Upgrade - We Know We Need to Go to 7

Post by snidleystash » 2019/01/10 21:23:49

TrevorH,

Thank you for your well-stated reply, and please excuse any brevity in my response.

cat /proc/mdstat
Personalities :

lspci -nn | grep -i raid
03:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS 1078 [1000:0060] (rev 04)

This is a running system in production - and actively in use worldwide.

What isn't known is what type of RAID. There isn't documentation on it. And the local talent, who seems pretty reasonable, has already removed the faulty drive. When I asked how many physical drives, I was told "one in, and one out" of the system. So I've been figuring most likely RAID 1, but possibly RAID 10? When I asked about confirmation of the type of RAID, he said he has to reboot to determine that, and I didn't think that wise without a more thorough plan of what to accomplish on the boot.

Like I said, I've just been tossed into the fray on this one. So I appreciate any advice.

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Centos 5 Upgrade - We Know We Need to Go to 7

Post by TrevorH » 2019/01/10 22:36:20

So you don't have linux software RAID. You have a hardware RAID card from LSI Logic.

You need to download something called megacli from LSI Logic (or whoever owns them this week). Install that and you can use it to ask the RAID controller about itself without needing to reboot. I've never used it - ours are all HP or Dell controllers - and I gather its syntax is... somewhat arcane. I'm sure Google can help you to find a copy to download and provide better instructions on how to use it to do common tasks.

So with a hardware RAID controller there's also a good chance that your server has hot swappable disk drive bays and you can pull out a bad drive and replace it with a new one and the controller will autodetect that and start a rebuild automatically. Grab megacli and get it to tell you about the setup. That'll clarify whether it's RAID 1, 5, 10 or whatever. The only thing you can be fairly sure of it that it isn't RAID 0 :-)
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

snidleystash
Posts: 5
Joined: 2019/01/10 19:35:32

Re: Centos 5 Upgrade - We Know We Need to Go to 7

Post by snidleystash » 2019/01/11 14:10:28

Thank you. This is a big help. :D

snidleystash
Posts: 5
Joined: 2019/01/10 19:35:32

Re: Centos 5 Upgrade - We Know We Need to Go to 7

Post by snidleystash » 2019/01/14 14:19:34

I have been unable to locate a package with MegaCLI in it for CentOS 5.x. I assume this may be because anybody who had it pulled it off the net when CentOS 5 hit EOL. Does anyone know of a good place to look for it?

tunk
Posts: 1204
Joined: 2017/02/22 15:08:17

Re: Centos 5 Upgrade - We Know We Need to Go to 7

Post by tunk » 2019/01/14 15:07:59

Can you ask the "local talent" how many disks and their sizes?
If you combine that with the output from df -h and fdisk -l, then you may be able to decipher the raid configuration(s).

snidleystash
Posts: 5
Joined: 2019/01/10 19:35:32

Re: Centos 5 Upgrade - We Know We Need to Go to 7

Post by snidleystash » 2019/01/14 15:32:31

Tunk,
Thanks for the reply.

I appreciate the response. I'm not too worried about the size, etc, because the replacement FRU is already in hand, and the bad device has been removed. This is mostly a matter of whether the process I'd asked about can be improved on, and as a side issue, I've wanted to get MegaCLI, if possible, because it seems a worthwhile tool.

Regarding your request:

sudo df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 251G 154G 84G 65% /
/dev/sda3 97M 40M 52M 44% /boot
tmpfs 4.9G 0 4.9G 0% /dev/shm

From looking at the output, this server was probably not ideally configured, as it has everything but /boot in the root file system. I had assumed it was built differently, as using the lvm command, I found two LVs. I should have done a "df" too.

lvm> lvdisplay
--- Logical volume ---
LV Name /dev/VolGroup00/LogVol00
VG Name VolGroup00
LV UUID eyON1i-Yiu2-Kne9-LXdR-YE4T-teuC-a3Ad1o
LV Write Access read/write
LV Status available
# open 1
LV Size 258.44 GB
Current LE 8270
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol01
VG Name VolGroup00
LV UUID amDIWn-l3PN-2ogz-XDU0-1cXT-MnkB-mKdRVc
LV Write Access read/write
LV Status available
# open 1
LV Size 11.72 GB
Current LE 375
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1

User avatar
TrevorH
Site Admin
Posts: 33191
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: Centos 5 Upgrade - We Know We Need to Go to 7

Post by TrevorH » 2019/01/14 16:03:48

No-one packages megacli for any version of CentOS so as far as I am aware, it's just a matter of downloading the tar file from LSI (or their replacment) and untarring it on the server and running it.

I grabbed the version from the broadcom website having told it that I was interested in "Group: Storage Adapters, Controllers, and ICs, Family: RAID Controller Cards, OEM: , Product: MegaRAID SAS 9266-4i, Asset type: All" and inside the 8-07-14_MegaCLI.zip file there is a Linux directory that contains MegaCli-8.07.14-1.noarch.rpm and I would suspect will just install with yum.
The future appears to be RHEL or Debian. I think I'm going Debian.
Info for USB installs on http://wiki.centos.org/HowTos/InstallFromUSBkey
CentOS 5 and 6 are deadest, do not use them.
Use the FAQ Luke

Post Reply