Can't login after cloning disk. (SELinux)

[bloxxom]

I'm working on a project to clone a standard install onto many servers. I have a 'golden' server that I installed from the CentOS iso, and then added some customizations. Then I created a tar archive of the 3 filesystems /, /boot, and /EFI.

I have a program (Like a specialized kind of rescue install) that I can run on the target system that creates partitions and then unpacks the tar archive accordingly. I modify the grub.cfg file, and the fstab, to insert the UUIDs of the partitions on the target system.

When I boot the target, it boots up normally, but I can't log in either on the console or via ssh although I can do both these things on the 'golden' system.

I get a login and password prompt, and it looks like it's going to log me in, but instead it just gives me the login prompt again. When I try to login via ssh, it complains "/bin/bash: Permission denied" and closes the connection.

There are about 29,000 files on the 'golden' system, and I've verified that the files on the target match them via md5sum. I've verified that all the files have identical permissions and ownership to the corresponding files on the 'golden' system. So as far as I can tell, the content of all the filesystems is identical (except for grub.cfg, fstab, and a couple other obvious files).

Purely because I ran out of ideas, I decided to try disabling selinux on the target system, by editing /etc/selinux/conf and setting SELINUX to disabled. This allowed me to log in, and everything appears to be working, both logging in to the console, and via ssh.

So, I guess there's something about the hardware on the 'golden' system that's encoded into some file, and it conflicts with the same hardware on the target system. Maybe a MAC address, or CPU identifier (I don't know much about selinux). Or maybe those UUIDs are encoded somewhere else that I've missed.

Is there a way to reset and re-enable selinux on the target so it will allow me to log in in this new environment? Even though I can't login, I can boot my rescue program and make modifications to the filesystem, which is how I was able to disable selinux.

[gulikoza]

SELinux stores additional file permissions (called context) and tar does not save/restore that information (some other programs, like dar will). The files are not labeled with correct SELinux context so the rules are not working correctly.

You have several options:
- use a backup/restore procedure that will store and restore SELinux labels
- disable SELinux (set it to permissive mode) for the duration of restore procedures
- add /.autorelabel file to the tar - this will automatically run relabeling after first boot and fix SELinux contexts

Also, cloning a large amount of servers is not really the best option from security point of view. Aside from the problems you have already found out about (SELinux), all the systems will have the same ssh keys and other information that is usually generated on first boot. A better solution would involve creating a kickstart file and automatically installing the servers and push the customizations with a custom rpm or some other provisioning software (ansible, puppet, ...)

[TrevorH]

For info, it takes around 10 minutes for me to install a fresh system from scratch using a kickstart. Give that another 10 minutes for yum update afterwards and a puppet run to configure everything that needs it and I can get a fresh system online in about 20 minutes. It'll be installed correctly, it'll have selinux on, it'll be set up for the role it was intended for and it'll be ready to go.

[bloxxom]

SELinux stores additional file permissions (called context) and tar does not save/restore that information (some other programs, like dar will). The files are not labeled with correct SELinux context so the rules are not working correctly.

Tar actually does have the option to preserve the extended attributes. However, the software I'm using for packing and unpacking it uses a kernel without selinux. I can't change this in the short term, though I do plan to update it in a few months.

You have several options:
- use a backup/restore procedure that will store and restore SELinux labels
- disable SELinux (set it to permissive mode) for the duration of restore procedures
- add /.autorelabel file to the tar - this will automatically run relabeling after first boot and fix SELinux contexts

I can probably just complete the install without selinux, as these images will be overwritten before they ship anyway. The install I'm cloning is to be used by our offshore manufacturer during initial test. However, I'd like to move our organization towards enabling selinux over time, so I'd like to get it enabled on these systems if possible.

The .autolabel idea does work, thanks I wasn't aware of that option. It seems to reboot after the labeling is done... is that expected? It's not taking a huge amount of time to complete.

[bloxxom]

For info, it takes around 10 minutes for me to install a fresh system from scratch using a kickstart. Give that another 10 minutes for yum update afterwards and a puppet run to configure everything that needs it and I can get a fresh system online in about 20 minutes. It'll be installed correctly, it'll have selinux on, it'll be set up for the role it was intended for and it'll be ready to go.

Thanks for the response. I'm finding the kickstart option to be very slow. For comparison, using the clone method, it takes 2.5 minutes from the time the kernel begins booting, until the time it begins rebooting after the clone is complete. After that, I have no further steps to perform because the image I'm cloning already has all the required packages installed.

When I use a kickstart file to drive a regular install, it takes 7 minutes from the time the kernel begins booting until I see the anaconda screen. Then anaconda appears to do nothing for another 6.5 minutes. Then it starts stepping through the install, and takes a further 10 minutes before it reboots into the installed system. So it's a total of about 23 minutes, 10 times longer than what the clone takes. This directly impacts the time it takes our contract manufacturer to get these servers through initial test, and therefore the number they can complete per day.

[gulikoza]

The .autolabel idea does work, thanks I wasn't aware of that option. It seems to reboot after the labeling is done... is that expected? It's not taking a huge amount of time to complete.

Yes, it will reboot after relabel, it's normal.