After upgrade to 7.4, finding an assortment of little issues

General support questions
OsakaWebbie
Posts: 5
Joined: 2017/10/11 11:19:21

After upgrade to 7.4, finding an assortment of little issues

Postby OsakaWebbie » 2017/10/12 13:17:21

I have a server on Linode that was born about six months ago and was running CentOS 7.3 until recently. Some of the server stuff (nginx, MariaDB, PHP, etc.) was built using a set of Puppet manifests generated by PuPHPet, if that matters. I'm more of a programmer, not a server admin, so I don't know what users various services were running as, or file ownership of system files - I was focusing on migrating the web apps I had built on a much older server. A couple weeks ago I ran "yum update", which updated over 200 packages, including the OS to 7.4, and that's when things changed. I have this haunting feeling that a lot changed, but I have no idea how to check. (For the overview of my system, the output of "getinfo.sh package" is at: http://pastebin.centos.org/358891/)

The first thing I noticed (because all my web app users screamed) was that PHP sessions suddenly didn't work. Previous to the update, I know that Nginx was running as user "www-user" (in the www-data group), but afterwards it was running as "nginx" (in the nginx group). In addition, I don't know what it was like before, but after the update, the /var/lib/php/session directory's ownership was "root:apache". Weird, since Apache isn't even installed! Even "www-user" was not in an "apache" group, so I know that either the owner or the group (or both) changed. I told Nginx to run as "www-user" again (to hopefully reduce the number of other surprises), and changed that directory to "root:www-data" ("www-data" is the primary group of "www-user") - that works, but I don't know if it's correct or not.

Then I noticed that my one Wordpress website located on that server was no longer successfully concatenating CSS files (the output would abruptly stop after 46,140 bytes regardless of what files were being combined). I turned that feature off as a band-aid, but I assume that something about the server update caused it to stop working, and I'm completely stumped as to what could cause that symptom. I also fear that other PHP processes might have the same problem but I haven't stumbled upon them yet.

Then, in the process of troubleshooting something else, I noticed that the hostname in all the system logs (cron, messages, maillog, etc.) changed from "puphpet2" (the name I gave to this Linode when I created it) to "mail" (which seems like a weird hostname to me - I'm quite sure I didn't change it to that). I confirmed with the "hostname" command, which reported "mail.mydomain.tld". It is true that the same weekend as the update, I was setting up postfix and dovecot, but not only should that not have anything to do with the hostname, but the hostname listed in the logs changed exactly when the server rebooted as part of the update. I'm assuming that an odd hostname is harmless, but it makes me think that the changes caused by that update run pretty deep.

I know this is a broad question, but does anyone know:
  • why those types of things, like directory ownership or hostname, would change during an update to the OS and packages?
  • what else I should look for (or even how to look - I have tried to look for oddities in logs, but nothing seems obviously errant)?
I'm aware that some of my system configuration was probably done by Puppet, not out-of-the-box CentOS (I can't remember whether I updated to 7.3 before or after running Puppet), and I don't expect you to envision my whole system, but if you can suggest ways to look for problems, that would be a help.

User avatar
TrevorH
Forum Moderator
Posts: 20962
Joined: 2009/09/24 10:40:56
Location: Brighton, UK

Re: After upgrade to 7.4, finding an assortment of little issues

Postby TrevorH » 2017/10/12 15:07:34

We don't supply nginx at all so any issues with that would be down to whoever you got your nginx package from. I know it's in EPEL but nginx also run their own yum repo that can be used too.

The /var/lib/php/session directory is owned by hte package "php" so if you have php installed and it gets updated then that will change back any ownership changes that you have made. Since the php package supplies the php module for use with Apache httpd and doesn't get used for nginx - which I believe uses phpfpm - you probably don't even need that package installed at all.

Hostname may very well be determined by doing a reverse DNS lookup on the ip address used by your server. If you run hostnamectl and set the hostname that way then it should be persistent.
CentOS 5 died in March 2017 - migrate NOW!
Full time Geek, part time moderator. Use the FAQ Luke

OsakaWebbie
Posts: 5
Joined: 2017/10/11 11:19:21

Re: After upgrade to 7.4, finding an assortment of little issues

Postby OsakaWebbie » 2017/10/16 16:26:46

Thanks for your reply, Trevor. I finally got some time to look at my system with your comments in mind.

TrevorH wrote:We don't supply nginx at all so any issues with that would be down to whoever you got your nginx package from. I know it's in EPEL but nginx also run their own yum repo that can be used too.

I simply told PuPHPet that I wanted nginx, and it set things up to install it. I don't understand Puppet manifests very well, but digging into the files, I think I found the relevant code snippet:

Code: Select all

      if $manage_repo {
        yumrepo { 'nginx-release':
          baseurl  => "http://nginx.org/packages/rhel/${os_rel}/\$basearch/",
          descr    => 'nginx repo',
          enabled  => '1',
          gpgcheck => '1',
          priority => '1',
          gpgkey   => 'http://nginx.org/keys/nginx_signing.key',
          before   => Package[$package_name],
        }

        file { '/etc/yum.repos.d/nginx-release.repo':
          ensure  => present,
          require => Yumrepo['nginx-release'],
        }
      }

Guessing from the baseurl, I assume that's a repo directly from the nginx people instead of EPEL. I don't know the implications of that in terms of strange changes to my system.

TrevorH wrote:The /var/lib/php/session directory is owned by hte package "php" so if you have php installed and it gets updated then that will change back any ownership changes that you have made. Since the php package supplies the php module for use with Apache httpd and doesn't get used for nginx - which I believe uses phpfpm - you probably don't even need that package installed at all.

With thousands of packages, most of which I've never heard of, I have no idea what is needed. "yum list | grep '^php' | wc -l" says there are 2449 packages starting with "php" - if the one named just "php" turns out to be needed by something else, by removing it I will have broken my production web server. Too scary for me! I just wish I knew what mischief it might do when updating (besides the one I learned the hard way). Or is there a way to know what packages "own" directories like you said?

TrevorH wrote:Hostname may very well be determined by doing a reverse DNS lookup on the ip address used by your server.

Apparently not in my case, if what you mean by reverse DNS lookup is "nslookup xxx.xxx.xxx.xxx". That returns "lixxxx-xxx.members.linode.com." (the "li" and 7 digits are probably the default hostname given by Linode, which I changed after startup).

TrevorH wrote:If you run hostnamectl and set the hostname that way then it should be persistent.

That's exactly what I did, months ago, to change it to "puphpet2". (I distinctly remember following that instruction at https://www.linode.com/docs/getting-sta ... e-hostname) I'm quite sure that I never intentionally ran "hostnamectl mail". Could some process that is part of setting up postfix and dovecot (including using webmin to work on them, because partway through the process I decided I was not smart enough to do it manually, so I added webmin and started using it) change the hostname indirectly? I do remember having some difficulty getting email to work, particularly POP3, and of course my DNS zone does have a record for mail.mydomain.tld. Although the moment that it changed in the logs was during the update, perhaps it was something that was changed before but didn't take effect until the machine was rebooted.

Anyway, my reason for posting here was not so much to diagnose the couple of changes I know about (although that's helpful, too), but to ask whether there are ways of figuring out what other changes might have occurred - is there a log of steps taken during updates or something?

jimj
Posts: 44
Joined: 2014/10/01 05:34:57

Re: After upgrade to 7.4, finding an assortment of little issues

Postby jimj » 2017/10/21 15:15:40

How to use yum history might have some helpful info.

OsakaWebbie
Posts: 5
Joined: 2017/10/11 11:19:21

Re: After upgrade to 7.4, finding an assortment of little issues

Postby OsakaWebbie » 2017/10/21 16:14:01

That link went to a page that was mostly about undoing things. But it taught me the existence of "yum history", and I had high hopes.

But I'm not getting much information from it. For example:

Code: Select all

[root](00:51:45)[~]$ yum history list php-fpm
Loaded plugins: priorities
ID     | Command line             | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
    45 | update                   | 2017-09-30 09:31 | I, O, U        |  329 EE
    40 | update                   | 2017-08-28 23:17 | I, U           |   71 EE
    19 | -d 0 -e 0 -y install php | 2017-04-05 17:19 | Install        |    3
history list

That's not much information - there is only one line about each update or installation. Even "-v" doesn't help. And for some packages it doesn't even work - I know there is a "php" package, but I get this:

Code: Select all

[root](00:56:50)[~]$ yum history list php
Loaded plugins: priorities
Bad transaction IDs, or package(s), given
Error: Failed history list

Am I missing a clue? I want to know the actual actions taken by a given update (in particular, actions besides just unzipping and copying the files).

jimj
Posts: 44
Joined: 2014/10/01 05:34:57

Re: After upgrade to 7.4, finding an assortment of little issues

Postby jimj » 2017/10/21 17:10:54

OsakaWebbie wrote:Am I missing a clue? I want to know the actual actions taken by a given update (in particular, actions besides just unzipping and copying the files).

The only other thing I can think of is the --scripts rpm option (I don't know if yum has a similar option). Below is an example looking at gedit's scripts. Good luck.

Code: Select all

$ rpm -q --scripts gedit
postinstall scriptlet (using /bin/sh):
update-desktop-database >&/dev/null || :
touch --no-create /usr/share/icons/hicolor >&/dev/null || :
postuninstall scriptlet (using /bin/sh):
update-desktop-database >&/dev/null || :
if [ $1 -eq 0 ]; then
  touch --no-create /usr/share/icons/hicolor >&/dev/null || :
  gtk-update-icon-cache /usr/share/icons/hicolor >&/dev/null || :
  glib-compile-schemas /usr/share/glib-2.0/schemas >&/dev/null || :
fi
posttrans scriptlet (using /bin/sh):
gtk-update-icon-cache /usr/share/icons/hicolor >&/dev/null || :
glib-compile-schemas /usr/share/glib-2.0/schemas >&/dev/null || :

OsakaWebbie
Posts: 5
Joined: 2017/10/11 11:19:21

Re: After upgrade to 7.4, finding an assortment of little issues

Postby OsakaWebbie » 2017/11/09 12:07:53

As I feared, the few problems I mentioned earlier were not the only ones. I have now discovered that I can't send mail via an email client. Today was the first time I tried since a few days before the fateful yum update, so I can't prove the failure happened that day, but since I haven't changed anything else on my server (or in Thunderbird), the update is very likely to be the cause. Postfix was apparently not one of the packages updated, so the cause is mysterious. See viewtopic.php?f=47&t=64972 for the specifics of that problem, and please lend a hand if you have troubleshooting ideas for it.

I tried your suggestion of "rpm -q --scripts <package>", but since there were 200+ packages, I don't know for what packages I should comb through scripts I don't really understand.

Webmin now tells me that once again, 68 packages have updates available. Notable programs recognized in the list this time are the kernel, MariaDB, nginx, and PHP. The 8 packages related to nginx are security updates, so I shouldn't ignore them, but after my last update experience, I'm scared - will things get better, or worse? :?