Suggestions for simpleton moving on from CentOS 7

[OsakaWebbie]

Looking for advice or links to good tutorials. My server is currently CentOS 7, but I need to start thinking about getting something newer, and I'm afraid I won't know how, or will need a huge chunk of time and/or knowledge I don't have.

Background: I am not a server admin, but just a programmer-turned-missionary who wrote a couple of online database services for a handful of churches/ministries and try to keep them maintained as only a small part of what I do. If I could just put them on a shared hoster to outsource the upkeep, I would, but there are two distinctive things that force me to have a VPS and maintain it myself:

• To generate PDFs with UTF-8 encoding and vertical Japanese text, I use a very particular version of TeX that doesn't come standard with texlive, plus Japanese fonts. (And it's possible that shared hosters strip down their servers to the point of not having texlive at all.)
• I have a single production codebase directory, but each "client" has its own subdomain, database, and files area outside the webroot, all controlled by my PHP code. Getting the different subdomains to point to the common codebase requires special dancing in nginx config files.

The existing server (well, two - a Vagrant VM and a matching Linode VPS) was originally built in 2016 starting with PuPHPet, an online tool that builds a Puppet manifest based on the user's preferences. But I never succeeded in figuring out how to edit the resulting manifest to do the quirky things I needed, so I manually customized the server and kept good notes in case I need to do it again.

Even the code, not to mention the server, only gets part-time love from me (a very small part of my time, in fact), so I probably need to start thinking now about CentOS 7 end-of-life. Plus, there is a Wordpress website on the same server, and WP is complaining about my PHP version, and it appears that it won't be trivial to upgrade PHP (currently 7.3 - I should probably jump to 8.2 even though I'm sure I'll have to fix a lot in my code). The last time I even tried to do a simple "yum update", I got errors I don't understand, so I think I should start over with 2023 technology.

When I looked at the CentOS version landscape, I discovered that it has changed to "Stream", and CentOS and RHEL have traded places in the development stream. I'm not the "bleeding edge" type, and apparently RHEL would now be free for someone like me, but it's not in the list of Linode's supported distros, so I guess that's not an option. The internet seems to think that even headless servers are mostly using Ubuntu these days instead of the Fedora side of the Linux world, but I don't know if that's true or just where the noise is because it's so popular for desktops. (I have no plans to use a Linux desktop, so I don't care who has the best GUI.) And either way, I've never built a server from scratch.

So I know this is a very general question, but I'm looking for suggestions for my situation: how best to create a stable, secure LEMP stack with email (only my one address and almost zero actual mail - currently using Postfix+Dovecot for POP3) and other normal stuff, that could handle the quirks I described earlier and thrive on neglect. I'm the only user (logging in as root with private key authentication only; all password auth is disabled), and the amount of real web usage traffic is quite low, but my users do depend on the database services. Whatever I do, I'll probably first build it on a VM, then buy a second Linode to set up the production server, and when it's ready, move the databases, client files, and DNS.

[TrevorH]

Yes, CentOS is dead once 7 goes EOL. Stream is a permanent beta and not suitable for production use. I'd suggest moving to one of the other rebuilds of RHEL 9 like Rocky/Alma/OEL. You can also sign up for a free RH developer subscription which comes with a free license to run up to 16 instances of RHEL itself. That may be your best option. The devsub expires after a year but can be renewed by visiting developers.redhat.com and clicking on the button that says "Renew" but only _after_ it expires. You then have to do some stuff with subscription-manager on your installed systems to get them re-registered.

I'd skip RHEL 8 and go to 9.

[OsakaWebbie]

Thanks, Trevor. Yes, I know I could sign up for RHEL, but Linode doesn't support it (or OEL) for deployment. (Plus, I wouldn't remember to keep RHEL registered - I don't want to mess with that.) So I turned my attention to Alma and Rocky, but while trying to learn about the differences between them (which seem very subtle), I ended up also learning about the recent drama with RedHat over access to downstream source code - yikes! Short-term, the Alma and Rocky folks say they're coping, but long-term seems like a big question mark, and I don't want to have to do another rebuild in a few years. I just don't have time or skills to be a real sysadmin.

Your signature mentions that you're thinking of going (or have already gone) to Debian, and some people say that's best because no for-profit company can strangle it. I'm definitely a proponent of open source and non-profit, but other people say that Debian is not the best to use on a server, particularly if you need it to be secure out-of-the-box because your enterprise requires it, or in my case because I wouldn't know how to harden it myself without a lot of study. So confused! I'm curious, why did you only suggest the RH ecosystem for my case? (Yes, I know this is a CentOS forum. But your signature...)

Then, no matter what I go with, I'll have to figure out how to make a VM out of it without a tool making a Puppet manifest for me. And even though I have not re-provisioned it in seven years, magic happens every time I say "vagrant up": forwarded ports, network interfaces, mounting shared folders, and a bunch of other stuff I don't understand. I'm trying to read Vagrant tutorials and docs, but it's slow going, because my Vagrantfile doesn't look anything like their examples, instead pointing to a labyrinth of Ruby, YAML, and JSON files that PuPHPet made.

[jlehtone]

Then, no matter what I go with, I'll have to figure out how to ...

Yes, that indeed is the truth (and the hard part). Whichever system we choose to use,
we have to learn somewhat to use. Even the RHEL 9 differs (in details) from RHEL 7.
However, if you are a bit familiar with RHEL (CentOS), then el9 is easier to learn than Debian.

It is true that Red Hat might do further actions, but both projects (AlmaLinux and Rocky Linux)
seem to have determination, community, and some funding to go on. Both might not survive
for eternity, but as long as their content remains similar a side-grade (like there was from
CentOS 8 to Alma 8, Rocky 8, etc) should be trivial.

Red Hat does support Ansible; RHEL docs have sections for it and "System Roles" package
is in the base distros. That (Puppet to Ansible) would be an another leap if Ansible can do what
you need "better" than Puppet. More or less than updates to your current Puppet (particularly
if you choose Debian)? Alas, I don't know whether Ansible can do better.
Isn't one of the benefits of automation (such as Puppet or Ansible) that one can reinstall easily,
i.e. more often? That is my experience.

What is the life-cycle of Debian? The RHEL (and look-alikes) run for a decade, the 9 to 2032,
unless things go really wrong. Does Debian require more than equivalent of regural
"yum update & reboot" more frequently than that?

[OsakaWebbie]

However, if you are a bit familiar with RHEL (CentOS)...

"Familiar with" might be stretching it, but perhaps a slight head start since I have notes about what I did last time, which include a few yum commands rather than apt, and some configuration files (although those are mostly for specific packages, not OS-level stuff). And things I learned along the way I soon forget and have to research anew every time. But okay, I'll give AlmaLinux 9 a shot. I'll lose this great community, though.

Both might not survive for eternity, but as long as their content remains similar a side-grade (like there was from CentOS 8 to Alma 8, Rocky 8, etc) should be trivial.

As I said, RHEL is not supported on Linode, so I don't know what I would side-grade to. Hopefully RedHat and the open-source community will work things out and Alma 9 will last awhile.

More or less than updates to your current Puppet (particularly if you choose Debian)? Alas, I don't know whether Ansible can do better.

I would have no clue how to update my current Puppet - it was auto-generated by PuPHPet and is quite complex. The definitions of stuff needed for both Vagrant (e.g. shared folders) and Puppet (e.g. packages, users, nginx basic config before my special stuff) are in a 557-line YAML file, but I don't know how much of that I actually need, and the code that uses those definitions and does the real work is sprinkled all over the place. The intent of PuPHPet was that the user would go back to the online GUI to get an updated ZIP if they wanted to change something as significant as the distro, but the project was abandoned a few years ago. And even back in 2016/2017 I tried to add simple things like install the correct latex tools/fonts and set up my special nginx folder structure, but I never succeeded in getting it all to work, even though there is a set of folders provided for adding shell scripts at various points in the process. So I should let PuPHPet RIP and start from scratch. For the provisioning part, it's probably not worth me learning Puppet or Ansible, and I suspect I would never figure out how to fold everything I need into those tools, so I think I should just stick with bash commands. If I'm really clever, perhaps I can turn the commands into a shell script that can be used as the provisioner, but I doubt it will be that linear of a process.

Isn't one of the benefits of automation (such as Puppet or Ansible) that one can reinstall easily, i.e. more often? That is my experience.

Yup, that's the theory. But for that to work, one must figure out how to write everything into the manifest/playbook. I never succeeded in doing that, so instead I have several pages of notes of what else needs doing manually. If I had never used PuPHPet, it would have been harder then but easier now, because my notes would contain ALL the steps to make a server, not just what to do after all the PuPHPet-generated stuff.

[jlehtone]

With Ansible one can replace some of the scripts with plays. That is what I did.

Code: Select all

- hosts: all
  vars:
    packages:
    - texlive
    - rsync

  tasks:
  - name: Install packages
    ansible.builtin.dnf:
      name: "{{ packages }}"
      state: present

This play does make sure that the packages listed in variable packages (texlive and rsync)
are installed on the system. If they are not, then they get installed. If we run the play again,
nothing is changed because both packages are already installed. If you notice that you need some
additional package, then simply add to the list and run that play again.

If you have a script and run the ansible-playbook command on the host that you are configuring,
then you could have a "run script" as a task:

Code: Select all

  - name: Run myscript
    ansible.builtin.command: /home/osakaweb/myscript.sh

Alas, that runs the script every time you run the play, so the script would repeat its actions.

As I said, RHEL is not supported on Linode

Obviously, whatever your service provide (Linode) does support is the subset of distros that you must choose from.

[OsakaWebbie]

Grr! This is the second time I didn't get notified when someone responded on this thread - the forum notification system is only batting .333 so far (no, yes, no). Sorry for the slow response - I don't think to continuously check for new posts, assuming instead that the system will tell me.

With Ansible...

Interesting that you wrote about Ansible, because probably about the same time you were writing, I was taking a closer look at Ansible. Since I'm having so much indecision about distros, provisioning tools would be more distro-agnostic in case I change my mind. At first, Ansible sounded appealing because it seems lighter weight than the others, with nothing special needing to be installed on the server. But then I discovered that the Ansible "control node" has to be running *nix or WSL. and I would prefer to keep my Windows PC clean without that overhead for something I would only use to run Ansible. I need a Linux VM, but only as a webserver for testing, not as a "hands-on" OS - I only run it when working on my webapps, which is a small percentage of my time. And there will still be many things I'd have no idea how to program into one of those provisioning tools, so I'd either spend a ton of time in the learning curve or just do them manually anyway.

Anyway, I then went back to looking at distros. Due to the resoundingly bigger Debian/Ubuntu usage percentage, I'm always running into tutorials, articles, etc. that are written for that family of distros (apt instead of yum, etc.), and I love the support of being in the biggest community. So I began asking myself, "For someone like me who doesn't go deep anyway, would switching from the RH ecosystem to Debian/Ubuntu and setting it up with CLI really be that hard?" Comparing Debian and Ubuntu, particularly for a VPS with limited RAM (and not wanting my VM to steal too much RAM from my PC either), this article convinced me to look first at Debian for my needs. I have now created a VM with Debian 12 and installed a few basic things. I am having trouble just booting it sometimes, and I can't yet SSH to it any way besides `vagrant ssh`, so I can't copy files to it yet (ideally I'd use rsync to copy between from the old VM, but so far that's not possible). Every step in the process has required some amount of research (from just a few minutes to still-stumped-after-two-full-days), but not because it's Debian instead of CentOS - my troubles so far are at the vagrant setup level, I think. I'll keep pounding away at it.

[jlehtone]

But then I discovered that the Ansible "control node" has to be running *nix or WSL.

One does not need a special "control node". One can copy the "playbooks/inventories" (they are mere text files) into Linux system, ssh there, manually install ansible, and then run with "localhost" as the host that you manage. There is also "ansible-pull" that downloads the playbook from (git) repository and applies it to localhost.

Every step in the process has required some amount of research (from just a few minutes to still-stumped-after-two-full-days), but not because it's Debian instead of CentOS - my troubles so far are at the vagrant setup level, I think. I'll keep pounding away at it.

"Whatever works" is a solid practical approach. (And one can make an Ansible playbook for Debian too.)

[TrevorH]

There's also ansible-playbook which connects to the remote machines via ssh and applies playbooks that way. It's not really a "control node", just somewhere central you store your playbooks and then roll out round the machines you control.