Today I’m trying to switch a server system (dns, mail, web and other fairly generic services) from CentOS 8 Stream to Alma 9 doing an in-place upgrade. The goal of this activity is not making an upgrade while maintaining business continuity (although if you are in a virtual environment and/or can clone your server, the downtime may be minimal). The goal of this activity is to minimize the total effort involved in the upgrade, even if this could imply some downtime.
As you may imagine, this one is not for the faint of heart. The procedure is not documented or recommended anywhere so if your system is critical, then it’s wiser to follow an official, tested upgrade path. Take your time, install a new system from scratch, port and test everything needed etc. But if you feel confident with your Linux box and with Linux system administration in general, maybe you can give this a try. In my case, after some tweaking it worked a lot better tha I ever expected it to.
One issue I thought I had to deal with is with packages which are built for el8, but are not ready for el9 (for example, my own Lux and Frank ones). Or simply, packages which are renamed between CentOS 8 and Almalinux 9. You’ll have to swap them. Theese packages could also pose a dependency issue, which might make the upgrade fail. So I decided to temporarily remove a bunch of packages, just to reinstall their el9 counterparts later, when the Alma 9 system will be born.
My choice was to remove my Lux and Frank packages, and the packages from the remi repos, which I use to have some updated PHP versions tu run my web services.
So I came up, from a sw repo point of view, with a CentOS/Epel system, which I was able to upgrade in about one single step.
Then, all the optional packages can be reinstalled. Rpm will help us a lot, leaving behind .rpmsave files which we will recover later.
I recommend keeping track of every command and especially the output of these commands, as you may have to review them later. Maybe, recording all the tertminal session would be a good idea.
Please note that in almost every step of this guide Your Mileage May Vary, depending on what you have installed on your system. This is only a summary guide, so don’t follow it blindly.
As usual first af all backup, backup, snapshot etc etc.. you already know.
First, disable all the modules. It seems that in el9 modules, even if still supported, where reduced to the bare minimum, i.e. no modules at all. So let’s disable them.
# dnf module disable "*"
...
Complete!
Then I had a look at the output of package-cleanup --leaves and package-cleanup --orphans (the package-cleanup command is in the yum-utils package, install it if needed). They revealed some apical packages I don’t really need, and I removed them just to keep things as simple as possible.
As I wrote before in my particular server I have some of software packages from my Lux and Frank C8 repositories and other optional repositories like for example remi. I’m going to remove them and reinstall when we will be in Alma. You are not forced to do this way. Ideally, you can swap repositories between the 8 and 9 ones, like what we’re going to do for Epel. But since this is my first time with the procedure, I preferred to keep things simple and manageable.
Moreover, my packages do not have to date their counterparts ready on Alma 9. Indeed I intend to use this particular server to build at leastthe more essential ones after I upgrade it 🙂 So I removed them. Again, keep a clear complete note of what you’re doing so you can easily dnf install them later. For remi, I simply uninstalled all their sw: I’ll easily reinstall it when we’re done, and it won’t bother us when we’re in the delicate part of swapping C8 with A9.
So, for example, keep note of the output of:
rpm -qa|egrep '\.el8\.remi\.[0-9.]*(x86_64|noarch)$'
and then:
rpm -e `rpm -qa|egrep '\.el8\.remi\.[0-9\.]*(x86_64|noarch)$'`
This and the following commands are likely going to leave behind a bunch of .rpmsave. You’ll have to review them when you’re going to reinstall them.
Or for Lux and Frank:
rpm -qa|egrep '\.el8\.(lux|frank\.)'|while read ppp
do
echo $ppp
rpm -e $ppp
done
Finally, delete the old C8 repository references, and other alike (again, YMMV: take a look at which *-release package you have).
rpm -e epel-release epel-next-release rpmfusion-free-release lux-release
and we need to get rid of theese two CentOS packages. They’re not going to come with us. (dependencies will be newly satisfied later with the corresponding Alma packages):
rpm -e centos-stream-release centos-stream-repos --nodeps
Now I would take note of what packages are still there:
rpm -qa | sort
rpm -qa --qf '%{NAME}\n' | sort
If you are on a VM, maybe this is a good place to make a snapshot.
Install the new infrastructural repositories:
dnf install https://repo.almalinux.org/almalinux/almalinux-release-latest-9.x86_64.rpm \
https://repo.almalinux.org/almalinux/almalinux-repos-latest-9.x86_64.rpm \
https://repo.almalinux.org/almalinux/almalinux-gpg-keys-latest-9.x86_64.rpm
and
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm \
https://dl.fedoraproject.org/pub/epel/epel-next-release-latest-9.noarch.rpm
Note that it said:
Many EPEL packages require the CodeReady Builder (CRB) repository.
It is recommended that you run /usr/bin/crb enable to enable the CRB repository.
You’ll better enable CRB later.
Finally we issue
dnf --releasever=9 --allowerasing --nobest --setopt=deltarpm=false distro-sync
Some cleanup:
restorecon -Rv /var/lib/rpm
dnf clean all
Maybe you’ll note a disturbing warning from dnf about rpm using sqlite instead of dbd now. Let’s convert the db now:
rpmdb --rebuilddb
warning: Converting database from bdb_ro to sqlite backend
I was waiting for this
init 6
By the time, if everything went as expected, you’re on Alma 9. But you’ll likely still have some el8 packages. See them with, for ex. rpm -qa|egrep '\.el8\.' . You’ll have to decide whether to delete, update or replace them, depending on the case. I list here the cases I had to take care about. It’s very simple, indeed.
Simply delete old CentOS materials:
rpm -e centos-gpg-keys centos-logos
This one raises a dependency issue, but since we’re going to fix it with the second command, let’s use –nodeps
rpm -e centos-logos-httpd --nodeps
dnf install system-logos-httpd system-logos
Just as examples, I found some el8 packages that do not seem to have a counterpart on el9 at the moment. I’ll have a look at them later, maybe I’ll recompile some. For now, I’m going to simply remove them, just to keep the system as clean as possible. Obviusly, your list may be different.
rpm -e GeoIP GeoIP-GeoLite-data iptstate aspell autogen-libopts
rpm -e fipscheck fipscheck-lib
I had to enable the almalinux-crb repository to be able to update this package:
dnf update sendmail-milter
.. and so on. The list depends on your system, but the procedure is strightforward.
My system now had installed 3 kernels, as usual. The newer one, which is the one in use now (check with uname -r) is from Alma 9; the other two are from the older C8. I removed the old ones. This is not strictly necessary since they should be eliminated one by one as you install new kernels in the normal day-by-day update procedure of your machine.
If you prefer to remove them now, find the packages to delete with something like rpm -qa|egrep '4\.18\.0-[0-9]+.*\.el8\.x86_64' (check the output, some other packages could sneak in that very coarse regexp).
Reinstall the packages I deleted at the beginning of this guide. I had to find the *.rpmsave files in /etc and /var that where left behind at that point, and possibly reapply the differences to the real config files.
By the time, surprisingly enough, the system runs smoothly. All dependencies are satisfied, there are no warnings, no particular SELinux issues etc. I feel I spared myself a lot of work reinstalling a new server, porting data and configus etc. Try it if you like, it worked for me and I was happy with it.