Tips for a Successful Red Hat Satellite 6.6 Upgrade

Red Hat Satellite 6.6 was released on 22 October 2019. The official release notes for the release can be found here. Time to upgrade?

I recently upgraded a server from Satellite 6.5.3 to 6.6.2 and thought it was worth sharing some tips in case others find this useful. If you are currently on an older release, you might find the following posts useful:

What’s new

One reason for upgrading is to keep your Satellite up to date, fully supported and able to take advantage of the newest features. Another big plus is the integration with Composer which allows you to build images using Satellite content. Satellite itself still cannot run on RHEL 8 (that capability will likely come in a future Satellite release, perhaps only with Satellite 7) but with many enterprises looking to deploy RHEL 8 it makes sense to upgrade when you can. The Red Hat Satellite Product Life Cycle shows that Satellite 6.5 currently moved out maintenance support in April 2020 so upgrading now isn’t a bad idea.

Some other highlights from the release notes:

Satellite 6.6 Release Notes
Red Hat Satellite 6.6 supports Ansible version 2.8
Support for Ansible Runner. Ansible Runner is the recommended way to run Ansible jobs instead of calling ansible-playbook
Deploy OpenSCAP with Ansible and reporting on hosts without using Puppet
New Task dashboard and notification
Reporting engine enhancements
Scaling enhancements – if you are running a large environment you can scale Satellite at install/upgrade time.

For a complete list of components that make up Red Hat Satellite, the Satellite 6 Component Versions shows us that Satellite 6.5 to 6.6 has minor updates to many of components including Pulp and MongoDB. Note that that these are smaller updates, rather than any major version bumps. This should reduce the risk of encountering upgrade bugs.

Whilst looking at the above list, it’s worth mentioning that MongoDB will be removed in the future – Red Hat Satellite to standardize on PostgreSQL backend. For now though, MongoDB it is!

Planning

Take time to read the documentation. There are 4 useful resources as a starting point:

Check the storage requirements before you begin as the requirements for some filesystems have may changed since your server was built. This applies to both the Satellite server and any Capsule servers – don’t forget to allocate space for /var/spool/squid on the Capsule!

Check the amount of CPU and memory you have allocated to your Satellite and Capsule servers. Over time the minimum requirements have increased so it’s worth reviewing.

Satellite Version	Install Guide	*CPU Requirement()**	*Memory()**
6.0	Satellite 6.0	4 CPU	12 GB
6.1	Satellite 6.1	4 CPU	16 GB
6.2	Satellite 6.2	4 CPU	16 GB
6.3	Satellite 6.3	4 CPU	20 GB
6.4	Satellite 6.4	4 CPU	20 GB
6.5	Satellite 6.5	4 CPU	20 GB
6.6	Satellite 6.6	4 CPU	20 GB

(*) Minimum values. There is also a note on Increasing promote/publish speed in Satellite 6 which details memory considerations depending on the size of your MongoDB database.

If you have a Red Hat Support contract, take the time to raise a proactive support case. If your upgrade fails it’s useful to have support on hand so they can quickly review what’s happened. Sometimes a second pair of eyes looking at an error message can be really useful. For more details, see How to open a proactive case for patching of upgrading production Red hat Satellite and Capsule.

Choose an appropriate time to perform the upgrade. The Satellite and Capsule services will be stopped during the upgrade so you want to avoid times when others may be using the server (e.g. patching their hosts, provisioning new servers). Another useful tip, make sure you’ve set aside dedicated time in your own calendar for the upgrade so you can focus on it. Trying to the do the upgrade at the same time as answering support calls or doing other project work is asking for trouble!

Draw up a checklist of actions that you are going to perform as part of the upgrade and use it! If you get distracted or something needs verifying it’s easy to lose track of where you got to.

Be prepared to postpone your upgrade. There are many things outside your control. For example, the next Spectre or Meltdown vulnerabilities may be just around the corner, just at a time when you’ll need Satellite available to help you patch. An unrelated failure in your infrastructure may need your focus, or you may simply get sick. Having a working 6.5 Satellite is probably more beneficial than a partially upgraded 6.6 Satellite!

Pre-Upgrade Checks

Satellite 6.5 is a prerequisite for the upgrade. This is the same as the previous releases – from the Red Hat Satellite Ask Me Anything Q&A from April 2019 that “Any .Z version of 6.4 can upgrade to 6.5, there’s no requirement to go to 6.4.3.” If you upgraded to Satellite 6.5 some time ago (as early as May 2019 according to the Red Hat Satellite Release Dates) and not performed any updates since that time, you might find there are a large number of O/S patches to apply. If you can get a reboot maintenance window to bring your server fully up to date it’s a good idea. Should you need to rollback during the upgrade at any point, you’ll be able to restart without needing to re-apply all those O/S updates again.

Run the “upgrade check” option in foreman-maintain to check that your Satellite is in a good state to upgrade to Satellite 6.6. This can be run online without interrupting Satellite services.

yum update rubygem-foreman_maintain
foreman-maintain upgrade check --target-version 6.6

If you don’t perform housekeeping on Satellite to remove old tasks, the “foreman-maintain upgrade check” command can remove paused or stopped task(s) older than 30 days. This saves on database space.

The upgrade check may show the following message:

Check for recommended disk speed of pulp, mongodb, pgsql dir.: 
| Finished                                                                      

Disk speed : 33 MB/sec                                                [WARNING]
Slow disk detected /var/lib/pulp mounted on /dev/mapper/vg_dhcp201-lv_root.
             Actual disk speed: 33 MB/sec
             Expected disk speed: 60 MB/sec.
WARNING: Low disk speed might have a negative impact on the system.
See https://access.redhat.com/solutions/3397771 before proceeding

If that happens it’s likely to be due to Bug 1641784 – Disk performance task during foreman-maintain upgrade check fails with low disk speed when using the latest version of rubygem-foreman_maintain (0.2.11-1) The following Red Hat knowledgebase article also has details: Red Hat Satellite upgrade check from 6.3 to 6.4 fails at “disk-performance” Don’t panic! If that happens add the whitelist option to foreman-maintain. For example:

yum update rubygem-foreman_maintain
foreman-maintain upgrade check --target-version 6.6 --whitelist="disk-performance"

You may also see the following:

Check for roles that have filters with multiple resources attached [FAIL] There are user roles with inconsistent filters

I had the foreman-maintain script automatically fix the errors, but it wasn’t immediately obvious how my server had gotten into this state. There is a related Bugzilla for this – Bugzilla 1703041[RFE] verbose output for failing upgrade checks – which suggests enhancing the diagnostic message – what roles have this error and what is the script doing to correct this?

Have some test machines connected to your Satellite and Capsule before the upgrade which you can use to compare yum and puppet behaviour before and afterwards. Ideally, have these in different lifecycle environments. You don’t have to go crazy, just a small, representative sample will do. I like to check things like ‘yum check-update’, ‘yum repolist all’ and ‘yum list-sec’ to see what content is available. Apart from Satellite tools, you would expect the commands to return similar output at the end of the upgrade.

Sync the repositories you need. At a minimum you will need Satellite 6.6 Tools (rhel-7-server-satellite-tools-6.6-rpms) but if you are running a Capsule you will also need some others:

rhel-7-server-ansible-2.8-rpm
rhel-7-server-satellite-capsule-6.6-rpms
rhel-7-server-satellite-maintenance-6-rpms
rhel-server-rhscl-7-rpms

Optional – perform some housekeeping on the Postgres and MongoDB servers. This can reduce the size and fragmentation of the databases and lead to better performance. For details, see the Post Upgrade section of this post. You may also decide to do this task after the upgrade.

Perform a backup! If your Satellite and Capsule are running as Virtual Machines, shut them down and take a snapshot of them.

During the upgrade

As I’m running the upgrade I like to keep a couple of terminal windows open on the Satellite/Capsule server. ‘watch df -h’ can be useful to keep an eye on filesystem space. If an upgrade fails it might roll back the transaction and clear the down space. Monitoring space as you upgrade allows you to keep an eye on things. ‘top’ is always useful to see what’s going on, as is a ‘tail -f’ of the upgrade log.

Update your checklist as you perform the tasks. If anything is unusual, copy and paste the errors into a file so you can review them later.

It can be handy to make a note of timings. If for some reason you need to roll back, make a correction and then restart the upgrade, you’ll have a better idea of how long it will take to reach the same point.

It may sound obvious, but remember you’ll be upgrading your Satellite server first and your Capsule servers afterwards.

Post upgrade

I usually like to reboot the server post-upgrade even after restarting the Satellite services. This gives some extra confidence that everything is working as expected.

Raise support cases if needed. Remember the ‘unusual events’ notes you made during the upgrade? It’s worth checking the Red Hat support site and Red Hat Bugzilla to see if there is some guidance but I always recommend raising a support case for each issue even if it’s been previously recorded. This helps the Red Hat team track and prioritize bugs and in turn helps your fellow Satellite administrators.

Update your Content Views. The guide recommends performing this task ahead of the upgrade, but you may also want to do this afterwards. You’ll want to update the Content Views to use rhel-{6,7}-server-satellite-tools-6.6-rpms rather than rhel-{6,7}-server-satellite-tools-6.5-rpms

If you are running your own Puppet environment you may have excluded a number of RPMs from the satellite tools repositories. Puppet 5 in satellite-tools includes the ‘puppet-agent’ package. If you still use your own older puppet instance (e.g. Puppet 3) you will need to add ‘puppet-agent’ to the exclude list otherwise things like ‘puppet’ and ‘facter’ will get upgraded and your older puppet clients will no longer work.

Perform a MongoDB clean. You might like to perform a ‘mongodb repair’ to reduce the size of your MongoDB database. See Increasing promote/publish speed in Satellite 6 for details. Previously, I was able to reduce MongoDB from around 26GB to 20GB.

If you had many old tasks in your database which you removed earlier and did not perform a vacuum of the Postgres database, you may like to do so. See Task Cleanup for Satellite 6.2, 6.3, 6.4 and later versions for details.

On the Satellite clients, update components from the Satellite Tools repositories. I tend to use Ansible for this.

Revisit those test machines you identified before you performed the upgrade. Are the outputs to ‘yum repolist all’, ‘yum check-update’ and ‘subscription-manager identity’ as expected? Is puppet running successfully?

Perform any additional post-upgrade steps as described in the Post Upgrade section of the Upgrading and Updating Red Hat Satellite guide. This might include upgrading Discovery, virt-who, removing old Satellite Tools and Capsule repositories and updating templates.

If your Satellite server was initially installed at 6.0 or 6.1 and has been upgraded over time, check to see whether Mirror On Sync is set to true or false for the Red Hat repositories. The following article has some additional background on this setting for Red Hat repos: Searching Red Hat repositories with the Mirror on Sync setting set to No (For some background on this setting in the context of third party repositories, check https://www.unixsysadmin.com/mirror-on-sync/)

If you haven’t already done so, upgrade your MongoDB storage engine to Wired Tiger. You can do this in a separate maintenance window if you like. One big advantage is the storage space which you’ll reclaim – in my instance this dropped from 28GB to 6GB. The upgrade took around an hour. For full details, see How to migrate mongoDB storage engine from MMAPv1 to WiredTiger in Satellite 6.5 or later? Important tip: make sure the Satellite server is running when you run the installer. Initially, I did not do this and the upgrade went horribly wrong! (Hey, that’s why we take backups!)

Here is what NOT do – you don’t need to stop services, Satellite needs MongoDB to be running!

# DO NOT RUN THE STOP COMMAND!
[root@capsule ]# satellite-maintain service stop
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services:
Stopping the following service(s):

rh-mongodb34-mongod, qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, goferd, httpd, puppetserver, foreman-proxy
- All services stopped                                                [OK]
--------------------------------------------------------------------------------

[root@capsule ]# satellite-installer --upgrade-mongo-storage-engine
Starting disk space check for upgrade
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

qdrouterd, qpidd, squid, pulp_celerybeat, pulp_resource_manager, pulp_streamer, pulp_workers, smart_proxy_dynflow_core, goferd, httpd
| All services stopped                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service stop --exclude "rh-mongodb34-mongod","postgresql","tomcat","dynflowd","foreman-proxy","puppetserver" finished successfully!
2020-03-06T05:56:34.113+0000	Failed: error connecting to db server: no reachable servers
mongodump --host localhost --out /var/tmp/mongodb_engine_upgrade failed! Check the output for error!
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod
- All services stopped                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service stop --only rh-mongodb34-mongod finished successfully!
rm -rf /var/lib/mongodb/* finished successfully!
sed -i.bak -e 's/mmapv1/wiredTiger/g' /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
mv /etc/opt/rh/rh-mongodb34/mongod.conf.bak /var/tmp/mongodb_engine_upgrade finished successfully!
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod
/ All services started                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!
2020-03-06T05:56:50.428+0000	the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2020-03-06T05:56:50.429+0000	Failed: mongorestore target '/var/tmp/mongodb_engine_upgrade/pulp_database' invalid: stat /var/tmp/mongodb_engine_upgrade/pulp_database: no such file or directory
mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error!
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod
| All services stopped                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service stop --only rh-mongodb34-mongod finished successfully!
mv -f /var/tmp/mongodb_engine_upgrade/mongod.conf.bak /etc/opt/rh/rh-mongodb34/mongod.conf finished successfully!
rm -rf /var/lib/mongodb/* finished successfully!
2020-03-06T05:57:01.030+0000	Failed: error connecting to db server: no reachable servers
mongorestore --host localhost --db=pulp_database --drop --dir=/var/tmp/mongodb_engine_upgrade/pulp_database failed! Check the output for error!
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):

rh-mongodb34-mongod
| All services started                                                [OK]
--------------------------------------------------------------------------------

foreman-maintain service start --only rh-mongodb34-mongod finished successfully!

Finally, if you are running Satellite on a Virtual Machine and performed a snapshot of the Satellite server before you began the upgrade, don’t forget to remove it once you are confident you won’t be needing it.

RHEL 8 Changes

There are also a couple of things to be aware of as you manage RHEL 8 clients from Satellite.

The name of the Satellite tools repositories has a naming format change compared to earlier releases, so you may need to update any custom scripts in case you used regular expressions such as “rhel-?-server-satellite-tools-6-?-rpms”:

RHEL Version	Satellite Tools Repository
6	`rhel-6-server-satellite-tools-6.6-rpms`
7	`rhel-7-server-satellite-tools-6.6-rpms`
8	`satellite-tools-6.6-for-rhel-8-x86_64-rpms`

When registering servers with Red Hat Satellite (or directly with Red Hat) using Subscription Manager, you may have been using a syntax similar to the following:
subscription-manager register --release=${rhelver}Server --org=CompanyName --activationkey=${initialactivationkey}
RHEL 8 no is no longer shipped as Server and Workstation, so the release string to use with RHEL 8 is simply 8. In the example above, here is what we would use:
subscription-manager register --release=${rhelver} --org=CompanyName --activationkey=${initialactivationkey}
Note that if you have accidentally registered a RHEL 8 instance with the Red Hat portal or Satellite, you can change the release setting as follows:
subscription-manager release --set=8

katello-package-upload has been removed in RHEL 8. See What happen to Katello-Package-Upload utility for RHEL 8 client connected to Red Hat Satellite Server? for further details.

Known Issues

Bug 1784293 – package gofer-2.12.5-5.el8sat.noarch requires /usr/bin/python2 When installing Satellite 6.6 tools on RHEL 8, the gofer package has a dependency on Python 2, whereas the Satellite 6.5 version of the equivalent package would happily use the more modern Python 3. This is reported fixed in Satellite 6.7

Bug 1537555 – Cannot remove GPGkey from client repo This isn’t specific to Satellite 6.6, but if you happen to create a custom repository with a custom GPG key and later decide to remove that key, a client will still attempt to use it. This is reported fixed in Satellite 6.7

Repository sync in Red Hat Satellite 6 failing with error Yum Metadata: unsupported operand type(s) for -=: ‘Retry’ and ‘int’ During sync you may get this message. If so, follow the instructions in the linked document.

Be aware that when you run a command such as foreman-maintain packages install foo or indeed decide you need to update a package from a non-Satellite repository, the installer may well kick off satellite-installer with the --upgrade option, and that in turn may lead to downtime of your Satellite services. See How to install or update packages in Red Hat Satellite 6.6 and Bugilla 1765519 – “foreman-maintain packages install” redundantly upgrades Satellite.

Enjoy the new features

The reporting engine is a great new feature that allows Satellite Admins to generate custom reports they can share with their managers on the health of their Red Hat environment. For further details see the Satellite 6 Reporting post.

Make use of the ‘foreman-maintain’ command to manage your Satellite and Capsule servers, since katello-service will now redirect to this.

[root@capsule ~]# katello-service --help
Redirecting to 'foreman-maintain service'
Usage:
    foreman-maintain service [OPTIONS] SUBCOMMAND [ARG] ...

Parameters:
    SUBCOMMAND                    subcommand
    [ARG] ...                     subcommand arguments

Subcommands:
    start                         Start applicable services
    stop                          Stop applicable services
    restart                       Restart applicable services
    status                        Get statuses of applicable services
    list                          List applicable services
    enable                        Enable applicable services
    disable                       Disable applicable services

Options:
    -h, --help                    print help

Of course, there are many other new features such as enhanced Container Management features to look at!

I hope these tips are helpful. Feel free to add your comments below and/or take a look the related comments on Reddit: Tips for a Successful Red Hat Satellite Upgrade. Good luck with your upgrades!

unixsysadmin.com

Systems Administration in the Enterprise