Last week Red Hat gave Satellite administrators a Valentine’s Day treat by releasing Satellite 6.4.2 on 14 February 2019. The official blog post for the release can be found here. Time to upgrade?

I recently upgraded a server from Satellite 6.3.5 to 6.4.2 and thought it was worth sharing some tips in case others find this useful. Note, if you are already running Satellite 6.4 and want to upgrade to Satellite 6.5, see my new post Tips for a Successful Red Hat Satellite 6.5 Upgrade.

What’s new

One reason for upgrading is to keep your Satellite up to date, fully supported and able to take advantage of the newest features. The Red Hat Satellite Product Life Cycle shows that Satellite 6.3 currently has maintenance support until May 2019 so upgrading before then isn’t a bad idea.

What’s new in Red Hat Satellite 6.4 highlights some of the new features in Satellite 6.4:

What’s new in Red Hat Satellite 6.4

Usability—updated UX; vertical navigation; notification drawer; repositories, subscription, and auditing pages; and auto-publish of composite content views.
Configuration management—Red Hat Satellite & Red Hat Ansible Tower integration, Puppet 5 requirement, and GIT pull request templates.
Supportability—provisioning to AWS GovCloud, load-balanced capsules, database offload, Docker private repo, and preservation of custom configs.
Performance/stability—Red Hat Enterprise Linux® performance co-pilot integration, rebase of MongoDB, tuning for postgres, and other performance and stability fixes.

For a complete list of components that make up Red Hat Satellite, the Satellite 6 Component Versions shows us that Satellite 6.3.5 to 6.4.2 upgrades a large number of components, including Pulp and MongoDB.

Whilst looking at the above list, it’s worth mentioning that MongoDB will be removed in the future – Red Hat Satellite to standardize on PostgreSQL backend. For now though, MongoDB it is!

Planning

Take time to read the documentation. There are 3 useful resources as a starting point:

• Products & Services > Product Documentation > Red Hat Satellite > 6.4 > Upgrading and Updating Red Hat Satellite
• Red Hat Satellite Upgrade Helper
• Common issues with Satellite 6 upgrade or installation

Check the storage requirements before you begin as the requirements for some filesystems have changed since Satellite 6.3. This applies to both the Satellite server and any Capsule servers – don’t forget to allocate space for /var/spool/squid on the Capsule!

Check the amount of CPU and memory you have allocated to your Satellite and Capsule servers. Over time the minimum requirements have increased so it’s worth reviewing.

Satellite Version	Install Guide	CPU Requirement(*)	Memory(*)
6.0	Satellite 6.0	4 CPU	12 GB
6.1	Satellite 6.1	4 CPU	16 GB
6.2	Satellite 6.2	4 CPU	16 GB
6.3	Satellite 6.3	4 CPU	20 GB
6.4	Satellite 6.4	4 CPU	20 GB

(*) Minimum values.

If you have a Red Hat Support contract, take the time to raise a proactive support case. If your upgrade fails it’s useful to have support on hand so they can quickly review what’s happened. Sometimes a second pair of eyes looking at an error message can be really useful. For more details, see How to open a proactive case for patching of upgrading production Red hat Satellite and Capsule.

Choose an appropriate time to perform the upgrade. The Satellite and Capsule services will be stopped during the upgrade so you want to avoid times when others may be using the server (e.g. patching their hosts, provisioning new servers). Another useful tip, make sure you’ve set aside dedicated time in your own calendar for the upgrade so you can focus on it. Trying to the do the upgrade at the same time as answering support calls or doing other project work is asking for trouble!

Draw up a checklist of actions that you are going to perform as part of the upgrade and use it! If you get distracted or something needs verifying it’s easy to lose track of where you got to.

Be prepared to postpone your upgrade. There are many things outside your control. For example, the next Spectre or Meltdown vulnerabilities may be just around the corner, just at a time when you’ll need Satellite available to help you patch. An unrelated failure in your infrastructure may need your focus, or you may simply get sick. Having a working 6.3.5 Satellite is probably more beneficial than a partially upgraded 6.4.2 Satellite!

Pre-Upgrade Checks

If you haven’t done so yet, you’ll need to upgrade to Puppet 4 running on Satellite 6.3.5.
Products & Services > Product Documentation > Red Hat Satellite > 6.3 > Upgrading and Updating Red Hat Satellite > Chapter 3. Upgrading Puppet has full details of what’s involved. Depending on how you use Puppet this may be trivial or quite involved.

Satellite 6.3.5 is a prerequisite for the upgrade. If you upgraded to Satellite 6.3.5 some time ago (as early as October 2018 according to the Red Hat Satellite Release Dates) and not performed any updates since that time, you might find there are a large number of O/S patches to apply. If you can get a reboot maintenance window to bring your server fully up to date it’s a good idea. Should you need to rollback during the upgrade at any point, you’ll be able to restart without needing to re-apply all those O/S updates again.

Run the “upgrade check” option in foreman-maintain to check that your Satellite is in a good state to upgrade to Satellite 6.4. This can be run online without interrupting Satellite services.

yum update rubygem-foreman_maintain
foreman-maintain upgrade check --target-version 6.4

If you don’t perform housekeeping on Satellite to remove old tasks, the “foreman-maintain upgrade check” command can remove paused or stopped task(s) older than 30 days. This saves on database space.

The upgrade check may fail with the following error:

The following steps ended up in failing state:

  [disk-performance]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="disk-performance"

If that happens it’s likely to be due to Bug 1641784 – Disk performance task during foreman-maintain upgrade check fails with low disk speed when using the latest version of rubygem-foreman_maintain (0.2.11-1) The following Red Hat knowledgebase article also has details: Red Hat Satellite upgrade check from 6.3 to 6.4 fails at “disk-performance” Don’t panic! If that happens add the whitelist option to foreman-maintain. For example:

yum update rubygem-foreman_maintain
foreman-maintain upgrade check --target-version 6.4 --whitelist="disk-performance"

Have some test machines connected to your Satellite and Capsule before the upgrade which you can use to compare yum and puppet behaviour before and afterwards. Ideally, have these in different lifecycle environments. You don’t have to go crazy, just a small, representative sample will do. I like to check things like ‘yum check-update’, ‘yum repolist all’ and ‘yum list-sec’ to see what content is available. Apart from Satellite tools, you would expect the commands to return similar output at the end of the upgrade.

Sync the repositories you need. At a minimum you will need Satellite 6.4 Tools (rhel-7-server-satellite-tools-6.4-rpms) but if you are running a Capsule you will also need some others:

• rhel-7-server-ansible-2.6-rpm
• rhel-7-server-satellite-capsule-6.4-rpms
• rhel-7-server-satellite-maintenance-6-rpms
• rhel-server-rhscl-7-rpms

Optional – perform some housekeeping on the Postgres and MongoDB servers. This can reduce the size and fragmentation of the databases and lead to better performance. For details, see the Post Upgrade section of this post. You may also decide to do this task after the upgrade.

Perform a backup! If your Satellite and Capsule are running as Virtual Machines, shut them down and take a snapshot of them.

During the upgrade

As I’m running the upgrade I like to keep a couple of terminal windows open on the Satellite/Capsule server. ‘watch df -h’ can be useful to keep an eye on filesystem space. If an upgrade fails it might roll back the transaction and clear the down space. Monitoring space as you upgrade allows you to keep an eye on things. ‘top’ is always useful to see what’s going on, as is a ‘tail -f’ of the upgrade log.

Update your checklist as you perform the tasks. If anything is unusual, copy and paste the errors into a file so you can review them later.

It can be handy to make a note of timings. If for some reason you need to roll back, make a correction and then restart the upgrade, you’ll have a better idea of how long it will take to reach the same point.

It may sound obvious, but remember you’ll be upgrading your Satellite server first and your Capsule servers afterwards.

Post upgrade

I usually like to reboot the server post-upgrade even after restarting the Satellite services. This gives some extra confidence that everything is working as expected.

Raise support cases if needed. Remember the ‘unusual events’ notes you made during the upgrade? It’s worth checking the Red Hat support site and Red Hat Bugzilla to see if there is some guidance but I always recommend raising a support case for each issue even if it’s been previously recorded. This helps the Red Hat team track and prioritize bugs and in turn helps your fellow Satellite administrators.

Update your Content Views. The guide recommends performing this task ahead of the upgrade, but you may also want to do this afterwards. You’ll want to update the Content Views to use rhel-{6,7}-server-satellite-tools-6.4-rpms rather than rhel-{6,7}-server-satellite-tools-6.3-rpms

If you are running your own Puppet environment you may have excluded a number of RPMs from the satellite tools repositories. Puppet 5 in satellite-tools includes the ‘puppet-agent’ package. If you still use your own older puppet instance (e.g. Puppet 3) you will need to add ‘puppet-agent’ to the exclude list otherwise things like ‘puppet’ and ‘facter’ will get upgraded and your older puppet clients will no longer work.

Perform a MongoDB clean. You might like to perform a ‘mongodb repair’ to reduce the size of your MongoDB database. See Increasing promote/publish speed in Satellite 6 for details. In my instance I was able to reduce MongoDB from around 26GB to 20GB.

If you had many old tasks in your database which you removed earlier and did not perform a vacuum of the Postgres database, you may like to do so. See Task Cleanup for Satellite 6.2, 6.3, 6.4 and later versions for details.

On the Satellite clients, update components from the Satellite Tools repositories. I tend to use Ansible for this.

Revisit those test machines you identified before you performed the upgrade. Are the outputs to ‘yum repolist all’, ‘yum check-update’ and ‘subscription-manager identity’ as expected? Is puppet running successfully?

Perform any additional post-upgrade steps as described in the Post Upgrade section of the Upgrading and Updating Red Hat Satellite guide. This might include upgrading Discovery, virt-who, removing old Satellite Tools and Capsule repositories and updating templates

If your Satellite server was initially installed at 6.0 or 6.1 and has been upgraded over time, check to see whether Mirror On Sync is set to true or false for the Red Hat repositories. The following article has some additional background on this setting for Red Hat repos: Searching Red Hat repositories with the Mirror on Sync setting set to No (For some background on this setting in the context of third party repositories, check https://www.unixsysadmin.com/mirror-on-sync/)

Finally, if you are running Satellite on a Virtual Machine and performed a snapshot of the Satellite server before you began the upgrade, don’t forget to remove it once you are confident you won’t be needing it.

Known Issues

In case it’s of use, here are a couple of Bugzilla records to be aware of:

Bugzilla 1647202 – Upgrading from Satellite 6.3 to Satellite 6.4 the gofer & katello-agent packages get installed in the satellite 6.4 server Most likely you will see errors in /var/log/messages confirming that goferd is running and cannot connect to ampqs://subscription.rhsm.redhat.com:5467

Bugzilla 1625649 – Yum plugins are loaded multiple times after updating the host to the latest katello-agent packages

Enjoy the new features

Say goodbye to the old top level menu structure.

And say hello to the new sidebar!

A number of the menus allow now have an “Export” button. This exports the data in CSV format. For example the screenshot below is from the content host report. This makes it easier to generate reports you can share with management.

Make use of the ‘foreman-maintain’ command to manage your Satellite and Capsule servers, since katello-service will now redirect to this.

[root@capsule ~]# katello-service --help
Redirecting to 'foreman-maintain service'
Usage:
    foreman-maintain service [OPTIONS] SUBCOMMAND [ARG] ...

Parameters:
    SUBCOMMAND                    subcommand
    [ARG] ...                     subcommand arguments

Subcommands:
    start                         Start applicable services
    stop                          Stop applicable services
    restart                       Restart applicable services
    status                        Get statuses of applicable services
    list                          List applicable services
    enable                        Enable applicable services
    disable                       Disable applicable services

Options:
    -h, --help                    print help

Of course, there are many other new features such as Ansible Roles to look at!

I hope these tips are helpful. Feel free to add your comments below and/or take a look the related comments on Reddit: Tips for a Successful Red Hat Satellite Upgrade. Good luck with your upgrades!