Update: May 2020. If you’re already on Satellite 6.5 and looking to upgrade to Satellite 6.6, see Tips for a Successful Red Hat Satellite 6.6 Upgrade.
Red Hat Satellite 6.5 was released just after Red Hat Summit on 14 May 2019. The official blog post for the release can be found here. Time to upgrade?
I recently upgraded a server from Satellite 6.4.2 to 6.5 and thought it was worth sharing some tips in case others find this useful. If you are currently on Satellite 6.3 you might want to take a look at my earlier Tips for a Successful Red Hat Satellite 6.4 Upgrade post.
If you are already on Satellite 6.5 and are looking to upgrade to Satellite 6.6, check out the Tips for a Successful Red Hat Satellite 6.6 Upgrade post.
What’s new
One reason for upgrading is to keep your Satellite up to date, fully supported and able to take advantage of the newest features. Another big plus is the ability to provision and support RHEL 8 servers. Satellite itself cannot yet run on RHEL 8 (that capability will likely come in a future Satellite release) but with many enterprises looking to deploy RHEL 8 it makes sense to upgrade when you can. The Red Hat Satellite Product Life Cycle shows that Satellite 6.4 currently has maintenance support until Fall 2019 so upgrading before then isn’t a bad idea.
The What’s new in Red Hat Satellite 6.5 webinar (available on demand) is worth a look if you want to get a feel for some of the new features, otherwise here are some highlights from the Red Hat Satellite 6.5 is now available post:
Red Hat Satellite 6.5 is now available
Red Hat Enterprise Linux 8 Provisioning & Patching
FIPS Support on RHEL 7
OpenSCAP Visibility
Satellite Admin Role – allows a user to manage Satellite infrastructure and create new organizations, but not manage the hosts themselves
Export Content Views
Container Admin – enhancements to managing container images from Satellite
For a complete list of components that make up Red Hat Satellite, the Satellite 6 Component Versions shows us that Satellite 6.4 to 6.5 upgrades many of components including Pulp and MongoDB. Note though that these are smaller updates, rather than any major version bumps. This should reduce the risk of encountering upgrade bugs.
Whilst looking at the above list, it’s worth mentioning that MongoDB will be removed in the future – Red Hat Satellite to standardize on PostgreSQL backend. For now though, MongoDB it is!
Planning
Take time to read the documentation. There are 4 useful resources as a starting point:
- Products & Services > Product Documentation > Red Hat Satellite > 6.5 > Upgrading and Updating Red Hat Satellite
- Red Hat Satellite Upgrade Helper
- Common issues with Satellite 6 upgrade or installation
- Red Hat Satellite 6.5 known issues
Check the storage requirements before you begin as the requirements for some filesystems have may changed since your server was built. This applies to both the Satellite server and any Capsule servers – don’t forget to allocate space for /var/spool/squid on the Capsule!
Check the amount of CPU and memory you have allocated to your Satellite and Capsule servers. Over time the minimum requirements have increased so it’s worth reviewing.
| Satellite Version | Install Guide | CPU Requirement(*) | Memory(*) |
| 6.0 | Satellite 6.0 | 4 CPU | 12 GB |
| 6.1 | Satellite 6.1 | 4 CPU | 16 GB |
| 6.2 | Satellite 6.2 | 4 CPU | 16 GB |
| 6.3 | Satellite 6.3 | 4 CPU | 20 GB |
| 6.4 | Satellite 6.4 | 4 CPU | 20 GB |
| 6.5 | Satellite 6.5 | 4 CPU | 20 GB |
(*) Minimum values. There is also a note on Increasing promote/publish speed in Satellite 6 which details memory considerations depending on the size of your MongoDB database.
If you have a Red Hat Support contract, take the time to raise a proactive support case. If your upgrade fails it’s useful to have support on hand so they can quickly review what’s happened. Sometimes a second pair of eyes looking at an error message can be really useful. For more details, see How to open a proactive case for patching of upgrading production Red hat Satellite and Capsule.
Choose an appropriate time to perform the upgrade. The Satellite and Capsule services will be stopped during the upgrade so you want to avoid times when others may be using the server (e.g. patching their hosts, provisioning new servers). Another useful tip, make sure you’ve set aside dedicated time in your own calendar for the upgrade so you can focus on it. Trying to the do the upgrade at the same time as answering support calls or doing other project work is asking for trouble!
Draw up a checklist of actions that you are going to perform as part of the upgrade and use it! If you get distracted or something needs verifying it’s easy to lose track of where you got to.
Be prepared to postpone your upgrade. There are many things outside your control. For example, the next Spectre or Meltdown vulnerabilities may be just around the corner, just at a time when you’ll need Satellite available to help you patch. An unrelated failure in your infrastructure may need your focus, or you may simply get sick. Having a working 6.4 Satellite is probably more beneficial than a partially upgraded 6.5 Satellite!
Pre-Upgrade Checks
Satellite 6.4 is a prerequisite for the upgrade. Note from the Red Hat Satellite Ask Me Anything Q&A from April 2019 that “Any .Z version of 6.4 can upgrade to 6.5, there’s no requirement to go to 6.4.3.” If you upgraded to Satellite 6.4 some time ago (as early as October 2018 according to the Red Hat Satellite Release Dates) and not performed any updates since that time, you might find there are a large number of O/S patches to apply. If you can get a reboot maintenance window to bring your server fully up to date it’s a good idea. Should you need to rollback during the upgrade at any point, you’ll be able to restart without needing to re-apply all those O/S updates again.
Run the “upgrade check” option in foreman-maintain to check that your Satellite is in a good state to upgrade to Satellite 6.5. This can be run online without interrupting Satellite services.
yum update rubygem-foreman_maintain
foreman-maintain upgrade check --target-version 6.5If you don’t perform housekeeping on Satellite to remove old tasks, the “foreman-maintain upgrade check” command can remove paused or stopped task(s) older than 30 days. This saves on database space.
The upgrade check may show the following message:
Check for recommended disk speed of pulp, mongodb, pgsql dir.:
| Finished
Disk speed : 33 MB/sec [WARNING]
Slow disk detected /var/lib/pulp mounted on /dev/mapper/vg_dhcp201-lv_root.
Actual disk speed: 33 MB/sec
Expected disk speed: 60 MB/sec.
WARNING: Low disk speed might have a negative impact on the system.
See https://access.redhat.com/solutions/3397771 before proceedingIf that happens it’s likely to be due to Bug 1641784 – Disk performance task during foreman-maintain upgrade check fails with low disk speed when using the latest version of rubygem-foreman_maintain (0.2.11-1) The following Red Hat knowledgebase article also has details: Red Hat Satellite upgrade check from 6.3 to 6.4 fails at “disk-performance” Don’t panic! If that happens add the whitelist option to foreman-maintain. For example:
yum update rubygem-foreman_maintain
foreman-maintain upgrade check --target-version 6.5 --whitelist="disk-performance"You may also see the following:Check for roles that have filters with multiple resources attached
[FAIL] There are user roles with inconsistent filters
I had the foreman-maintain script automatically fix the errors, but it wasn’t immediately obvious how my server had gotten into this state. There is a related Bugzilla for this – Bugzilla 1703041[RFE] verbose output for failing upgrade checks – which suggests enhancing the diagnostic message – what roles have this error and what is the script doing to correct this?
Have some test machines connected to your Satellite and Capsule before the upgrade which you can use to compare yum and puppet behaviour before and afterwards. Ideally, have these in different lifecycle environments. You don’t have to go crazy, just a small, representative sample will do. I like to check things like ‘yum check-update’, ‘yum repolist all’ and ‘yum list-sec’ to see what content is available. Apart from Satellite tools, you would expect the commands to return similar output at the end of the upgrade.
Sync the repositories you need. At a minimum you will need Satellite 6.5 Tools (rhel-7-server-satellite-tools-6.5-rpms) but if you are running a Capsule you will also need some others:
- rhel-7-server-ansible-2.6-rpm
- rhel-7-server-satellite-capsule-6.5-rpms
- rhel-7-server-satellite-maintenance-6-rpms
- rhel-server-rhscl-7-rpms
Optional – perform some housekeeping on the Postgres and MongoDB servers. This can reduce the size and fragmentation of the databases and lead to better performance. For details, see the Post Upgrade section of this post. You may also decide to do this task after the upgrade.
Perform a backup! If your Satellite and Capsule are running as Virtual Machines, shut them down and take a snapshot of them.
During the upgrade
As I’m running the upgrade I like to keep a couple of terminal windows open on the Satellite/Capsule server. ‘watch df -h’ can be useful to keep an eye on filesystem space. If an upgrade fails it might roll back the transaction and clear the down space. Monitoring space as you upgrade allows you to keep an eye on things. ‘top’ is always useful to see what’s going on, as is a ‘tail -f’ of the upgrade log.
Update your checklist as you perform the tasks. If anything is unusual, copy and paste the errors into a file so you can review them later.
It can be handy to make a note of timings. If for some reason you need to roll back, make a correction and then restart the upgrade, you’ll have a better idea of how long it will take to reach the same point.
It may sound obvious, but remember you’ll be upgrading your Satellite server first and your Capsule servers afterwards.
Post upgrade
I usually like to reboot the server post-upgrade even after restarting the Satellite services. This gives some extra confidence that everything is working as expected.
Raise support cases if needed. Remember the ‘unusual events’ notes you made during the upgrade? It’s worth checking the Red Hat support site and Red Hat Bugzilla to see if there is some guidance but I always recommend raising a support case for each issue even if it’s been previously recorded. This helps the Red Hat team track and prioritize bugs and in turn helps your fellow Satellite administrators.
Update your Content Views. The guide recommends performing this task ahead of the upgrade, but you may also want to do this afterwards. You’ll want to update the Content Views to use rhel-{6,7}-server-satellite-tools-6.5-rpms rather than rhel-{6,7}-server-satellite-tools-6.4-rpms
If you are running your own Puppet environment you may have excluded a number of RPMs from the satellite tools repositories. Puppet 5 in satellite-tools includes the ‘puppet-agent’ package. If you still use your own older puppet instance (e.g. Puppet 3) you will need to add ‘puppet-agent’ to the exclude list otherwise things like ‘puppet’ and ‘facter’ will get upgraded and your older puppet clients will no longer work.
Perform a MongoDB clean. You might like to perform a ‘mongodb repair’ to reduce the size of your MongoDB database. See Increasing promote/publish speed in Satellite 6 for details. In my instance I was able to reduce MongoDB from around 26GB to 20GB.
If you had many old tasks in your database which you removed earlier and did not perform a vacuum of the Postgres database, you may like to do so. See Task Cleanup for Satellite 6.2, 6.3, 6.4 and later versions for details.
On the Satellite clients, update components from the Satellite Tools repositories. I tend to use Ansible for this.
Revisit those test machines you identified before you performed the upgrade. Are the outputs to ‘yum repolist all’, ‘yum check-update’ and ‘subscription-manager identity’ as expected? Is puppet running successfully?
Perform any additional post-upgrade steps as described in the Post Upgrade section of the Upgrading and Updating Red Hat Satellite guide. This might include upgrading Discovery, virt-who, removing old Satellite Tools and Capsule repositories and updating templates.
If your Satellite server was initially installed at 6.0 or 6.1 and has been upgraded over time, check to see whether Mirror On Sync is set to true or false for the Red Hat repositories. The following article has some additional background on this setting for Red Hat repos: Searching Red Hat repositories with the Mirror on Sync setting set to No (For some background on this setting in the context of third party repositories, check https://www.unixsysadmin.com/mirror-on-sync/)
Finally, if you are running Satellite on a Virtual Machine and performed a snapshot of the Satellite server before you began the upgrade, don’t forget to remove it once you are confident you won’t be needing it.
RHEL 8 Changes
There are also a couple of things to be aware of as you manage RHEL 8 clients from Satellite.
The name of the Satellite tools repositories has a naming format change compared to earlier releases, so you may need to update any custom scripts in case you used regular expressions such as “rhel-?-server-satellite-tools-6-?-rpms”:
| RHEL Version | Satellite Tools Repository |
| 6 | rhel-6-server-satellite-tools-6.5-rpms |
| 7 | rhel-7-server-satellite-tools-6.5-rpms |
| 8 | satellite-tools-6.5-for-rhel-8-x86_64-rpms |
When registering servers with Red Hat Satellite (or directly with Red Hat) using Subscription Manager, you may have been using a syntax similar to the following:subscription-manager register --release=${rhelver}Server --org=CompanyName --activationkey=${initialactivationkey}
RHEL 8 no is no longer shipped as Server and Workstation, so the release string to use with RHEL 8 is simply 8. In the example above, here is what we would use:subscription-manager register --release=${rhelver} --org=CompanyName --activationkey=${initialactivationkey}
Note that if you have accidentally registered a RHEL 8 instance with the Red Hat portal or Satellite, you can change the release setting as follows:subscription-manager release --set=8
katello-package-upload has been removed in RHEL 8. See What happen to Katello-Package-Upload utility for RHEL 8 client connected to Red Hat Satellite Server? for further details.
Known Issues
In case it’s of use, here is one Bugzilla record to be aware of:
Bugzilla 1703018 Registries: “undefined method `auto_complete_search_registries_path'”. Going to Containers -> Registries in WebUI causes an error.
For other hints, it’s worth taking a look at Red Hat Satellite 6.5 known issues.
Enjoy the new features
The reporting engine is a great new feature that allows Satellite Admins to generate custom reports they can share with their managers on the health of their Red Hat environment. For further details see the Satellite 6 Reporting post.
Make use of the ‘foreman-maintain’ command to manage your Satellite and Capsule servers, since katello-service will now redirect to this.
[root@capsule ~]# katello-service --help
Redirecting to 'foreman-maintain service'
Usage:
foreman-maintain service [OPTIONS] SUBCOMMAND [ARG] ...
Parameters:
SUBCOMMAND subcommand
[ARG] ... subcommand arguments
Subcommands:
start Start applicable services
stop Stop applicable services
restart Restart applicable services
status Get statuses of applicable services
list List applicable services
enable Enable applicable services
disable Disable applicable services
Options:
-h, --help print helpOf course, there are many other new features such as enhanced Container Management features to look at!
I hope these tips are helpful. Feel free to add your comments below and/or take a look the related comments on Reddit: Tips for a Successful Red Hat Satellite Upgrade. Good luck with your upgrades!
Mentions