Ansible Resources And Best Practices with AWX and Tower

I am currently doing a lot of work with Ansible, AWX, Ansible Tower and Red Hat Satellite to automate and manage a RHEL 6, RHEL 7 and RHEL 8 environment. During that time I bookmarked a number of useful resources and have shared them on this page along with my notes.

How to use existing Vault files in Ansible?

Link: How to use existing Vault files in Ansible Tower There are a number of different places where you can store secret data – in AWX/Tower itself, within an Ansible Vault or in an external source. Storing credentials within AWX/Tower is a good idea if you intend to run all jobs through AWX/Tower. If you want the flexibility to run jobs on both the CLI and AWX/Tower then using Ansible Vault is a good idea. However, as per the link you will likely want to store the Vault in the playbook area rather than the inventory area.

It is not recommended to keep the vault files with the inventory – as this would mean it decrypts the file every time the inventory sync runs. The way I’ve solved this now is by using “vars_files” in the playbook. It looks like this:

  # Secrets
  vars_files:
    - '../../secrets/{{ tower_env }}/vault.yml'

In Tower, I pass in the tower_env variable e.g. “dev” or “qa”, which then decrypts the corresponding vault file when a playbook runs – rather then when syncing inventories

Secret Management with Ansible

Link: Secret Management with Ansible Encrypting your secret data with Ansible Vault is sensible – it means you can control who has access to data which you will only need at runtime (API keys, Database credentials, etc). However, you’ll need to credentials to access the vault itself. This article reviews the advantages of using git-crypt to protect those Vault credentials.

Add ServiceNow Inventory to AWX/Tower

Link: Add Custom Dynamic Inventory in Ansible Tower (ServiceNow) There are a couple of very useful topics within this article. The first shows how to create a custom credential type within AWX/Tower. Dynamic Inventories will likely use scripts which in turn call APIs. These scripts require credentials and parameters of varying types – keys, username/password combinations, environments (Production, Stage, Test, etc) and so on. A custom credential allows you to define the variables that your script will need. The variables will then be set as environment variables at runtime. The article then goes on to detail how you are able to import your ServiceNow hosts into AWX/Ansible Tower using the ansible-sn-inventory script.

Private Ansible Galaxy

Link: Private Ansible Galaxy? As you use Ansible more and more, the number of playbooks and roles that need to be managed grows. Ansible-galaxy is a very nice way of making sure that all your custom roles are of a similar standard (You’ll have to enforce a naming convention yourself – see AWX / Tower Naming Conventions and Best Practice). To initialise a repository:

ansible-galaxy init test-role --offline

Create your role and publish it to git. For example, you have a role that sets up /etc/motd, another that configures /etc/issue. You would then create a ‘master’ role which references the other roles using the following syntax in requirements.yml:

- src: git+ssh://git@<company_git_server>/<rolerepo>.git
  scm: git

Define Variables when conditions are met

Link: Ansible: Conditionally define variables in vars file if a certain condition is met

The example listed suggests you can use the following to set variables according to the role/function of a server:

- include_vars: test_environment_vars.yml
  when: global_platform == "test"

- include_vars: staging_environment_vars.yml
  when: global_platform == "staging"

- include_vars: prod_environment_vars.yml
  when: 
    - global_platform != "test" 
    - global_platform != "staging"

You can use this approach for maintaining other variables. For example, you have some sysctl parameters that should be applied for different versions of database software, such as Oracle.

Pass Extra Variables to playbooks

There will be some tasks you will want to be run by support staff in AWX or Ansible Tower. For example, setting the root password on all servers. AWX gives you a controlled, auditable, repeatable way of achieving this. Other tasks will be run by experienced sysadmins that want to use the command line. In an ideal world, you want to create code that can run on the CLI and Ansible Tower/AWX.

Link: Ansible – Pass extra variables to Playbook. You can choose to store your variable in AWX/Tower but need to simulate them on the command line. The following code can help achieve this:

ansible-playbook extra_var_json.yml -e '{"my_string": "JSON with spaces", "my_float": 1.618, "my_list": ["HGTTG", 42]}'

View only Ansible Failures

Link: Only see failures. When first starting out, you will want to ensure that your Ansible control host, Tower or AWX instance can manage all your servers. If the majority of servers are in a good state, and you only want to report on failures, use the ANSIBLE_DISPLAY_OK_HOSTS and ANSIBLE_DISPLAY_SKIPPED_HOSTS environment variables or display_ok_hosts and display_skipped_hosts in your ansible ini file. Here’s a useful example on the command line:

ANSIBLE_DISPLAY_OK_HOSTS=false ANSIBLE_DISPLAY_SKIPPED_HOSTS=false ansible all -m ping

There is also the –one-line (-o) flag. This makes it easier to use shell tools such as grep, sed and awk to manipulate the results:

ansible --one-line ..... | grep -v SUCCESS

Running Ansible against a host that is not in the inventory

Link: Ansible Tower Question. This is quite common in my environment, especially when provisioning new servers that don’t (yet) appear in our external inventory sources such as Red Hat Satellite, Service Now, etc and/or we don’t want to call the external inventory source because of the amount of time it takes to run. A nice solution is to create or update an inventory when a job is launched. For example:

- name: create inventory, add hosts to it, then launch my playbook against those hosts
  hosts: localhost
  vars:
    your_host: mywebhost1.example.com
    app_name: Example
  tasks:
  - name: Create the inventory
    uri:
      url: https://mytowerhost/api/v2/inventories/
      method: POST
      body_format: json
      headers:
        Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXX
      body:
        name: "{{ app_name }}"
        organization: 1
      return_content: yes
      status_code: [201]
    ignore_errors: true ##(incase inventory already exists
    register: tower_inv

   - name: Add hosts
     uri:
       url: https://mytowerhost/api/v2/inventories/{{ tower_inv.json.id }}/hosts/
       method: POST
       body_format: json
       headers:
         Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXX
       body:
         name: "{{ your_host }}"
         variables:
           ansible_host: "{{ your_host }}"
           ansible_user: yourRootUser
           ansible_pass: yourRootPass ##etc, etc..
       status_code: [201]

    - name: Call your playbook with the hosts/inventory you just created
      uri:              ##replace 20 with id of the the job template you want
        url: https://mytowerhost/api/v2/job_templates/20/launch 
        method: POST
        body_format: json
        headers:
          Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXX
        body:
          inventory: "{{ tower_inv.json.id }}"

Using pipefail with shell module in Ansible

Link: Using pipefail with shell module in Ansible. One of the great things about Ansible is that it’s easy to get started. One of the first things users will do is run ‘ad-hoc’ commands on hosts, and pretty quickly they will try to pipe one command into another. Use the ‘set -o pipefail’ option in your shell commands to ensure that failures are correctly reported:

ansible myhost -i hostfile -m shell -a 'set -o pipefail && /path/to/nonexistant/script | wc -l'

The return code from this host will now return FAILED.

Combine Two Lists

Link: How to combine two lists. This will be useful if you have two sets of variables which you need to ‘merge’ together. For example, you have parameters that go to all servers and a number of bespoke ones that need to be combined with them. Example, common sysctl parameters and then some custom ones. The ‘union’ function can be helpful here:

all_settings="{{ foo|map(attribute='settings')|union(bar|map(attribute='settings')) }}"

Ansible Lint / Yamllint

Link: Linting your Ansible Playbooks and make a Continuous Integration (CI) solution. ‘ansible-lint‘ and ‘yamllint‘ are useful tools for making sure that your yaml code and playbooks conform to best practice and do what you expect them to do. For example, in a complex YAML file, have you accidentally defined the same key-value pair twice? Which one will take precedence? These tools can help deal with this. One solution is to run these tools prior to them being uploaded into source code repositories, the other is to run tests when commits are made.

Satellite 6 / Callback Integration

Link: Use Satellite 6 as an inventory source.
Link: Connecting Satellite 6 and Ansible Tower.

Linking your AWX/Tower server to Red Hat Satellite 6 is a great way to manage your environment. Callbacks are a nice way to running a Playbook as part of the provisioning process. Even if your servers are not provisioned by Satellite, you can still call playbook from AWX/Tower at first boot via systemd unit file or custom script. (See the related in this post about adding a host to an inventory if it does not already exist). A callback script to setup a systemd unit could look as follows and would typically be included in the %post section of a kickstart file:

#!/bin/bash
cat > /etc/systemd/system/ansible-callback.service << EOF
[Unit]
Description=Provisioning callback to Ansible
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/curl -k -s --data "host_config_key=XYZ" https://awx.example.com/api/v1/job_templates/NN/callback/
ExecStartPost=/usr/bin/systemctl disable ansible-callback

[Install]
WantedBy=multi-user.target
EOF

# Runs during first boot, removes itself
/usr/bin/systemctl enable ansible-callback

Thanks to http://100things.wzzrd.com/2017/03/30/Connecting-Satellite-6-and-Ansible-Tower.html for this idea.

Async Actions for long running tasks

It may be the case that you have playbooks which take a long time to run. In my case, we run some scripts on our Satellite server to manipulate content views and synchronise Red Hat Satellite capsule servers. Due to bandwidth limitations the synchronisation can take many hours. The playbooks look like this:

 - name: Manipulate RHEL 6 Content View biz unit 1
   command: /path/to/script.sh rhel6-biz-unit-1
   register: output
   tags: rhel6

- debug:
    msg: "{{output.stdout_lines}}"
  tags: rhel6

 - name: Manipulate RHEL 7 Content View biz unit 1
   command: /path/to/script.sh rhel7-biz-unit-1
   register: output
   tags: rhel7

- debug:
    msg: "{{output.stdout_lines}}"
  tags: rhel7

When running under AWX/Tower, the script suddenly aborts exactly 1 hour into the job:

{
    "unreachable": true,
    "msg": "Failed to connect to the host via ssh: Shared connection to satellite.example.com closed.",
    "changed": false
}

Right at that time, Adam Miller posted the following:

I think one of the most-powerful least-utilized features of #Ansible is Async Actions and Polling. I need to write a blog that talks about common scenarios and patterns to utilize it along with performance comparison numbers. In the mean time, check it out https://t.co/vhm0aqxk3R
— Adam Miller (@TheMaxamillion) October 1, 2019

Sure enough, Async Actions were what I needed so I re-wrote the playbooks as follows:

 - name: Manipulate RHEL 6 Content View biz unit 1
   command: /path/to/script.sh rhel6-biz-unit-1
   register: output
   async: 14400
   tags: rhel6

- debug:
    msg: "{{output.stdout_lines}}"
  tags: rhel6

 - name: Manipulate RHEL 7 Content View biz unit 1
   command: /path/to/script.sh rhel7-biz-unit-1
   register: output
   async: 14400
   tags: rhel7

- debug:
    msg: "{{output.stdout_lines}}"
  tags: rhel7

AWX/Tower is now to process these long running jobs successfully. Take a look at the links in Adam’s post for further details on asynchronous actions.

Securing AWX

At the London Red Hat forum in 2019 someone asked what is the best practice for securing AWX/Ansible Tower itself? Given that AWX has the access rights to connect to most of your servers, keeping it secure becomes an important topic. Some suggestions:

Always pull code from your (secured) Git repository. Although it is possible to store code locally on an AWX server, if compromised by bad file permissions someone could manipulate a task to do something malicious. Don’t let that happen, set your projects to always pull from Git on launch.
Along with the point above, ensure your Git repository is secured – for example limit user access, use read-only deployment keys, enable two factor authentication, etc.
Use the option ‘delete on update’ to ensure the local repository in its entirety is removed prior to performing an update from your source code repository.
Authenticate users with an identity source such as LDAP. If a user leaves or changes role, a check can be made to make sure they are part of an LDAP group and prevent access if that requirement is not met. Local AWX users are a no-go.
Limit access on the server itself only to those that require it.
Use standard O/S hardening on the AWX/Tower server – configure a local firewall, enable SELinux, configure Intrusion Detection, enforce strong authentication, beware of access via SSH authorized keys.
Audit. Regularly review logs for both access and job completion to ensure they meet the expectations for your environment – for example a sudden submission of ‘ad-hoc’ commands might be the sign of an attacker probing your servers.

Visualise and make use of Ansible facts

visansible is a very handy tool if you want to visually map out your environment. To get started, make use of the setup module to query the facts from your environment.

# Display facts from all hosts and store them indexed by I(hostname) at C(/tmp/facts).
# ansible all -m setup --tree /tmp/facts

The output in /tmp/facts can be useful for all sorts of data processing and analysis. However, combine it with visansible and you can visualize the findings. Another useful tool for the facts is ansible-cmdb which can generate a dynamic HTML page with useful information you might get from a traditional Configuration Management Database (CMDB).

2 thoughts on “Ansible Resources And Best Practices with AWX and Tower”

Adam Miller says: @ twitter.com

October 2, 2019 at 6:41 pm

Adam Miller liked this Article on twitter.com.

Will D. says: @ twitter.com

October 9, 2019 at 5:17 pm

Will D. liked this Article on twitter.com.

unixsysadmin.com

Systems Administration in the Enterprise