virtualization – dankolb.net

Molecule for existing Ansible roles

I previously walked through Ansible role creation with Molecule, but it’s just as easy to add to existing roles. Creating a Molecule scenario to test an existing role allows for easy testing and modification of that role with all the benefits that Molecule provides.

Existing role

For another easy example, we’ll just use a simple role that installs a webserver. To prevent a complete copy/paste, this time we will be using Apache rather than Nginx. To show the existing role in its current state:

~/Projects/example_playbooks/apache_install$ tree
.
└── tasks
    └── main.yml

1 directory, 1 file
~/Projects/example_playbooks/apache_install$ cat tasks/main.yml 
---
# install and start apache    
- name: install apache
  yum:
    name: httpd
    state: present
  become: "yes"

- name: ensure apache running and enabled
  systemd:
    name: httpd
    state: started
    enabled: "yes"
  become: "yes"

The Molecule and Ansible version used for this example is:

~/Projects/example_playbooks/apache_install$ ansible --version && molecule --version
ansible 2.6.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/dan/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/dan/.local/lib/python2.7/site-packages/ansible
  executable location = /home/dan/.local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
molecule, version 2.17.0

Init scenario

Because the role already exists, we will only be creating scenario, rather than a whole new role. The init scenario parameters are almost exactly the same as init role, and result in the same directory structure as if we created the role with molecule.

Molecule init scenario usage information:

~/Projects/example_playbooks/apache_install$ molecule init scenario --help
Usage: molecule init scenario [OPTIONS]

  Initialize a new scenario for use with Molecule.

Options:
  --dependency-name [galaxy]      Name of dependency to initialize. (galaxy)
  -d, --driver-name [azure|delegated|docker|ec2|gce|lxc|lxd|openstack|vagrant]
                                  Name of driver to initialize. (docker)
  --lint-name [yamllint]          Name of lint to initialize. (ansible-lint)
  --provisioner-name [ansible]    Name of provisioner to initialize. (ansible)
  -r, --role-name TEXT            Name of the role to create.  [required]
  -s, --scenario-name TEXT        Name of the scenario to create. (default)
                                  [required]
  --verifier-name [goss|inspec|testinfra]
                                  Name of verifier to initialize. (testinfra)
  --help                          Show this message and exit.

We create the scenario using the existing role name and specifying using vagrant as the driver. Once initialized, the Molecule directory structure will be the same as if we created the role with Molecule, but without any role directories being created (such as handlers, meta, etc).

~/Projects/example_playbooks/apache_install$ molecule init scenario --role-name apache_install --driver-name vagrant
--> Initializing new scenario default...
Initialized scenario in /home/dan/Projects/example_playbooks/apache_install/molecule/default successfully.
~/Projects/example_playbooks/apache_install$ tree
.
├── molecule
│   └── default
│       ├── INSTALL.rst
│       ├── molecule.yml
│       ├── playbook.yml
│       ├── prepare.yml
│       └── tests
│           └── test_default.py
└── tasks
    └── main.yml

4 directories, 6 files

Configuration

The Molecule configuration will be the default provided by molecule. As done previously, I edit this to use CentOS 7, rather than the default Ubuntu 16.04. Additionally, I update the name of the VM to something different to distinguish the VM if needed.

In this example our tests are very similiar to my previous example. The primary (and possibly only) differences in our tests from the previous example is we’re testing for the httpd service rather than nginx.

~/Projects/example_playbooks/apache_install$ cat molecule/default/molecule.yml 
---
dependency:
  name: galaxy
driver:
  name: vagrant
  provider:
    name: virtualbox
lint:
  name: yamllint
platforms:
  - name: apache
    box: centos/7
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8
~/Projects/example_playbooks/apache_install$ cat molecule/default/tests/test_default.py
import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')


def test_apache_installed(host):
    apache = host.package("httpd")
    assert apache.is_installed


def test_apache_config(host):
    apache = host.file('/etc/httpd/conf/httpd.conf')
    assert apache.exists


def test_apache_running_and_enabled(host):
    apache = host.service("httpd")
    assert apache.is_running
    assert apache.is_enabled

Molecule test

Since we’ve updated our Molecule configuration to use the Vagrant box we want, and updated our tests to ensure that our role is doing what we want, we can now run any of the Molecule commands (test, create, converge, etc) just as we would if we would have started the role using Molecule.

~/Projects/example_playbooks/apache_install$ molecule test
--> Validating schema /home/dan/Projects/example_playbooks/apache_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── lint
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    └── destroy

--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /home/dan/Projects/example_playbooks/apache_install/...
Lint completed successfully.
--> Executing Flake8 on files found in /home/dan/Projects/example_playbooks/apache_install/molecule/default/tests/...
Lint completed successfully.
--> Executing Ansible Lint on /home/dan/Projects/example_playbooks/apache_install/molecule/default/playbook.yml...
Lint completed successfully.
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config] ************************************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=3    changed=2    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'default'
--> Action: 'syntax'

    playbook: /home/dan/Projects/example_playbooks/apache_install/molecule/default/playbook.yml

--> Scenario: 'default'
--> Action: 'create'

    PLAY [Create] ******************************************************************

    TASK [Create molecule instance(s)] *********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config dict] *******************************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Convert instance config dict to a list] **********************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=4    changed=2    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'prepare'

    PLAY [Prepare] *****************************************************************

    TASK [Install python for Ansible] **********************************************
    ok: [apache]

    PLAY RECAP *********************************************************************
    apache                     : ok=1    changed=0    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'converge'

    PLAY [Converge] ****************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [apache]

    TASK [apache_install : install apache] *****************************************
    changed: [apache]

    TASK [apache_install : ensure apache running and enabled] **********************
    changed: [apache]

    PLAY RECAP *********************************************************************
    apache                     : ok=3    changed=2    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'idempotence'
Idempotence completed successfully.
--> Scenario: 'default'
--> Action: 'side_effect'
Skipping, side effect playbook not configured.
--> Scenario: 'default'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/dan/Projects/example_playbooks/apache_install/molecule/default/tests/...
    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.12, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
    rootdir: /home/dan/Projects/example_playbooks/apache_install/molecule/default, inifile:
    plugins: testinfra-1.14.1
collected 3 items                                                              

    tests/test_default.py ...                                                [100%]

    =========================== 3 passed in 5.62 seconds ===========================
Verifier completed successfully.
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config] ************************************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=3    changed=2    unreachable=0    failed=0

Conclusion

Molecule not only provides great defaults and a consistent directory structure when creating a new role, but also makes it easy and efficient to add a Molecule workflow for testing existing roles. Adding Molecule scenarios to existing roles is simple and efficient for testing existing roles across Operating Systems and Ansible versions to improve their reliability.

Ansible role creation with Molecule

Molecule is a way to quickly create and test Ansible roles. It acts as a wrapper around various platforms (GCE, VirtualBox, Docker, LXC, etc) and provides easy commands for linting, running, and testing roles. There’s a bit of a learning curve in figuring out what its doing, and what it wants, but that time is well made up with the productivity increase in using it effectively.

Installation

Molecule and Ansible can be installed via pip. I typically run on a Fedora system, and have run into issues with libselinux when using a virtual environment. A quick search provides some work arounds, but really it’s easiest to just use the --user flag to install molecule with the user scheme.

pip install --upgrade --user ansible
pip install --ugprade --user molecule

If you don’t already have ansible/molecule installed, that’ll give you some significant output. Pip is good about drawing attention to errors (though the resolution may not always be terribly clear), but the last couple lines of output will provide libraries and their versions that were installed.

Getting started

I have a ~/Projects directory that fills up with half finished projects on my personal computer. Really this works to consolidate things rather than filling up ~/. Wherever you keep your projects, to get started just create a playbooks directory.

~/$ mkdir ~/Projects/example_playbooks
~/$ cd ~/Projects/example_playbooks

Since I’m not including the installation output, below provides the software versions of this example. Both Ansible and Molecule move quick and do have some significant (but not necessarily breaking) changes between point releases, these instructions might not work verbatim if the version numbers vary significantly.

~/Projects/example_playbooks$ ansible --version
ansible 2.6.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/dan/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/dan/.local/lib/python2.7/site-packages/ansible
  executable location = /home/dan/.local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]
~/Projects/example_playbooks$ molecule --version
molecule, version 2.17.0
~/Projects/example_playbooks$ tree
.

0 directories, 0 files
~/Projects/example_playbooks$

Create a role

Molecule has pretty excellent help output with molecule --help.. In this example, we’re going to create a role with molecule and use the vagrant provider. molecule defaults to Docker for provisioning, but I prefer to use vagrant with VirtualBox (see section below for why I prefer that).

Creating a role, and specifying the name and driver will create a role directory structure.

~/Projects/example_playbooks$ molecule init role --role-name nginx_install --driver-name vagrant
--> Initializing new role nginx_install...
Initialized role in /home/dan/Projects/example_playbooks/nginx_install successfully.
~/Projects/example_playbooks$ tree
.
└── nginx_install
    ├── defaults
    │   └── main.yml
    ├── handlers
    │   └── main.yml
    ├── meta
    │   └── main.yml
    ├── molecule
    │   └── default
    │       ├── INSTALL.rst
    │       ├── molecule.yml
    │       ├── playbook.yml
    │       ├── prepare.yml
    │       └── tests
    │           ├── test_default.py
    │           └── test_default.pyc
    ├── README.md
    ├── tasks
    │   └── main.yml
    └── vars
        └── main.yml

9 directories, 12 files

As you can see, that command creates quite a few directories. Most of these are standard/best-practices for Ansible.

defaults – default values to variables for the role
handlers – specific handlers to notify based on actions in Ansible
meta – Ansible-Galaxy info for the role if you are uploading this to Ansible-Galaxy
molecule – molecule specific information (configuration, instance information, playbooks to run with molecule, etc)
README.md – Information about the role. Well documented, excellent feature (I’m a big fan of documentation, should be obvious if you’re reading this)
tasks – tsaks for the role
vars – other variables for the role

Why I prefer Vagrant and VirtualBox over Docker

Docker is great, don’t get me wrong. I’m a huge proponent of Linux. Docker on Mac, vs Docker on Linux, vs Docker on Windows are all different things. VirtualBox is far more cross platform than Docker. Something that can be done on Fedora is far different than the latest iteration of OSX, and even more different than Windows. Also, using a VPN client that dictates IP routes causes serious issues in networking between Docker containers. Finally, Systemd with Docker requires specific images and root access specifying mounting a cgroup volume. Since the majority of the work I do is not using Docker for orchestration and instead relies on services running on systemd, a VM is a better solution for my use-case (and closer to “production”) than a container. Yes, a container is far lighter than a VM, but not an issue with decently modern hardware (for my use case). Ultimately, while I might be able to make something work for Docker locally on my system, odds are it’s not going to work for anyone not running the same OS/Distro.

Modifications from default Molecule

There are a few defaults I always change when using molecule as it uses Cookie-Cutter to create a default configuration. The first, molecule defaults to Ubuntu, but almost all of the systems I interact with are RHEL based. Also I prefer to specify the memory and CPUs rather than relying on the box defaults. Finally, we’re using nginx in this example we may as well set up port forwarding to hit the webserver locally..

These changes are made through modification of the molecule/default/molecule.yml file to look like something below. The molecule.yml is the configuration used by molecule for instances, tests, provisioning, etc.

Heads up, a raw copy/pasta of below will result in an error. Read on to see why

~/Projects/example_playbooks/nginx_install$ cat molecule/default/molecule.yml 
---
dependency:
  name: galaxy
driver:
  name: vagrant
  provider:
    name: virtualbox
lint:
  name: yamllint
platforms:
  - name: nginx_install
    box: centos/7
    instance_raw_config_args:
      - "vm.network 'forwarded_port', guest: 80, host: 9000"
    memory: 512
    cpus: 1
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

Once we’ve got the molecule configuration to our liking, time to start working on the role itself. Ansible role tasks are in tasks/main.yml for the role. This example is pretty simple, so all we’re doing is installing a repository to install nginx, installing nginx, and starting/enabling nginx. The only Ansible modules we need for this is yum for package installation, and systemd to start and enable the service.

~/Projects/example_playbooks/nginx_install$ cat tasks/main.yml 
---
# tasks file for nginx_install
- name: Install epel-release for nginx
  yum:
    name: epel-release
    state: present
  become: "yes"

- name: install nginx
  yum:
    name: nginx
    state: present
  become: "yes"

- name: ensure nginx running and enabled
  systemd:
    name: nginx
    state: started
    enabled: "yes"
  become: "yes"

Molecule does some great things. It handles the Orchestration of the virtual environment to test, lints Ansible syntax, runs a test suite, and even lints that test suite, as well as destroying the orchestrated environment at the end.

Writing tests for the role

We can manually test the role with some SSHing and curl, but testinfra is included as the default test step of molecule. Testinfra uses pytest and makes it easy to test the system after the role is run to ensure the role has the results that we expected.

This role is pretty simple, so our tests are pretty simple. Since we’re just installing and starting nginx, there’s not a whole lot more we’re looking for in our test. Of course molecule provides a good default, and testinfra documentation even uses nginx in their quickstart.

Tests – quantity or quality?

The tests below are three tests that are all pretty simple. The overall count of tests really doesn’t matter. Below we’ve got three tests, but we could easily have one, or five. This may vary based on the Test Developer, but I chose the three below because it follows a pretty logical order.

Make sure nginx is installed
Make sure nginx configuration was installed correctly
Make sure nginx is running and enabled

This is easiest looking at it backwards. If we had one test to see if nginx is running, if that fails do we have any idea why? Was it installed? Was the configuration wrong? Was it not started? My approach is to first make sure it is installed, if not, the rest of our tests fail and we can see pretty easily why. So we see it’s installed, so next we check if the configuration exists (in a more elaborate example, we’d probably check to make sure there is some expected text in the configuration file). Finally, we make sure nginx is running and enabled. The tests follow a logical flow of prerequisites to get to our ultimate state, and knock out some troubleshooting steps along the way.

cat molecule/default/tests/test_default.py
import os

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    os.environ['MOLECULE_INVENTORY_FILE']).get_hosts('all')


def test_nginx_installed(host):
    nginx = host.package('nginx')
    assert nginx.is_installed

def test_nginx_config_exists(host):
    nginx_config = host.file('/etc/nginx/nginx.conf')
    assert nginx_config.exists

def test_nginx_running(host):
    nginx_service = host.service('nginx')
    assert nginx_service.is_running
    assert nginx_service.is_enabled

Running Molecule

We’ve got our role written, and our tests. We could just run molecule test and work through all the steps. But, I prefer running create, converge, and test all separately, and in that order. This separates the various steps and makes any fails easier to track down.

Molecule create

The first step of Molecule is the creation of the VirtualMachine. For Docker and vagrant providers, Molecule includes a default create playbook. Running molecule create will create the VirtualMachine for our role based on the molecule.yml configuration.

~/Projects/example_playbooks/nginx_install$ molecule create
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── create
    └── prepare

--> Scenario: 'default'
--> Action: 'create'

    PLAY [Create] ******************************************************************

    TASK [Create molecule instance(s)] *********************************************
    failed: [localhost] (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
    fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

    PLAY RECAP *********************************************************************
    localhost                  : ok=0    changed=0    unreachable=0    failed=1   


ERROR:

This was entirely unplanned, but as I was gathering output for this command I got an error. Ansible has a no_log property for tasks that is intended to prevent the outputting secrets. Obviously in the create part here we received an error with no usable output to determine the cause of the error. We can set the environment variable of MOLECULE_DEBUG to log errors, but the first thing I do (because it’s less typing) is add the --debug flag.

~/Projects/example_playbooks/nginx_install$ molecule --debug create
...
        }, 
        "item": {
            "box": "centos/7", 
            "cpus": 1, 
            "instance_raw_config_args": [
                "vm.network 'forwarded_port', guest: 80, host: 9000"
            ], 
            "memory": 512, 
            "name": "nginx_install"
        }, 
        "msg": "ERROR: See log file '/tmp/molecule/nginx_install/default/vagrant-nginx_install.err'"
    }

    PLAY RECAP *********************************************************************
    localhost                  : ok=0    changed=0    unreachable=0    failed=1

Reading into the error tells us it was an “error” in Vagrant, and not necessarily one with molecule itself. We can look at the file provided in the error output for more clues.

~/Projects/example_playbooks/nginx_install$ cat /tmp/molecule/nginx_install/default/vagrant-nginx_install.err
### 2018-09-07 17:32:59 ###
### 2018-09-07 17:32:59 ###
There are errors in the configuration of this machine. Please fix
the following errors and try again:

vm:
* The hostname set for the VM should only contain letters, numbers,
hyphens or dots. It cannot start with a hyphen or dot.

### 2018-09-07 17:33:20 ###
### 2018-09-07 17:33:20 ###
There are errors in the configuration of this machine. Please fix
the following errors and try again:

vm:
* The hostname set for the VM should only contain letters, numbers,
hyphens or dots. It cannot start with a hyphen or dot.

Well, that’s easy. Our hostname can’t contain _. A quick edit to the molecule.yml should fix this right up.

~/Projects/example_playbooks/nginx_install$ grep -A1 platform molecule/default/molecule.yml 
platforms:
  - name: nginx-install

Try again on the create:

~/Projects/example_playbooks/nginx_install$ molecule create
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── create
    └── prepare

--> Scenario: 'default'
--> Action: 'create'

    PLAY [Create] ******************************************************************

    TASK [Create molecule instance(s)] *********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config dict] *******************************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Convert instance config dict to a list] **********************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=4    changed=2    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'prepare'

    PLAY [Prepare] *****************************************************************

    TASK [Install python for Ansible] **********************************************
    ok: [nginx-install]

    PLAY RECAP *********************************************************************
    nginx-install              : ok=1    changed=0    unreachable=0    failed=0

Molecule converge

Molecule create only acts as orchestration. The converge step is what runs our playbook that calls our role. There’s good reason to do these steps separate. First, the create step ensures our VirtualMachine is provisioned and started correctly. Once it’s up, we’ve got less troubleshooting when actually running the playbook.

When first learning Ansible or working on a more complicated role, we could just run converge after every task added (or every few depending on our confidence) to our role to make sure it does what we intend for it to do. Because we only have three simple tasks, we can run converge to test all tasks at the same time.

~/Projects/example_playbooks/nginx_install$ molecule converge
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── dependency
    ├── create
    ├── prepare
    └── converge

--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'default'
--> Action: 'create'
Skipping, instances already created.
--> Scenario: 'default'
--> Action: 'prepare'
Skipping, instances already prepared.
--> Scenario: 'default'
--> Action: 'converge'

    PLAY [Converge] ****************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [nginx-install]

    TASK [nginx_install : Install epel-release for nginx] **************************
    changed: [nginx-install]

    TASK [nginx_install : install nginx] *******************************************
    changed: [nginx-install]

    TASK [nginx_install : ensure nginx running and enabled] ************************
    changed: [nginx-install]

    PLAY RECAP *********************************************************************
    nginx-install              : ok=4    changed=3    unreachable=0    failed=0

Cool. It worked. We think, anyway. While our playbooks ran, running our tests will really make sure that it all worked.

Molecule test

Next we run test. This goes through all the steps and will tell us if what we think we’re doing is actually working based our our testinfra tests. Destroying any existing VirtualMachine, checking syntax on role, creating the VirtualMachine, linting and running our tests, etc. If there are any issues, this should let us know.

~/Projects/example_playbooks/nginx_install$ molecule test
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── lint
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    └── destroy

--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /home/dan/Projects/example_playbooks/nginx_install/...
Lint completed successfully.
--> Executing Flake8 on files found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/...
    /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/test_default.py:13:1: E302 expected 2 blank lines, found 1
    /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/test_default.py:17:1: E302 expected 2 blank lines, found 1
    /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/test_default.py:21:1: W391 blank line at end of file
An error occurred during the test sequence action: 'lint'. Cleaning up.
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config] ************************************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=3    changed=2    unreachable=0    failed=0

Another unintended failure. Lint issues in the python tests. Flake provides excellent output for pep errors, so we know exactly what to fix based on the output.

Addressing those issues and rerunning results in the following:

~/Projects/example_playbooks/nginx_install$ molecule test
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    ├── lint
    ├── destroy
    ├── dependency
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    └── destroy

--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /home/dan/Projects/example_playbooks/nginx_install/...
Lint completed successfully.
--> Executing Flake8 on files found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/...
Lint completed successfully.
--> Executing Ansible Lint on /home/dan/Projects/example_playbooks/nginx_install/molecule/default/playbook.yml...
Lint completed successfully.
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Populate instance config] ************************************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    skipping: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=2    changed=0    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'dependency'
Skipping, missing the requirements file.
--> Scenario: 'default'
--> Action: 'syntax'

    playbook: /home/dan/Projects/example_playbooks/nginx_install/molecule/default/playbook.yml

--> Scenario: 'default'
--> Action: 'create'

    PLAY [Create] ******************************************************************

    TASK [Create molecule instance(s)] *********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config dict] *******************************************
    ok: [localhost] => (item=None)
    ok: [localhost]

    TASK [Convert instance config dict to a list] **********************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=4    changed=2    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'prepare'

    PLAY [Prepare] *****************************************************************

    TASK [Install python for Ansible] **********************************************
    ok: [nginx-install]

    PLAY RECAP *********************************************************************
    nginx-install              : ok=1    changed=0    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'converge'

    PLAY [Converge] ****************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [nginx-install]

    TASK [nginx_install : Install epel-release for nginx] **************************
    changed: [nginx-install]

    TASK [nginx_install : install nginx] *******************************************
    changed: [nginx-install]

    TASK [nginx_install : ensure nginx running and enabled] ************************
    changed: [nginx-install]

    PLAY RECAP *********************************************************************
    nginx-install              : ok=4    changed=3    unreachable=0    failed=0


--> Scenario: 'default'
--> Action: 'idempotence'
Idempotence completed successfully.
--> Scenario: 'default'
--> Action: 'side_effect'
Skipping, side effect playbook not configured.
--> Scenario: 'default'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/...
    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.12, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
    rootdir: /home/dan/Projects/example_playbooks/nginx_install/molecule/default, inifile:
    plugins: testinfra-1.14.1
collected 3 items                                                              

    tests/test_default.py ...                                                [100%]

    =========================== 3 passed in 5.33 seconds ===========================
Verifier completed successfully.
--> Scenario: 'default'
--> Action: 'destroy'

    PLAY [Destroy] *****************************************************************

    TASK [Destroy molecule instance(s)] ********************************************
    changed: [localhost] => (item=None)
    changed: [localhost]

    TASK [Populate instance config] ************************************************
    ok: [localhost]

    TASK [Dump instance config] ****************************************************
    changed: [localhost]

    PLAY RECAP *********************************************************************
    localhost                  : ok=3    changed=2    unreachable=0    failed=0

Great! With all that, now we know that our Ansible and python tests are linted, and our tests run meaning our role does what we intend for it to do.

Molecule verify

I kind of skipped a step here. I’ve described the steps of:

molecule create – create the VirtualMachine to make sure molecule is configured correctly.
molecule converge – run multiple times as we add tasks to our role.
molecule test – once we’re happy, run all the steps of Molecule.

Really though, since molecule test runs through all the steps (creation, linting, testing, deletion…), and earlier I laid out the steps of running converge to manually test each time, this doesn’t really fit the workflow I metioned. We can seperate out the molecule steps a little further.

Rather than running molecule test, instead we can run molecule verify seperately:

~/Projects/example_playbooks/nginx_install$ molecule verify
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    └── verify

--> Scenario: 'default'
--> Action: 'verify'
--> Executing Testinfra tests found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/...
    ============================= test session starts ==============================
    platform linux2 -- Python 2.7.12, pytest-3.3.1, py-1.5.2, pluggy-0.6.0
    rootdir: /home/dan/Projects/example_playbooks/nginx_install/molecule/default, inifile:
    plugins: testinfra-1.14.1
collected 3 items                                                              

    tests/test_default.py ...                                                [100%]

    =========================== 3 passed in 5.25 seconds ===========================
Verifier completed successfully.
~/Projects/example_playbooks/nginx_install$ molecule lint
--> Validating schema /home/dan/Projects/example_playbooks/nginx_install/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
    └── lint

--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /home/dan/Projects/example_playbooks/nginx_install/...
Lint completed successfully.
--> Executing Flake8 on files found in /home/dan/Projects/example_playbooks/nginx_install/molecule/default/tests/...
Lint completed successfully.
--> Executing Ansible Lint on /home/dan/Projects/example_playbooks/nginx_install/molecule/default/playbook.yml...
Lint completed successfully.

Conclusion

Molecule is a great abstraction for multiple steps to the create/test/clean up steps of testing an Ansible role during development. Not only does it create and provide sane defaults to the directory structure of a role, it makes it easy to create a test a role during development. While there may be a bit of a learning curve, the increased productivity of testing during development makes it an absolutely worthwhile investment.

Vagrant

Simply put, Vagrant is software which provides orchestration for a number of virtualization technologies. It’s open source, and created by HashiCorp. Vagrant allows for defining the properties of a virtual machine such as memory, CPUs, name, provisioning, etc, and allows for the quick creation and access of that virtual machine.

Vagrant is an incredibly powerful tool in both development and operations. Whether its for a shared development environment of a legacy application or testing server configuration management tools across varying operating system, or development of anything in between, adding Vagrant to your workflow is incredibly useful.

Key Terms

Vagrant has a few key terms to keep in mind during an introduction.

Vagrantfile
- Configuration file defining the machine(s) to create
Provider
- Virtualization Platform (libvirt, VirtualBox, docker, etc)
Box
- Base image (Operating system + configuration to use)
Provisioner
- Additional configuration management to be completed after orchestration is complete
Plugin
- extends the functionality of Vagrant

Vagrantfile

The Vagrantfile is a ruby configuration file that defines the machine or machines for orchestration. A default Vagrant file will be created in the working directory on running vagrant init. Exact configuration options vary by provider, meaning some configuration specifics may be available to libvirt but not VirtualBox.

Providers

A Vagrant Provider is the virtualization platform that Vagrant interacts with. Fedora defaults to libvirt, but this can instead be VirtualBox, docker, even a cloud provider such as Rackspace or AWS. This can be defined at the command line or more conveniently via an environment variable.

Box

A box is the image to use with Vagrant. This is defined as the config.vm.box to set the base image of the machine to be created. Boxes can be custom created and stored in an artifact repository, or can be found and retrieved directly from Vagrant. A box can have default parameters, such as the number of CPUs and memory, that can be overriden via the Vagrantfile.

Once the Vagrantfile is defined and vagrant up is run for the first time, the box is pulled from a remote respository. Once the box is pulled its cached locally, meaning that there isn’t a large download each time a box is spun up. The default configuration is also to include checks to determine if a box has an update, which can be updated with a separate command (vagrant box update).

Provisioner

A provisioner is a method for hooking in further configuration management after the virtual machine orchestrated by Vagrant is up and accessible. This can be as simple as running a few bash commands, to as complex as running an ansible playbook to connect to an existing salt master. The big configuration managment methods are supported (Ansible, Salt, Chef, Puppet…) so its easy to set up the virtual machine exactly how you want or even test new or existing playbooks/manifests/state files to run on your existing infrastructure.

Plugin

Plugins (as expected) are ways to increase the functionality and features of Vagrant. These include interactions with cloud providers, methods for creating custom boxes, and an method for creating your own needed functionality through custom plugins.

Using Vagrant

Vagrant configurations can be as complex as your imagination can create, but getting started is super easy. For this example I’m going to go over how to set up a quick Jenkins instance I used to test Jenkins Job Builder files.

Installation

For a simple example all you need is Vagrant and VirtualBox. Follow the installation instructions for your Operating System as this obviously varies.

Environment Variable configuration

I use Fedora which defaults Vagrant to using the libvirt provider. This is all well and good, but to ensure cross platformness of my Vagrantfiles, I prefer to use VirtualBox. This can be overridden by setting the VAGRANT_DEFAULT_PROVIDER environment variable to virtualbox. If you’re not familiar, this can be done via export VAGRANT_DEFAULT_PROVIDER=virtualbox.

Vagrantfile Configuration

In a new and empty directy, create a generic new Vagrantfile with:

vagrant init

This creates a well commented vagrantfile that I’d recommend taking a few minutes taking a look at. We’ll need to modify this just a bit. The comments can be left in, but in the end we need the following lines:

Vagrant.configure("2") do |config|
  config.vm.box = "centos/7"
  config.vm.network "forwarded_port", guest: 8080, host: 8080
  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"
    vb.cpus = 2
  end
  config.vm.provision "shell", inline: <<-SHELL
    sudo yum install -y wget
    sudo yum update -y
    sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
    sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
    sudo yum install -y jenkins
    sudo yum install -y java-1.8.0-openjdk
    sudo yum install -y vim git ansible python-requests unzip
    sudo systemctl start jenkins
    sleep 300
    cat /var/lib/jenkins/secrets/initialAdminPassword
  SHELL
end

Basic config

One nice thing about this configuration is that it is very human readable. We’re using the basic CentOS7 image available for our base and making port 8080 on the virtual machine accessible on the localhost through port forwarding. Next we set the configuration to be two CPUs and 2GB of memory.

Provisioning

To provision the virtual machine, this just just using bash commands. Theres a lot of benefits to using server configuration management such as Ansible for this, but for this simple example we’re just running a handful of commands specific to RHEL/CentOS 7.

The provision step updates all packages via yum, installs epel-release to allow the installation of other packages (git, ansible), installs java, then installs and enables the latest official Jenkins LTS. Finally, it starts Jenkins and outputs the username and password of the admin user so we can log into Jenkins.

Running the example

Spin it up with vagrant up. Not including all output for brevity:

$ vagrant  up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'centos/7'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'centos/7' is up to date...
...
==> default:   python2-jmespath.noarch 0:0.9.0-3.el7
==> default:   python2-pyasn1.noarch 0:0.1.9-7.el7
==> default:   sshpass.x86_64 0:1.06-2.el7
==> default:   vim-common.x86_64 2:7.4.160-4.el7
==> default:   vim-filesystem.x86_64 2:7.4.160-4.el7
==> default: Complete!
==> default: 3297c139343744c9a05c235e6d73762e

The virtual machine can be accessed via vagrant ssh. Vagrant takes care of the host and port forwarding.

$ vagrant ssh
[vagrant@localhost ~]$ hostname
localhost.localdomain

Ideally, you wouldn’t need to log into the virtual machine and make changes, but its always handy to be able to troubleshoot or add further configuration. If you find yourself needing to log in to virtual machine, it is probably something that should be added to the provisioner step in the Vagrantfile for the next time you spin it up.

Access the Jenkins instance by opening your browser to http://localhost:8080. Setup and configuration of Jenkins is beyond the scope of this post.

If you’re all done with the virtual machine, or at some point changes were made to the point that you need to start fresh, this virtual machine can be completely removed via vagrant destroy.

$ vagrant  destroy
    default: Are you sure you want to destroy the 'default' VM? [y/N] y
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...

Overview

Vagrant is an easy and programmatic way to spin up a local virtual machine for review or test software, configuration management, or just toying around with a different operating system. Thanks to the fairly shallow learning curve in getting started with Vagrant, it is a great way to add to your productivity in a wide range of development or operations tasks.