Ansible - Provision Docker Swarm Mode (1.12)

Lately I have been working quite a bit with the latest Docker Swarm Mode released in Docker 1.12 and so far it has been pretty awesome. The days of spinning up a Docker Swarm cluster with all of the complexity (Consul, Registrator, and etc.) are old-school now. To do a comparison you can checkout a great post by Scott Lowe here and also checkout a Vagrant lab that I put together for learning almost a year ago here. With the latest release of Docker 1.12 they have made the process SO much easier and much less complex. To read up on the functionality and ease head over here. Now the purpose of this post is not to go step by step on how easy it now is to provision a cluster but rather to take it to another level. Let’s automate the provisioning using Ansible instead. This allows for a much more consistent and predictable provisioning method as well as the ability to easily scale your cluster(s). In addition to this post I highly recommend you fork or clone my GitHub repo to spin up a 7-node Docker Swarm Mode cluster completely automated and provisioned using Ansible and Vagrant. You can read more about the usage and some examples within that repo as well. So what I WILL cover in this post is the process along with examples of how to use Ansible to provision a Docker Swarm Mode cluster.

• Assumptions:
  • You have already provisioned your nodes in which will comprise your Docker Swarm Mode cluster
    • OS Installed (Ubuntu 16.04 in my case) as well as Docker Engine installed…Checkout my Ansible role for this as well here
  • Identified the interface which will be used for all Docker Swarm Mode communications and such (enp0s8 in my case)

So the first thing we will ensure is correct is our Ansible inventory and groups. We need to define the overall Docker nodes group (docker-nodes) as well as which nodes are considered managers (docker-swarm-managers) and which nodes are workers (docker-swarm-workers). Below is an example of the Ansible inventory that I use:

[docker-nodes]
node[0:6]

[docker-swarm-managers]
node[0:2]

[docker-swarm-workers]
node[3:6]
```jinja2

Now we need to define some Ansible variables which are required to
successfully provision our cluster:



```yaml
docker_swarm_addr: "{{ hostvars[inventory_hostname]['ansible_' + docker_swarm_interface]['ipv4']['address'] }}"
docker_swarm_cert_expiry: '2160h0m0s' # Validity period for node certificates (default 2160h0m0s)
docker_swarm_dispatcher_heartbeat_duration: '5s' # Dispatcher heartbeat period (default 5s)
docker_swarm_interface: "enp0s8"
docker_swarm_managers_ansible_group: 'docker-swarm-managers'
docker_swarm_networks:
  - name: 'my_net'
    driver: 'overlay'
    state: 'present'
  - name: 'test'
    driver: 'overlay'
    state: 'absent'
docker_swarm_primary_manager: '{{ groups[docker_swarm_managers_ansible_group][0] }}'
## docker_swarm_primary_manager: 'node0'
docker_swarm_task_history_limit: '5' # Task history retention limit (default 5)
docker_swarm_workers_ansible_group: 'docker-swarm-workers'
docker_swarm_port: "2377"
```jinja2



As you can see from the above we can define which node will be
considered as the _docker_swarm_primary_manager_ in two different
ways. The first (default) method is to use the Ansible inventory group
name and choose the first node in that group or the second method would
be to just define the actual node name. Some of the other variables are
to define/update actual Docker Swarm cluster settings (default values by
default) as well as defining some Docker networks to manage.

And now for the actual Ansible playbook that we will be using to make
all of the magic happen behind this:

<noscript><pre>---
- hosts: docker-nodes
  become: true
  vars:
    docker_swarm_addr: &quot;{{ hostvars[inventory_hostname][&#39;ansible_&#39; + docker_swarm_interface][&#39;ipv4&#39;][&#39;address&#39;] }}&quot;
    docker_swarm_cert_expiry: &#39;2160h0m0s&#39; # Validity period for node certificates (default 2160h0m0s)
    docker_swarm_dispatcher_heartbeat_duration: &#39;5s&#39; # Dispatcher heartbeat period (default 5s)
    docker_swarm_interface: &quot;enp0s8&quot;
    # docker_swarm_managers_ansible_group: &#39;docker-swarm-managers&#39;
    docker_swarm_networks:
      - name: &#39;my_net&#39;
        driver: &#39;overlay&#39;
        state: &#39;present&#39;
      - name: &#39;test&#39;
        driver: &#39;overlay&#39;
        state: &#39;absent&#39;
    docker_swarm_primary_manager: &#39;{{ groups[docker_swarm_managers_ansible_group][0] }}&#39;
    # docker_swarm_primary_manager: &#39;node0&#39;
    docker_swarm_task_history_limit: &#39;5&#39; # Task history retention limit (default 5)
    # docker_swarm_workers_ansible_group: &#39;docker-swarm-workers&#39;
    docker_swarm_port: &quot;2377&quot;
  tasks:
    - name: docker_swarm | Installing EPEL Repo (RedHat)
      yum:
        name: &quot;epel-release&quot;
        state: &quot;present&quot;
      when: &gt;
            ansible_os_family == &quot;RedHat&quot; and
            ansible_distribution != &quot;Fedora&quot;

    - name: docker_swarm | Installing Pre-Reqs
      apt:
        name: &quot;python-pip&quot;
        state: &quot;present&quot;
      when: ansible_os_family == &quot;Debian&quot;

## Installing these for future functionality
    - name: docker_swarm | Installing Python Pre-Reqs
      pip:
        name: &quot;{{ item }}&quot;
        state: &quot;present&quot;
      with_items:
        - &#39;docker-py&#39;
##

    - name: docker_swarm | Ensuring Docker Engine Is Running
      service:
        name: &quot;docker&quot;
        state: &quot;started&quot;

    - name: docker_swarm | Checking Swarm Mode Status
      command: &quot;docker info&quot;
      register: &quot;docker_info&quot;
      changed_when: false

    - name: docker_swarm | Init Docker Swarm Mode On First Manager
      command: &gt;
              docker swarm init
              --listen-addr {{ docker_swarm_addr }}:{{ docker_swarm_port }}
              --advertise-addr {{ docker_swarm_addr }}
      when: &gt;
            &#39;Swarm: inactive&#39; in docker_info.stdout and
            inventory_hostname == docker_swarm_primary_manager

    - name: docker_swarm | Capturing Docker Swarm Worker join-token
      command: &quot;docker swarm join-token -q worker&quot;
      changed_when: false
      register: &quot;docker_swarm_worker_token&quot;
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager

    - name: docker_swarm | Capturing Docker Swarm Manager join-token
      command: &quot;docker swarm join-token -q manager&quot;
      changed_when: false
      register: &quot;docker_swarm_manager_token&quot;
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager

    - name: docker_swarm | Defining Docker Swarm Manager Address
      set_fact:
        docker_swarm_manager_address: &quot;{{ docker_swarm_addr }}:{{ docker_swarm_port }}&quot;
      changed_when: false
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager

    - name: docker_swarm | Defining Docker Swarm Manager Address
      set_fact:
        docker_swarm_manager_address: &quot;{{ hostvars[docker_swarm_primary_manager][&#39;docker_swarm_manager_address&#39;] }}&quot;
      changed_when: false
      when: &gt;
            inventory_hostname != docker_swarm_primary_manager

    - name: docker_swarm | Defining Docker Swarm Manager join-token
      set_fact:
        docker_swarm_manager_token: &quot;{{ hostvars[docker_swarm_primary_manager][&#39;docker_swarm_manager_token&#39;] }}&quot;
      changed_when: false
      when: &gt;
            inventory_hostname != docker_swarm_primary_manager

    - name: docker_swarm | Defining Docker Swarm Worker join-token
      set_fact:
        docker_swarm_worker_token: &quot;{{ hostvars[docker_swarm_primary_manager][&#39;docker_swarm_worker_token&#39;] }}&quot;
      changed_when: false
      when: &gt;
            inventory_hostname != docker_swarm_primary_manager

    - name: docker_swarm | Joining Additional Docker Swarm Managers To Cluster
      command: &gt;
              docker swarm join
              --listen-addr {{ docker_swarm_addr }}:{{ docker_swarm_port }}
              --advertise-addr {{ docker_swarm_addr }}
              --token {{ docker_swarm_manager_token.stdout }}
              {{ docker_swarm_manager_address }}
      when: &gt;
            inventory_hostname != docker_swarm_primary_manager and
            inventory_hostname not in groups[docker_swarm_workers_ansible_group] and
            &#39;Swarm: active&#39; not in docker_info.stdout and
            &#39;Swarm: pending&#39; not in docker_info.stdout

    - name: docker_swarm | Joining Docker Swarm Workers To Cluster
      command: &gt;
             docker swarm join
             --listen-addr {{ docker_swarm_addr }}:{{ docker_swarm_port }}
             --advertise-addr {{ docker_swarm_addr }}
             --token {{ docker_swarm_worker_token.stdout }}
             {{ docker_swarm_manager_address }}
      when: &gt;
            inventory_hostname in groups[docker_swarm_workers_ansible_group] and
            &#39;Swarm: active&#39; not in docker_info.stdout and
            &#39;Swarm: pending&#39; not in docker_info.stdout

    - name: docker_swarm | Capturing Docker Swarm Networks
      command: &quot;docker network ls&quot;
      changed_when: false
      register: &quot;docker_networks&quot;
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager

    - name: docker_swarm | Creating Docker Swarm Networks
      command: &quot;docker network create --driver {{ item.driver }} {{ item.name }}&quot;
      with_items: &#39;{{ docker_swarm_networks }}&#39;
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager and
            item.state|lower == &quot;present&quot; and
            item.name not in docker_networks.stdout

    - name: docker_swarm | Removing Docker Swarm Networks
      command: &quot;docker network rm {{ item.name }}&quot;
      with_items: &#39;{{ docker_swarm_networks }}&#39;
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager and
            item.state|lower == &quot;absent&quot; and
            item.name in docker_networks.stdout

## Below is for Ansible 2.2
    # - name: docker_swarm | Managing Docker Swarm Networks
    #   docker_network:
    #     name: &quot;{{ item.name }}&quot;
    #     driver: &quot;{{ item.driver }}&quot;
    #     state: &quot;{{ item.state }}&quot;
    #   with_items: &#39;{{ docker_swarm_networks }}&#39;
    #   when: &gt;
    #         inventory_hostname == docker_swarm_primary_manager

    - name: docker_swarm | Updating Docker Swarm Dispatch Heartbeat Duration
      command: &quot;docker swarm update --dispatcher-heartbeat {{ docker_swarm_dispatcher_heartbeat_duration }}&quot;
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager

    - name: docker_swarm | Updating Docker Swarm Certificate Expiry Duration
      command: &quot;docker swarm update --cert-expiry {{ docker_swarm_cert_expiry }}&quot;
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager

    - name: docker_swarm | Updating Docker Swarm Task History Limit
      command: &quot;docker swarm update --task-history-limit {{ docker_swarm_task_history_limit }}&quot;
      when: &gt;
            inventory_hostname == docker_swarm_primary_manager
</pre></noscript><script src="https://gist.github.com/mrlesmithjr/398fa971ccc1355e1ce223851f77c5bf.js"> </script>

As you can see we are actually capturing both the manager and worker
tokens for the cluster as well. This is the key for success here. So how
are we doing this? Let's take a look at the tasks that accomplish this.

First we need to capture the tokens on our
docker_swarm_primary_manager:



```yaml
- name: docker_swarm | Capturing Docker Swarm Worker join-token
  command: "docker swarm join-token -q worker"
  changed_when: false
  register: "docker_swarm_worker_token"
  when: >
        inventory_hostname == docker_swarm_primary_manager

- name: docker_swarm | Capturing Docker Swarm Manager join-token
  command: "docker swarm join-token -q manager"
  changed_when: false
  register: "docker_swarm_manager_token"
  when: >
        inventory_hostname == docker_swarm_primary_manager
```jinja2



If you notice from above we are literally just running the docker swarm
command to query our join-token for the manager and worker tokens and
registering those facts. Easy as that!

Next we need to define the facts for both join-tokens in order for our
additional nodes to successfully join our Swarm cluster. And we do that
with the following tasks:



```yaml
- name: docker_swarm | Defining Docker Swarm Manager join-token
  set_fact:
    docker_swarm_manager_token: "{{ hostvars[docker_swarm_primary_manager]['docker_swarm_manager_token'] }}"
  changed_when: false
  when: >
        inventory_hostname != docker_swarm_primary_manager

- name: docker_swarm | Defining Docker Swarm Worker join-token
  set_fact:
    docker_swarm_worker_token: "{{ hostvars[docker_swarm_primary_manager]['docker_swarm_worker_token'] }}"
  changed_when: false
  when: >
        inventory_hostname != docker_swarm_primary_manager

Now that these facts are registered we can now leverage these on our additional Swarm nodes.

Some additional tasks we are able to do are to also manage our Docker networks including overlay networks for our Swarm cluster. We can also update some additional settings for our Docker Swarm cluster which we do in the last three tasks of the playbook.

So there you have it. An easy way to provision a Docker Swarm (1.12) mode cluster using Ansible allowing for easily scaled out provisioning.

Stay tuned for additional posts coming in the future in regards to some additional Docker Swarm mode goodness.

Enjoy!