Ansible – HAProxy

Ansible – HAProxy

In this previous post I setup KeepAliveD using a fictitious tenant using Ansible. In this post I will be building upon that same configuration and creating the HAProxy setup.

Below is the vars/tenant_1.yml file that contains the specific tenant variables that will be used.

---
tenant_name: tenant_1
config_forward_rules_allow_spec: 'false'    # set to true to configure specific firewall rules under forward_rules_allow_spec

tenant_subnets:
  - { tenant_subnet: '192.168.70.0/24' }  # Web
  - { tenant_subnet: '192.168.71.0/24' }  # App
  - { tenant_subnet: '192.168.72.0/24' }  # DB

tenant_vips:
  - 10.10.10.100
  - 10.10.10.101
  - 10.10.10.102
  - 10.10.10.103
  - 10.10.10.104

###### Firewall Setup ######
###### Note.....Firewall rules are by default dropped in default setup ######

# Port specific rules - used to add firewall rules that should be port based rules **Note...SSH is allowed by default so there is no need to specify rules for SSH.
# Another note is that when configuring load balancer setup further down...The ports that will be load balanced will be added to the firewall by default as well.
# This section will normally be used to allow a specific port to a host|subnet etc. which does not require load balancing.
# Make sure to change config_forward_rules_allow_spec at the top to true to ensure the below firewall rules are applied.
forward_rules_allow_spec:
  - { protocol: 'tcp', port: '8080', source: '0.0.0.0/0', destination: '192.168.70.0/24' }

# Generic non port specific rules to allow. Ex. allowing a host|subnet to communicate from|to one another on every port available.
forward_rules_allow_gen:
  - { source: '192.168.70.0/24', destination: '0.0.0.0/0' }
  - { source: '192.168.71.0/24', destination: '0.0.0.0/0' }
  - { source: '192.168.72.0/24', destination: '0.0.0.0/0' }

# Define specific rules below to be dropped by firewall. Ex. explicitly denying communications between hosts|subnets.
forward_rules_out_drop:
  - { source: '192.168.70.0/24', destination: '192.168.71.0/24' }
  - { source: '192.168.70.0/24', destination: '192.168.72.0/24' }
  - { source: '192.168.71.0/24', destination: '192.168.70.0/24' }
  - { source: '192.168.71.0/24', destination: '192.168.72.0/24' }
  - { source: '192.168.72.0/24', destination: '192.168.70.0/24' }
  - { source: '192.168.72.0/24', destination: '192.168.71.0/24' }


###### Load Balancer Setup ######

# Allocate vips from tenant_vips assigned above....these will be used as variables for lb_details and lb_defs. You may choose to just enter IP addresses though.
web_vip: '10.10.10.100'
db_vip: '10.10.10.101'
app_vip: '10.10.10.102'

balance_method: 'roundrobin'         # Set to one of the below types to configure load balancing method
#leastconn - The server with the lowest number of connections receives the connection
#roundrobin - Each server is used in turns, according to their weights.
#source - Source IP hashed and divided by total weight of servers designates which server will receive the request

# Defines the load balancing group setup
lb_details:
  - { name: 'web', protocol: 'tcp', listen_port: '80', tenant_vip: '{{ web_vip }}', balance_type: '{{ balance_method }}' } 
  - { name: 'db', protocol: 'tcp', listen_port: '3306', tenant_vip: '{{ db_vip }}', balance_type: '{{ balance_method }}' }
  - { name: 'rabbitmq-mgmt', protocol: 'tcp', listen_port: '15672', tenant_vip: '{{ app_vip }}', balance_type: '{{ balance_method }}' }
  - { name: 'redis', protocol: 'tcp', listen_port: '6379', tenant_vip: '{{ app_vip }}', balance_type: '{{ balance_method }}' }
  - { name: 'rabbitmq', protocol: 'tcp', listen_port: '5672', tenant_vip: '{{ app_vip }}', balance_type: '{{ balance_method }}' }

# Defines the load balancing servers within the load balancing group
lb_defs:
  - { lb_def_name: 'web', protocol: 'tcp', listen_port: '80', tenant_vip: '{{ web_vip }}', lb_group: 'web', server: 'ans-cloud-web01', backend_port: '80' }
  - { lb_def_name: 'web', protocol: 'tcp', listen_port: '80', tenant_vip: '{{ web_vip }}', lb_group: 'web', server: 'ans-cloud-web02', backend_port: '80' }
  - { lb_def_name: 'web', protocol: 'tcp', listen_port: '80', tenant_vip: '{{ web_vip }}', lb_group: 'web', server: 'ans-cloud-web03', backend_port: '80' }
  - { lb_def_name: 'db', protocol: 'tcp', listen_port: '3306', tenant_vip: '{{ db_vip }}', lb_group: 'db', server: 'ans-cloud-db01', backend_port: '3306' }
  - { lb_def_name: 'db', protocol: 'tcp', listen_port: '3306', tenant_vip: '{{ db_vip }}', lb_group: 'db', server: 'ans-cloud-db02', backend_port: '3306' }
  - { lb_def_name: 'rabbitmq-mgmt', protocol: 'tcp', listen_port: '15672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq-mgmt', server: 'ans-cloud-app01', backend_port: '15672' }
  - { lb_def_name: 'rabbitmq-mgmt', protocol: 'tcp', listen_port: '15672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq-mgmt', server: 'ans-cloud-app02', backend_port: '15672' }
  - { lb_def_name: 'rabbitmq-mgmt', protocol: 'tcp', listen_port: '15672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq-mgmt', server: 'ans-cloud-app03', backend_port: '15672' }
  - { lb_def_name: 'redis', protocol: 'tcp', listen_port: '6379', tenant_vip: '{{ app_vip }}', lb_group: 'redis', server: 'ans-cloud-app01', backend_port: '6379' }
  - { lb_def_name: 'redis', protocol: 'tcp', listen_port: '6379', tenant_vip: '{{ app_vip }}', lb_group: 'redis', server: 'ans-cloud-app02', backend_port: '6379' }
  - { lb_def_name: 'redis', protocol: 'tcp', listen_port: '6379', tenant_vip: '{{ app_vip }}', lb_group: 'redis', server: 'ans-cloud-app03', backend_port: '6379' }
  - { lb_def_name: 'rabbitmq', protocol: 'tcp', listen_port: '5672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq', server: 'ans-cloud-app01', backend_port: '5672' }
  - { lb_def_name: 'rabbitmq', protocol: 'tcp', listen_port: '5672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq', server: 'ans-cloud-app02', backend_port: '5672' }
  - { lb_def_name: 'rabbitmq', protocol: 'tcp', listen_port: '5672', tenant_vip: '{{ app_vip }}', lb_group: 'rabbitmq', server: 'ans-cloud-app03', backend_port: '5672' }

Below is the haproxy.cfg.j2 template that I will use.

# {{ ansible_managed }}
global
#        log logstash    local0 #Change logstash to your naming
        log /dev/log    local0
        log /dev/log    local1 notice
#       log-send-hostname
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        maxconn 40000
        spread-checks 3
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
        ssl-default-bind-options no-sslv3




defaults
        log     global
        mode    tcp
        maxconn 40000
        option  httplog
        option  dontlognull
        option redispatch
        option tcp-smart-accept
        option tcp-smart-connect
        retries 3
        timeout queue 5000
        timeout connect 50000
        timeout client 50000
        timeout server 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http


userlist STATSUSERS
        group admin users admin
        user admin insecure-password admin

listen admin_page 0.0.0.0:9090
        mode http
        stats enable
        stats refresh 60s
        stats uri /
        acl AuthOkay_ReadOnly http_auth(STATSUSERS)
        acl AuthOkay_Admin http_auth_group(STATSUSERS) admin
        stats http-request auth realm admin_page unless AuthOkay_ReadOnly

{% for lb_group_def in lb_details %}
listen {{ tenant_name }}_{{ lb_group_def.name }}-{{ lb_group_def.tenant_vip }}:{{ lb_group_def.listen_port }} {{ lb_group_def.tenant_vip }}:{{ lb_group_def.listen_port }}
        mode tcp
        option tcpka
        option tcplog
        #balance leastconn - The server with the lowest number of connections receives the connection
        #balance roundrobin - Each server is used in turns, according to their weights.
        #balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
        balance {{ lb_group_def.balance_type }}
        default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
{% for item in lb_defs %}
{% if item.lb_group == lb_group_def.name %}
        server {{ item.server }} {{ item.server }}:{{ item.backend_port }} check
{% endif %}
{% endfor %}
{% endfor %}

And below is what we end up with as an haproxy.cfg for our setup by using the variables and template above.

# Ansible managed: /home/administrator/ansible/projects/ans-cloud-rt01/templates/etc/haproxy/haproxy.cfg.j2 modified on 2015-04-10 12:58:14 by administrator on ansible
global
#        log logstash    local0 #Change logstash to your naming
        log /dev/log    local0
        log /dev/log    local1 notice
#       log-send-hostname
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        maxconn 40000
        spread-checks 3
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
        ssl-default-bind-options no-sslv3




defaults
        log     global
        mode    tcp
        maxconn 40000
        option  httplog
        option  dontlognull
        option redispatch
        option tcp-smart-accept
        option tcp-smart-connect
        retries 3
        timeout queue 5000
        timeout connect 50000
        timeout client 50000
        timeout server 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http


userlist STATSUSERS
        group admin users admin
        user admin insecure-password admin

listen admin_page 0.0.0.0:9090
        mode http
        stats enable
        stats refresh 60s
        stats uri /
        acl AuthOkay_ReadOnly http_auth(STATSUSERS)
        acl AuthOkay_Admin http_auth_group(STATSUSERS) admin
        stats http-request auth realm admin_page unless AuthOkay_ReadOnly

listen tenant_1_web-10.10.10.100:80 10.10.10.100:80
        mode tcp
        option tcpka
        option tcplog
        #balance leastconn - The server with the lowest number of connections receives the connection
        #balance roundrobin - Each server is used in turns, according to their weights.
        #balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
        balance roundrobin
        default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
        server ans-cloud-web01 ans-cloud-web01:80 check
        server ans-cloud-web02 ans-cloud-web02:80 check
        server ans-cloud-web03 ans-cloud-web03:80 check
listen tenant_1_db-10.10.10.101:3306 10.10.10.101:3306
        mode tcp
        option tcpka
        option tcplog
        #balance leastconn - The server with the lowest number of connections receives the connection
        #balance roundrobin - Each server is used in turns, according to their weights.
        #balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
        balance roundrobin
        default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
        server ans-cloud-db01 ans-cloud-db01:3306 check
        server ans-cloud-db02 ans-cloud-db02:3306 check
listen tenant_1_rabbitmq-mgmt-10.10.10.102:15672 10.10.10.102:15672
        mode tcp
        option tcpka
        option tcplog
        #balance leastconn - The server with the lowest number of connections receives the connection
        #balance roundrobin - Each server is used in turns, according to their weights.
        #balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
        balance roundrobin
        default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
        server ans-cloud-app01 ans-cloud-app01:15672 check
        server ans-cloud-app02 ans-cloud-app02:15672 check
        server ans-cloud-app03 ans-cloud-app03:15672 check
listen tenant_1_redis-10.10.10.102:6379 10.10.10.102:6379
        mode tcp
        option tcpka
        option tcplog
        #balance leastconn - The server with the lowest number of connections receives the connection
        #balance roundrobin - Each server is used in turns, according to their weights.
        #balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
        balance roundrobin
        default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
        server ans-cloud-app01 ans-cloud-app01:6379 check
        server ans-cloud-app02 ans-cloud-app02:6379 check
        server ans-cloud-app03 ans-cloud-app03:6379 check
listen tenant_1_rabbitmq-10.10.10.102:5672 10.10.10.102:5672
        mode tcp
        option tcpka
        option tcplog
        #balance leastconn - The server with the lowest number of connections receives the connection
        #balance roundrobin - Each server is used in turns, according to their weights.
        #balance source - Source IP hashed and divided by total weight of servers designates which server will receive the request
        balance roundrobin
        default-server inter 2s downinter 5s rise 3 fall 2 slowstart 60s maxconn 256 maxqueue 128 weight 100
        server ans-cloud-app01 ans-cloud-app01:5672 check
        server ans-cloud-app02 ans-cloud-app02:5672 check
        server ans-cloud-app03 ans-cloud-app03:5672 check

 

Below is the group_vars code used across the different Ansible posts based on tenant setup

Leave a Reply

Your email address will not be published. Required fields are marked *

*