Ansible - IP Sets and DShield Block List
In this post, we are going to look into how we can leverage Ansible to manage a Linux firewall using ipset along with a DShield block list. Now, why would we want to do this you might be asking yourself. One good reason for this would be to ensure that we do not allow any traffic inbound or outbound to any IP subnet that shows up in the top 20 attacking class C (/24) subnets over the last three days. But the overall reason is for the hell of it and it is fun!
NOTE: You can checkout my Ansible role ansible-ipset for additional usage.
DShield Block List
As you may or may not already know the DShield block list comes as a text file. So the first thing we need to do is convert this into a consumable format for Ansible. In this case, we will be converting it to YAML.
Example DShield Block List (TXT)
Below we have an example of what a DShield block list looks like in its native TXT format.
block.txt:
#
# DShield.org Recommended Block List
# (c) 2018 DShield.org
# some rights reserved. Details http://creativecommons.org/licenses/by-nc-sa/2.5/
# use on your own risk. No warranties implied.
# primary URL: http://feeds.dshield.org/block.txt
# PGP Sign.: http://feeds.dshield.org/block.txt.asc
#
# comments: [email protected]
# updated: Wed Jan 17 22:00:15 2018 UTC
#
# This list summarizes the top 20 attacking class C (/24) subnets
# over the last three days. The number of 'attacks' indicates the
# number of targets reporting scans from this subnet.
#
#
# Columns (tab delimited):
#
# (1) start of netblock
# (2) end of netblock
# (3) subnet (/24 for class C)
# (4) number of targets scanned
# (5) name of Network
# (6) Country
# (7) contact email address
#
# If a range is assigned to multiple users, the first one is listed.
#
Start End Netmask Attacks Name Country email
77.72.82.0 77.72.82.255 24 9424 NETUP-AS , RU [email protected]
191.101.167.0 191.101.167.255 24 6276 Digital Energy Technologies Chile SpA, CL [email protected]
181.214.87.0 181.214.87.255 24 6193 WZCOM-US - WZ Communications Inc., US [email protected]
5.188.86.0 5.188.86.255 24 5959 PIN-AS , RU [email protected]
77.72.85.0 77.72.85.255 24 4197 NETUP-AS , RU [email protected]
5.188.203.0 5.188.203.255 24 3819 PIN-AS , RU [email protected]
196.52.43.0 196.52.43.255 24 3647 LEASEWEB-NL Netherlands, NL [email protected]
216.158.238.0 216.158.238.255 24 3394 NJIIX-AS-1 - NEW JERSEY INTERNATIONAL INTERNET EXCHANGE LLC, US [email protected]
5.188.10.0 5.188.10.255 24 3309 PIN-AS , RU [email protected]
46.29.162.0 46.29.162.255 24 3186 ASBAXET , RU [email protected]
109.248.9.0 109.248.9.255 24 3100 DGRID , EE [email protected]
80.82.70.0 80.82.70.255 24 2196 QUASINETWORKS , NL [email protected]
185.35.62.0 185.35.62.255 24 2144 KS-ASN1 This ASN is used for Internet security research. Internet-scale port scanning activities are launched from it. Don_t hesitate to contact [email protected] would you have any question., CH [email protected]
93.174.93.0 93.174.93.255 24 1985 QUASINETWORKS , NL [email protected]
141.212.122.0 141.212.122.255 24 1948 UMICH-AS-5 - University of Michigan, US [email protected]
85.93.20.0 85.93.20.255 24 1887 ASGHOSTNET , DE [email protected]
80.82.77.0 80.82.77.255 24 1754 QUASINETWORKS , NL [email protected]
5.188.11.0 5.188.11.255 24 1443 PIN-AS , RU [email protected]
125.212.217.0 125.212.217.255 24 1360 VIETEL-AS-AP Viettel Corporation, VN [email protected]
104.236.178.0 104.236.178.255 24 1329 DIGITALOCEAN-ASN - Digital Ocean, Inc., US [email protected]
Example DShield Block List (YAML)
Below we have an example of what a DShield block list COULD look like in
YAML structure.
block.yml:
---
dshield_block_list:
- start: 77.72.82.0
end: 77.72.82.255
netmask: 24
attacks: 8204
- start: 191.101.167.0
end: 191.101.167.255
netmask: 24
attacks: 4533
- start: 181.214.87.0
end: 181.214.87.255
netmask: 24
attacks: 4405
- start: 5.188.86.0
end: 5.188.86.255
netmask: 24
attacks: 4226
- start: 77.72.85.0
end: 77.72.85.255
netmask: 24
attacks: 3279
- start: 46.29.162.0
end: 46.29.162.255
netmask: 24
attacks: 2999
- start: 109.248.9.0
end: 109.248.9.255
netmask: 24
attacks: 2790
- start: 196.52.43.0
end: 196.52.43.255
netmask: 24
attacks: 2667
- start: 5.188.203.0
end: 5.188.203.255
netmask: 24
attacks: 2592
- start: 5.188.10.0
end: 5.188.10.255
netmask: 24
attacks: 2575
- start: 216.158.238.0
end: 216.158.238.255
netmask: 24
attacks: 2382
- start: 80.82.70.0
end: 80.82.70.255
netmask: 24
attacks: 2046
- start: 185.35.62.0
end: 185.35.62.255
netmask: 24
attacks: 1933
- start: 85.93.20.0
end: 85.93.20.255
netmask: 24
attacks: 1664
- start: 141.212.122.0
end: 141.212.122.255
netmask: 24
attacks: 1473
- start: 93.174.93.0
end: 93.174.93.255
netmask: 24
attacks: 1433
- start: 80.82.77.0
end: 80.82.77.255
netmask: 24
attacks: 1411
- start: 104.236.178.0
end: 104.236.178.255
netmask: 24
attacks: 1156
- start: 180.97.106.0
end: 180.97.106.255
netmask: 24
attacks: 1104
- start: 5.188.11.0
end: 5.188.11.255
netmask: 24
attacks: 1088
So as you can see from the above we can easily iterate over this structure using Ansible to define the IP sets for our firewall.
Example DShield Block List IP Set
Below is an example of what our ipset COULD look like once we have iterated
over our block.yml structure to generate our ipset rules.
Name: dshield_block_list
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 1600
References: 2
Members:
5.188.10.0/24
5.188.86.0/24
5.188.203.0/24
85.93.20.0/24
46.29.162.0/24
80.82.70.0/24
185.35.62.0/24
93.174.93.0/24
141.212.122.0/24
216.158.238.0/24
181.214.87.0/24
109.248.9.0/24
191.101.167.0/24
104.236.178.0/24
196.52.43.0/24
5.188.11.0/24
77.72.82.0/24
77.72.85.0/24
180.97.106.0/24
80.82.77.0/24
Converting DShield Block List From TXT To YAML
So the above all looks great but how do we convert the native DShield block.txt
to block.yml? In order to do the conversion we will execute a few Ansible tasks
along with a Jinja2 template to do our conversion.
So follow along as we break down each Ansible task below.
Default Ansible Variables
Below you will find the default defined Ansible variables in use for the following tasks:
ipset_config_file: /etc/ipset/ipsets
ipset_dshield_block_list: "{{ lookup('file', ipset_dshield_block_file) }}"
ipset_dshield_block_file: /tmp/block.txt
ipset_dshield_block_file_yaml: /tmp/block.yml
ipset_enable_dshield_block_list: true
ipset_iptables_config_file: /etc/iptables/iptables
ipset_iptables_default_forward_policy: ACCEPT
ipset_iptables_default_input_policy: ACCEPT
ipset_iptables_default_output_policy: ACCEPT
ipset_iptables_dshield_chains:
- INPUT
- OUTPUT
Downloading DShield Block List
In this task we will be downloading the DShield Block list to our localhost. So
the following Ansible task will do just that for us:
- name: configure | Downloading DShield.org Block List
get_url:
url: http://feeds.dshield.org/block.txt
dest: "{{ ipset_dshield_block_file }}"
delegate_to: localhost
when: ipset_enable_dshield_block_list
Generating YAML DShield Block List
In this task we will be using a Jinja2 template to generate our block.yml file
which will also be performed on our localhost.
- name: configure | Generating DShield.org YAML Block List
template:
src: block.yml.j2
dest: "{{ ipset_dshield_block_file_yaml }}"
delegate_to: localhost
when: ipset_enable_dshield_block_list
As you can see from the above task we are using a template called block.yml.j2.
So of course as you are wondering, what does this template look like. So that is
what you will find below. And as you can see, it is rather quite simple.
block.yml.j2:
---
dshield_block_list:
{% for item in ipset_dshield_block_list.split('\n') %}
{% if '#' not in item %}
{% set list = item.split('\t') %}
{% if list[0]|ipaddr %}
- start: {{ list[0] }}
end: {{ list[1] }}
netmask: {{ list[2] }}
attacks: {{ list[3] }}
{% endif %}
{% endif %}
{% endfor %}
So based on the above template here is what is going on here. First off we are
iterating over the ipset_dshield_block_list which from our default Ansible variables this is defined as "{{ lookup('file', ipset_dshield_block_file) }}". So this is doing a file lookup using
the defined variable of ipset_dshield_block_file. And again, if we look at our
default Ansible variables this is defined as
/tmp/block.txt.
Now for the remaining bits of the template we are discarding any lines which
contain # because these are comments. Then we are splitting the lines which do
not contain # by tabs into our elements. And because our element [0] is an
IP address we are checking to ensure that is truly an IP address by if list[0]|ipaddr.
Then if that is true, we then define our variables which we are looking for. In
this case those would be start, end, netmask, and attacks.
Importing DShield Block List Variables
Now that our block.yml file has been generated. We can now import these variables
into Ansible to use for further consumption.
- name: configure | Importing dshield_block_list Variables
include_vars:
file: "{{ ipset_dshield_block_file_yaml }}"
when: ipset_enable_dshield_block_list
And once we do the above we now have a variable dshield_block_list which is
available for Ansible to iterate over. This variable is defined in our template
block.yml.j2 which we used to generate our block.yml file.
Generating IP Set Rules
Now that we have our DShield block list converted to YAML and imported as a
variable called dshield_block_list we are ready to generate our ipset rules.
In order to generate our ipset rules we will again use a Jinja2 template to
do this.
ipsets.j2:
flush
{% if ipset_enable_dshield_block_list %}
create dshield_block_list hash:net maxelem 65536
{% for _dshield_block_list in dshield_block_list %}
add dshield_block_list {{ _dshield_block_list['start'] }}-{{ _dshield_block_list['end'] }}
{% endfor %}
{% endif %}
As you can see in the above template we are using the variable dshield_block_list
to iterate over our DShield block list.
And using the following Ansible task we will use the above template to generate our ipsets rules.
- name: configure | Generating ipset Rules {{ ipset_config_file }}
template:
src: ipsets.j2
dest: "{{ ipset_config_file }}"
become: true
And the following is an example of what our ipsets rules COULD look like:
flush
create dshield_block_list hash:net maxelem 65536
add dshield_block_list 77.72.82.0-77.72.82.255
add dshield_block_list 191.101.167.0-191.101.167.255
add dshield_block_list 181.214.87.0-181.214.87.255
add dshield_block_list 5.188.86.0-5.188.86.255
add dshield_block_list 77.72.85.0-77.72.85.255
add dshield_block_list 46.29.162.0-46.29.162.255
add dshield_block_list 109.248.9.0-109.248.9.255
add dshield_block_list 196.52.43.0-196.52.43.255
add dshield_block_list 5.188.203.0-5.188.203.255
add dshield_block_list 5.188.10.0-5.188.10.255
add dshield_block_list 216.158.238.0-216.158.238.255
add dshield_block_list 80.82.70.0-80.82.70.255
add dshield_block_list 185.35.62.0-185.35.62.255
add dshield_block_list 85.93.20.0-85.93.20.255
add dshield_block_list 141.212.122.0-141.212.122.255
add dshield_block_list 93.174.93.0-93.174.93.255
add dshield_block_list 80.82.77.0-80.82.77.255
add dshield_block_list 104.236.178.0-104.236.178.255
add dshield_block_list 180.97.106.0-180.97.106.255
add dshield_block_list 5.188.11.0-5.188.11.255
Generating IPTables Rules
Now that we have generated our ipset rules. We are now ready to generate our
iptables rules which will use our ipset rules we have generated.
We will again be using a Jinja2 template to generate our iptables rules:
*filter
:INPUT {{ ipset_iptables_default_input_policy|upper }}
:FORWARD {{ ipset_iptables_default_forward_policy|upper }}
:OUTPUT {{ ipset_iptables_default_output_policy|upper }}
{% if ipset_enable_dshield_block_list and ipset_iptables_dshield_chains is defined %}
{% for _chain in ipset_iptables_dshield_chains %}
{% if _chain|upper == "INPUT" %}
-A {{ _chain|upper }} -m set --match-set dshield_block_list src -j DROP
{% elif _chain|upper == "FORWARD" %}
-A {{ _chain|upper }} -m set --match-set dshield_block_list src -j DROP
-A {{ _chain|upper }} -m set --match-set dshield_block_list dst -j DROP
{% elif _chain|upper == "OUTPUT" %}
-A {{ _chain|upper }} -m set --match-set dshield_block_list dst -j DROP
{% endif %}
{% endfor %}
{% endif %}
COMMIT
And using the following Ansible task we can generate our iptables rules using
the above Jinja2 template:
- name: configure | Generating IPTables Rules {{ ipset_iptables_config_file }}
template:
src: iptables.j2
dest: "{{ ipset_iptables_config_file }}"
become: true
register: _ipset_iptables_rules_generated
And the following is an example of what our iptables rules COULD look like:
*filter
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT
-A INPUT -m set --match-set dshield_block_list src -j DROP
-A OUTPUT -m set --match-set dshield_block_list dst -j DROP
COMMIT
Stitching It All Together
Now we are ready to stitch together our ipset rules and iptables rules.
Because our iptables rules rely on our ipset rules we must first load our
ipset rules. But first we SHOULD clear our existing iptables and ipset
rules.
Clearing IPTables Rules and IP Set Rules
We will use the following Ansible task to clear our iptables and ipset rules:
- name: configure | Clearing IPTables Rules and ipset Rules
shell: >
iptables -P INPUT ACCEPT &&
iptables -P FORWARD ACCEPT &&
iptables -P OUTPUT ACCEPT &&
iptables -F &&
iptables -X &&
ipset destroy
become: true
The above will do the following:
- Reset default INPUT policy to ACCEPT
- Reset default FORWARD policy to ACCEPT
- Reset default OUTPUT policy to ACCEPT
- Flush (delete) all iptables rules
- Delete all iptables user chains
- Destroy all ipset rules
Importing IP Set Rules
Now we are ready to import(restore) our generated ipset rules. And we will
accomplish this with the following Ansible task:
- name: configure | Restoring ipset Rules {{ ipset_config_file }}
shell: ipset restore -! < {{ ipset_config_file }}
become: true
Importing IPTables Rules
We are now finally ready to import(restore) our generated iptables rules. Which
we will accomplish by using the following Ansible task:
- name: configure | Restoring IPTables Rules {{ ipset_iptables_config_file }}
command: iptables-restore {{ ipset_iptables_config_file }}
become: true
Validating IP Set Rules and IPTables Rules
Now for the final step we can now validate that our ipset rules and iptables
rules are in place now.
vagrant@node0:~$ sudo ipset list dshield_block_list
Name: dshield_block_list
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 1600
References: 2
Members:
5.188.10.0/24
5.188.86.0/24
5.188.203.0/24
85.93.20.0/24
46.29.162.0/24
80.82.70.0/24
185.35.62.0/24
93.174.93.0/24
141.212.122.0/24
216.158.238.0/24
181.214.87.0/24
109.248.9.0/24
191.101.167.0/24
104.236.178.0/24
196.52.43.0/24
5.188.11.0/24
77.72.82.0/24
77.72.85.0/24
180.97.106.0/24
80.82.77.0/24
vagrant@node0:~$ sudo iptables -L -v -n
Chain INPUT (policy ACCEPT 163 packets, 107K bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 match-set dshield_block_list src
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 130 packets, 13305 bytes)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 match-set dshield_block_list dst
Persistence Across Reboots
And now for the final piece of this. How do we persist our ipset rules and
iptables rules across reboots. Ah, the million dollar question. Will all of
this still exist after a reboot. Well, of course, and we can do that by yet
again another Jinja2 template and an Ansible task.
NOTE: The below example is for
Debianbased systems ONLY.
Our Jinja2 template COULD look like below:
#! /usr/bin/env bash
{{ ansible_managed|comment }}
/sbin/ipset restore -! < {{ ipset_config_file }}
/sbin/iptables-restore {{ ipset_iptables_config_file }}
And our Ansible task would look like below:
- name: configure | Creating Firewall Load On Reboot Script
template:
src: etc/network/if-pre-up.d/firewall.j2
dest: /etc/network/if-pre-up.d/firewall
owner: root
group: root
mode: "u=rwx,g=rwx,o="
become: true
when: ansible_os_family == "Debian"
And there you have it, our ipset rules and iptables rules will persist across
reboots.
And this concludes this post on doing some fun stuff with our firewall!
Enjoy!
Leave a comment