2015년 7월 4일 토요일

Setting up fail2ban on CentOS 7 with firewalld on a Digital Ocean droplet

When running a server in the cloud, it is a good idea to at least secure SSH so that brute forcing attempts will automatically be blocked after a set number of incorrect tries. Having set up fail2ban with firewalld in Archlinux on a laptop, I figured the process would be pretty much the same for CentOS 7 in a Digital Ocean droplet.

As soon as you login into your new Droplet, I recommend that you follow the steps in this DO article about creating a regular user account, enabling sudo, disabling root logins over SSH, etc. You should also read the follow-up article about how to set up firewalld in CentOS 7 (TL;DR sudo systemctl enable firewalld, sudo systemctl start firewalld)

Now you need to install fail2ban. Although it is not in the default CentOS yum repositories, it is available through EPEL which is included in the CentOS Extras repository (which is enabled by default).

sudo yum install epel-release

Starting from RHEL/CentOS 7, iptables and rsyslog have been replaced with firewalld and journalctl (part of systemd), so you must be sure to yum install the following packages:

fail2ban-systemd (to ensure journalctl compatibility)
fail2ban-firewalld
ipset (to ban IP's with firewalld)
fail2ban-server

Now cd into /etc/fail2ban where you will see the systemwide fail2ban config file jail.conf which you can use as a reference but should not edit directly. Instead, in the subdir /etc/fail2ban/jail.d/ you should make your own config file ending with the .conf extension (I named mine local.conf). You will also notice that in this sub-directory there will be two other .conf files named 00-firewalld.conf and 00-systemd.conf which contain the following:

[centjun@juncent7 jail.d]$ cat 00-firewalld.conf 
# This file is part of the fail2ban-firewalld package to configure the use of
# the firewalld actions as the default actions.  You can remove this package
# (along with the empty fail2ban meta-package) if you do not use firewalld
[DEFAULT]
banaction = firewallcmd-ipset

[centjun@juncent7 jail.d]$ cat 00-systemd.conf 
# This file is part of the fail2ban-systemd package to configure the use of
# the systemd journal as the default backend.  You can remove this package
# (along with the empty fail2ban meta-package) if you do not want to use the
# journal backend
[DEFAULT]
backend=systemd

In Archlinux, these config files are not included in the jail.d subdir, so I had to add banaction=... and backend=... in local.conf (in the case of Archlinux, not CentOS7).

The /etc/fail2ban/jail.d/local.conf which I used in Archlinux is as follows:

[DEFAULT]
bantime = 18000
banaction = firewallcmd-ipset
backend = systemd
sender = fail2ban@example.com
destemail = root
ignoreip = 127.0.0.1 192.168.0.0/16
use_dns = no
maxretry = 21

action = %(action_mwl)s

[sshd]
enabled = true

However when I used the settings above in CentOS 7 and then tried to start the fail2ban systemd service with sudo systemctl start fail2ban I got the following errors in journalctl:

Jul 04 14:36:19 juncent7 fail2ban-client[1674]: ERROR  Found no accessible config files for 'action.d/sendmail-whois-lines' under /e...ail2ban
Jul 04 14:36:19 juncent7 fail2ban-client[1674]: ERROR  Error in action definition sendmail-whois-lines[name=sshd, dest="centjun", lo...INPUT"]
Jul 04 14:36:19 juncent7 fail2ban-client[1674]: ERROR  Errors in jail 'sshd'. Skipping...

Since fail2ban uses sendmail as its default mta, I checked to see if sendmail was installed:

[centjun@juncent7 jail.d]$ rpm -q sendmail
package sendmail is not installed

Instead, I found that postfix was installed, so I then added mta = postfix to local.conf, but still got similar errors:

Jul 04 14:45:13 juncent7 fail2ban-client[1715]: ERROR  Found no accessible config files for 'action.d/postfix-whois-lines' under /etc/fail2ban
Jul 04 14:45:13 juncent7 fail2ban-client[1715]: ERROR  Error in action definition postfix-whois-lines[name=sshd, dest="centjun", log...INPUT"]
Jul 04 14:45:13 juncent7 fail2ban-client[1715]: ERROR  Errors in jail 'sshd'. Skipping...

I checked /etc/fail2ban/action.d/ for the files postfix-whois-lines and sendmail-whois-lines but both of the them were nowhere to be found.

Finally I edited the default action in local.conf to just ban the offending IP instead of sending a mail to root:

action = $(action_)s

Now when starting fail2ban.service with systemctl start fail2ban, everything works fine:

[centjun@juncent7 jail.d]$ systemctl status fail2ban
fail2ban.service - Fail2Ban Service
   Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled)
   Active: active (running) since Sat 2015-07-04 15:08:04 SGT; 44min ago
     Docs: man:fail2ban(1)
  Process: 1803 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)
 Main PID: 1806 (fail2ban-server)
   CGroup: /system.slice/fail2ban.service
           └─1806 /usr/bin/python -Es /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b

[centjun@juncent7 jail.d]$ sudo fail2ban-client status
[sudo] password for centjun:
Status
|- Number of jail:      1
`- Jail list:   sshd

Just in case you think it is too much hassle to do minimal server hardening, take a look at just a few of the dozens of unauthorized login attempts recorded by PAM in journalctl:

Jul 04 15:21:42 juncent7 sshd[1828]: reverse mapping checking getaddrinfo for 107.30.65.218.broad.xy.jx.dynamic.163data.com.cn [218...ATTEMPT!
Jul 04 15:21:43 juncent7 sshd[1828]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.65...ser=root
Jul 04 15:21:43 juncent7 sshd[1828]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:21:45 juncent7 sshd[1828]: Failed password for root from 218.65.30.107 port 57804 ssh2
Jul 04 15:21:45 juncent7 sshd[1828]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:21:47 juncent7 sshd[1828]: Failed password for root from 218.65.30.107 port 57804 ssh2
Jul 04 15:21:49 juncent7 sshd[1828]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:21:51 juncent7 sshd[1828]: Failed password for root from 218.65.30.107 port 57804 ssh2
Jul 04 15:21:54 juncent7 sshd[1828]: Received disconnect from 218.65.30.107: 11:  [preauth]
Jul 04 15:21:54 juncent7 sshd[1828]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.65.30.107  user=root
Jul 04 15:21:55 juncent7 sshd[1830]: reverse mapping checking getaddrinfo for 107.30.65.218.broad.xy.jx.dynamic.163data.com.cn [218...ATTEMPT!
Jul 04 15:21:55 juncent7 sshd[1830]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.65...ser=root
Jul 04 15:21:55 juncent7 sshd[1830]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:21:58 juncent7 sshd[1830]: Failed password for root from 218.65.30.107 port 53038 ssh2
Jul 04 15:22:03 juncent7 sshd[1830]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:22:04 juncent7 sshd[1830]: Failed password for root from 218.65.30.107 port 53038 ssh2
Jul 04 15:22:05 juncent7 sshd[1830]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:22:06 juncent7 sshd[1830]: Failed password for root from 218.65.30.107 port 53038 ssh2
Jul 04 15:22:07 juncent7 sshd[1830]: Received disconnect from 218.65.30.107: 11:  [preauth]
...

When I switched to root user PAM informed me of a large number of failed login attempts on the root account:

[centjun@juncent7 jail.d]$ su -
Password:
Last login: Fri Jul  3 23:08:03 SGT 2015 from xxx.xxx.xxx.xxx (redacted) on pts/0
Last failed login: Sat Jul  4 15:22:42 SGT 2015 from 218.65.30.107 on ssh:notty
There were 65 failed login attempts since the last successful login.

This large number of unsuccessful logins occurred within just 1 hour of launching my new droplet!

When I ran a reverse DNS on one of the IP's associated with a failed login, it appeared to be an address from China.


Comments on certain settings in /etc/fail2ban/jail.d/local.conf

ignoreip = 

It is a good idea to put your remote IP (the IP of the computer you are connecting from) into ignoreip above (multiple IP's can be entered in the whitelist, separated with spaces) to avoid getting locked out of your cloud server by an attacker sending packets with a spoofed source header (of course the attacker would have to know your remote IP address).

use_dns = no

It is safer not to use dns to ban hostnames because an attacker can change the PTR of an IP that they control to point to another hostname that could make fail2ban to mistakenly ban valid domains (http://www.fail2ban.org/wiki/index.php/Hostnames_or_IP_Addresses).


Thoughts about Digital Ocean Droplets

I was pleasantly surprised with my first experience using Digital Ocean (DO) cloud instances. The only other cloud server I have used is AWS EC2 (free usage tier) and I think DO compares favorably. Although DO is missing the "enterprisey" features of AWS, I think DO is much easier to use and better-suited for quickly spinning up a server instance in the cloud. Whereas AWS micro instances only give you 8GB of storage and 1 CPU, DO's smallest instance gives you 1 CPU with 30 GB of SSD!

For me, DO's killer feature is the ability to take a snapshot of an instance (once you have shut it down), delete the Droplet, and then later create a new Droplet from the snapshot you created earlier. The reason this is so awesome is that DO snapshots are free (unlike those on AWS EC2, which take up gigabytes of Elastic Storage) and allow you to use cloud instances on an hourly basis if you are so inclined. Here are some good forum posts about using DO on an hourly basis:

https://www.digitalocean.com/community/questions/pricing-monthly-or-hourly

https://www.digitalocean.com/community/questions/restore-snapshot-after-destroying-a-droplet

On AWS EC2, however, snapshots are destroyed along with an image, so you couldn't easily replicate this DO feature over there.

References:

https://fedoraproject.org/wiki/Fail2ban_with_FirewallD
Fedora wiki about setting up fail2ban with firewalld; also explains the meaning of variables in fail2ban config files

https://wiki.gentoo.org/wiki/Fail2ban#Actions
Gentoo wiki about fail2ban