Opening Ports for Openstack in Firewalld

Last week I made a post about opening ports in Ubuntu's ufw firewall when using Devstack (Openstack upstream). Today I will show you how to do the same thing in firewalld dynamic firewall which is now the default in RHEL 7+ and Fedora.

The ports to be opened are the same, but you must also enable two additional services in firewalld, namely http and vnc-server. If you don't enable the former, you will be unable to access Horizon web UI, and if you don't enable the latter, you will not be able to see the console through Horizon when you launch an instance on Nova compute.

In the case of ufw, however, http port 80 was opened by default and vnc was enabled by simply opening 6080/tcp.

I wrote a Bash script to open the necessary ports for Openstack in firewalld. I have tested it on Openstack Kilo running on F23. You can find the script at the following link:

https://gitlab.com/gojun077/openstack-conf/blob/67f98aa4b93ab268e386028ec0e764547d0a1bb2/firewalld_openstack_rdo.sh

#!/bin/bash
# firewalld_openstack_rdo.sh
# Created by Jun Go gojun077@gmail.com
# Last Updated 2016-06-07

# Script that will permanently open ports needed by
# Redhat Distribution of Openstack (RDO) in Firewalld

# Tested with Openstack Kilo RDO 7

# This script should be run as root

# DEFAULT FIREWALLD ZONE
DZONE=FedoraServer

#################
#   NETWORK
#   IFACES
#################
EXT0=br-ex
INT0=br-enp5s0

#################
#   TCP PORTS
#################
AMQP=5672
CEILOM=8777
CINDER=8776
GLANCE=9292
GLANCEREG=9191
NEUTRON=9696
NOVNCPROX=6080
#NOVAEC2=8773
#NOVAMETA=8775
#NOVAISCSI=3260
#NOVAREDIS=6379
#NOVAS3=3333
SHEEPDOG=7000

#################
#   UDP PORTS
#################
CEILUDP=4952
OVSNEUTRONVXLAN=4789

TCPPORTS=($AMQP
   $CEILOM
   $CINDER
   $GLANCE
   $GLANCEREG
   $NEUTRON
   $NOVNCPROX
   $SHEEPDOG
  )

UDPPORTS=($CEILUDP
   $OVSNEUTRONVXLAN
   )

# ADD NETWORK IFACES TO DEFAULT ZONE
firewall-cmd --permanent --zone=$DZONE --add-interface=$EXT0
firewall-cmd --permanent --zone=$DZONE --add-interface=$INT0

# ENABLE SERVICES REQ'D FOR OPENSTACK
# Horizon (http)
firewall-cmd --permanent --zone=$DZONE --add-service=http
# vnc-server (for some reason, enabling TCP 6080 is not enough)
firewall-cmd --permanent --zone=$DZONE --add-service=vnc-server

for i in ${TCPPORTS[*]}; do
  firewall-cmd --permanent --zone=$DZONE --add-port="$i"/tcp
done

for j in ${UDPPORTS[*]}; do
  firewall-cmd --permanent --zone=$DZONE --add-port="$j"/udp
done

# Apply permanent rules as the current runtime config
firewall-cmd --reload

# List Default Zone Firewall Info (along with ports & svcs)
firewall-cmd --list-all

Internet connection sharing through a computer with two NIC's

Recently a new server room was built at the office and now all the network and server gear has been moved there. I used to use wireless connections on my notebooks out of convenience, but now that the wireless router is inside the server room instead of in the middle of an open floor plan office, the wireless signal is too weak.

My notebooks are connected to the internal network, however. Most of the test servers have dual NIC's so on a server with Fedora 23 installed, I first enabled port forwarding manually:

# check if port forwarding is enabled (it isn't)

[fedgro@fx8350no2 sysctl.d]$ sudo sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

# enable it

[fedgro@fx8350no2 sysctl.d]$ sudo sysctl net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

# verify that port forwarding has been enabled:

[fedgro@fx8350no2 sysctl.d]$ sudo sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

The change above will only apply until the next reboot, so to make the changes permanent, I want to edit sysctl.conf

In recent versions of Fedora using systemd, this conf file is no longer monolithic and is generated at boot by systemd from files in /usr/lib/sysctl.d/ (distribution default kernel settings) and in /etc/sysctl.d/ (custom kernel settings)

Here is the content of each directory:

[fedgro@fx8350no2 sysctl.d]$ cd /usr/lib/sysctl.d/
[fedgro@fx8350no2 sysctl.d]$ ls
00-system.conf              50-coredump.conf  60-libvirtd.conf
10-default-yama-scope.conf  50-default.conf
[fedgro@fx8350no2 sysctl.d]$ cd /etc/sysctl.d/
[fedgro@fx8350no2 sysctl.d]$ ls
50-libreswan.conf  99-sysctl.conf

Under /etc/sysctl.d I created a custom file 98-ipv4-forward.conf containing the single line

net.ipv4.ip_forward=1

Now that ipv4 port forwarding has been enabled, it is time to set up IP masquerading in the firewall. Fedora 23 uses firewalld instead of iptables, so to make the appropriate settings you can use the GUI firewall-config or the cli tool firewall-cmd.

You can see that firewalld is running:

[fedgro@fx8350no2 sysctl.d]$ systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2016-02-22 10:13:28 KST; 23h ago
 Main PID: 2132 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─2132 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid

Feb 22 10:13:28 fx8350no2 systemd[1]: Starting firewalld - dynamic firewal.....
Feb 22 10:13:28 fx8350no2 systemd[1]: Started firewalld - dynamic firewall...n.
Hint: Some lines were ellipsized, use -l to show in full.

Now I use firewall-cmd below to set up masquerading:

# Check the default firewall zone
# (FedoraServer is the default for F23)
[fedgro@fx8350no2 sysctl.d]$ sudo firewall-cmd --get-default
FedoraServer

Check the firewall zone for the NIC connected to a router providing access to the Internet:

[fedgro@fx8350no2 sysctl.d]$ sudo firewall-cmd --get-zone-of-interface=enp5s0
FedoraServer
 
Add IP masquerading to the default zone:

[fedgro@fx8350no2 sysctl.d]$ sudo firewall-cmd --zone=FedoraServer --add-masquerade
success

[fedgro@fx8350no2 sysctl.d]$ sudo firewall-cmd --permanent --zone=FedoraServer --add-masquerade

success

Verify changes have been made to FedoraServer zone:
[fedgro@fx8350no2 sysctl.d]$ firewall-cmd --zone=FedoraServer --list-all
FedoraServer (default, active)
  interfaces: enp5s0
  sources:
  services: cockpit dhcpv6-client http https ipp ipp-client samba-client sane ssh
  ports: 80/tcp 631/tcp
  protocols:
  masquerade: yes
  forward-ports:
  icmp-blocks:
  rich rules:

Now from my notebook connected to the internal network, I simply add the following route (to the server with dual NIC's):

sudo ip r add default via 192.168.95.145 dev enp1s0

where 192.168.95.145 is the IP of the NIC connected to the internal network on the dual-NIC server, and where enp1s0 is the ethernet port on my notebook.

And now the notebook connected to the internal network can talk to the outside world. Let's test on Google's DNS server 8.8.8.8:

[archjun@pinkS310 ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=51 time=36.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=51 time=35.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=51 time=36.1 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=51 time=36.6 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=51 time=35.9 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 35.342/36.157/36.718/0.496 ms
[archjun@pinkS310 ~]$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (192.168.95.145)  0.299 ms  0.278 ms  0.279 ms
 2  192.168.30.1 (192.168.30.1)  0.657 ms  0.695 ms  0.743 ms
 3  REDACTED (REDACTED)  11.346 ms  11.357 ms  11.349 ms
 4  REDACTED (REDACTED)  9.661 ms  13.730 ms  13.750 ms
 5  REDACTED (REDACTED)  9.541 ms  9.571 ms  9.566 ms
 6  112.189.28.97 (112.189.28.97)  8.167 ms  6.349 ms 112.189.29.97 (112.189.29.97)  15.201 ms
 7  112.174.103.225 (112.174.103.225)  10.270 ms 112.174.103.213 (112.174.103.213)  8.211 ms 112.174.119.205 (112.174.119.205)  6.721 ms
 8  112.174.48.162 (112.174.48.162)  6.676 ms  6.688 ms 112.174.8.42 (112.174.8.42)  6.898 ms
 9  112.174.83.218 (112.174.83.218)  6.854 ms 112.174.83.34 (112.174.83.34)  6.857 ms 112.174.84.122 (112.174.84.122)  6.957 ms
10  72.14.194.106 (72.14.194.106)  36.267 ms  36.246 ms  36.211 ms
11  216.239.54.13 (216.239.54.13)  36.528 ms 216.239.50.241 (216.239.50.241)  36.149 ms 216.239.54.13 (216.239.54.13)  41.606 ms
12  209.85.254.17 (209.85.254.17)  34.989 ms 209.85.246.89 (209.85.246.89)  37.053 ms 209.85.255.241 (209.85.255.241)  36.305 ms
13  google-public-dns-a.google.com (8.8.8.8)  36.593 ms  36.592 ms  42.129 ms

You can see that the first hop in the traceroute is to 192.168.95.145, the server on the internal network which has 2 NIC's. 1 NIC is connected to the internal network while one NIC is connected to the external network. It forwards the packets from the internal 192.168.95.0/24 network to the external network on subnet 192.168.30.0/24 which has access to the Internet through a router.

References:
https://wiki.archlinux.org/index.php/Internet_sharing
http://www.rudraraj.net/2014/12/21/internet-connection-sharing-ics-on-centos-7/

(the second link provides some useful commands for firewall-cmd)

Transferring files without ssh: netcat, darkhttpd, and Python

ssh is indispensable when working on remote machines, but to my surprise (and frustration) many of the big telecoms in Korea have started disabling sshd on most machines due to security audit recommendations. This is ridiculous when you consider that the audits don't flag the rampant use of telnet (which sends all traffic in cleartext) for managing machines on the internal network. At some sites that disable sshd, sysadmins are using old-fashioned ftp in place of sftp or vsftpd!

To do my work, whether it's applying patches or setting up Apache, at a minimum I need to be able to transfer files between machines. When you aren't given access to ssh (which also means no scp or vsftpd), oftentimes netcat (nc), darkhttpd or Python's built-in webservers will do nicely for file transfers.

Netcat

Lots of old-school sysadmins are familiar with using netcat to transfer files between machines, and there are many tutorials on the Internet. Here is an example of using netcat (both GNU netcat/nc and BSD nc will work with each other) to transfer a file to a machine running firewalld dynamic firewall.

By default, firewalld will keep all ports closed except those necessary for web browsing (port 80 http or 443 https) and certain user-defined services like nfs, ssh, etc. The remote machine in this example is on my local network and has sshd and rpcbind (for NFS) running. firewalld will ignore a regular ping scan from nmap, but if we run nmap -Pn hostname, we can see a list of open ports (-Pn Treat all hosts as online -- skip host discovery).

[archjun@latitude630 playground]$ nmap -Pn 192.168.10.57

Starting Nmap 6.47 ( http://nmap.org ) at 2015-10-16 23:07 KST
Nmap scan report for 192.168.10.57
Host is up (0.85s latency).
Not shown: 996 filtered ports
PORT     STATE  SERVICE
22/tcp   open   ssh
111/tcp  open   rpcbind
873/tcp  closed rsync
2049/tcp open   nfs

Before transferring files to a remote machine using netcat, first I need to temporarily open a port for netcat to use. Let's use tcp port 4444:

[archjun@d257 playground]$ sudo firewall-cmd --zone=internal --add-port=4444/tcp
[sudo] password for archjun: 
success

btw, firewalld has the concept of zones which have different security policies. Network interfaces can be placed The internal zone applies to the local network only. The available zones are:

[archjun@d257 playground]$ firewall-cmd --get-zones
block dmz drop external home internal public trusted work

Running nmap from latitude630  on the remote machine d257 now shows tcp port 4444:

[archjun@latitude630 playground]$ nmap -Pn 192.168.10.57

Starting Nmap 6.47 ( http://nmap.org ) at 2015-10-16 23:21 KST
Nmap scan report for 192.168.10.57
Host is up (0.61s latency).
Not shown: 994 filtered ports
PORT     STATE  SERVICE
22/tcp   open   ssh
111/tcp  open   rpcbind
873/tcp  closed rsync
2049/tcp open   nfs
4444/tcp closed krb524

Now from the remote machine d257 I will start BSD nc and tell it to listen on tcp port 4444 and to redirect all traffic to the file rcv_nc_test.txt:

[archjun@d257 playground]$ nc -l 4444 > rcv_nc_test.txt

From the sending machine, I will start GNU netcat/nc and tell it to send the file test_new_vimrc which contains the following text:

abc
#aabcde
hopefully no more temp files generated in editing path..
One more try... hopefully no more .un~ files will be generated
in the edited file's PATH

[archjun@latitude630 playground]$ nc 192.168.10.57 4444 < test_new_vimrc

nc on both the receiver and sender will not give any indication that the transfer is complete, so on the receiving end, I sent Ctrl-C to terminate the netcat session. Let's see if the content of the text file from latitude630 was sent to rcv_nc_test.txt on d257:

[archjun@d257 playground]$ ls
1  3    foo-replace      orig1      orig2  play-w-bash-functions.sh  test_sed_replace.txt
2  foo  netcat_file.svg  orig1.old  orig3  rcv_nc_test.txt

[archjun@d257 playground]$ cat rcv_nc_test.txt 
abc
#aabcde
hopefully no more temp files generated in editing path..
One more try... hopefully no more .un~ files will be generated
in the edited file's PATH

The content of test_new_vimrc from latitude630 was successfully redirected to rcv_nc_test.txt on d257! nc can also transfer binary files just fine.


Built-in Webservers in Python 2 and Python 3

Nowadays on most Linux installations (RHEL/CentOS, Ubuntu) I work on in the field, python 2 is installed by default. Python 2 comes with its own webserver module called SimpleHTTPServer. When it is invoked from the command line, it will by default serve up the current directory over http on port 8000. Since the remote machine is running firewalld, I first have to open tcp port 8000:

[archjun@d257 bin]$ sudo firewall-cmd --zone=internal --add-port=8000/tcp
[sudo] password for archjun: 
success

[archjun@d257 playground]$ python2 -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
192.168.10.63 - - [16/Oct/2015 23:18:36] "GET / HTTP/1.1" 200 -
192.168.10.63 - - [16/Oct/2015 23:18:36] "GET /favicon.ico HTTP/1.1" 404 -
192.168.10.63 - - [16/Oct/2015 23:18:41] "GET /foo-replace HTTP/1.1" 200 -

Note that in Archlinux python 2 must be invoked with python2. Since 2014, python 3 has been the default python in Arch. In Python 3, invoking the built-in webserver is a bit different. The module name is http.server:

[archjun@d257 playground]$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 ...
192.168.10.63 - - [16/Oct/2015 23:31:54] "GET /test_sed_replace.txt HTTP/1.1" 200 -
192.168.10.63 - - [16/Oct/2015 23:31:58] "GET /play-w-bash-functions.sh HTTP/1.1" 200 -

Now if I navigate to 192.168.10.57:8000 from another machine on the local network, I get the following HTML page listing the contents of ~/playground:

Much easier than configuring apache/httpd, isn't it? Although the page reads, "Directory listing for /" it is actually serving up ~/playground from host d257. Output is the same for SimpleHTTPServer and http.server. If you want to change the port number, simply add the port you wish to use after the module name, i.e. python -m http.server 8080. Note that if you want to use a port below 1024, you must invoke the webserver as root (but be aware of the security risks).


Darkhttpd

I am a big fan of darkhttpd. I normally use it in a PXE server setup with tftpboot and dnsmasq to serve up files over http as I detailed in this post. While python's built-in webserver is fine for serving up a few files in a pinch, darkhttpd will handle tens of GB of transfers without any hiccups, as I can attest to when installing Linux on multiple machines from a PXE server sending files over 1 gigabit Ethernet.

By default, darkhttpd will share the specified directory on tcp port 8080, but you can specify a different port with the --port option:

[archjun@d257 playground]$ darkhttpd . --port 8000
darkhttpd/1.11, copyright (c) 2003-2015 Emil Mikulic.
listening on: http://0.0.0.0:8000/
1445006481 192.168.10.63 "GET /" 200 1024 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
1445006504 192.168.10.63 "GET /netcat_file.svg" 200 1463 "http://192.168.10.57:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"

Navigating to 192.168.10.57:8000 from another machine shows the following HTML page:

btw if you launch darkhttpd as root, it will share the specified directory on tcp port 80.


Conclusion

If you ever find yourself on a locked-down machine without ssh, give netcat, python webservers and darkhttpd a try. In the examples above, opening ports using firewalld is quite easy compared to editing and reloading static iptables rules. I am so glad that RHEL 7.x uses firewalld by default!

Unable to reload rules in firewalld 0.3.14.2-1 Archlinux (workaround: restart firewalld.service)

firewalld is an upstream dynamic firewall project from Fedora. Personally, I find working with firewalld to be much more pleasant than trying to decipher and write arcane iptables static firewall rules.

Although Archlinux is generally a cutting-edge Linux distro that gives users access to the newest packages from upstream and the newest kernels, in my humble opinion it lags behind Fedora/RHEL/CentOS in the security department. SELinux and firewalld are setup by default in Fedora et al, but in Archlinux only firewalld is available from the default repositories. As of June 2015, SELinux can be installed on Arch, but SELinux policies have not been customized for Archlinux (i.e. the default policies assume Fedora/RHEL paths and filenames).

A test machine running Arch and firewalld also happened to be an NFS server, but NFS clients could no longer connect to the server because of the firewall. Since I only use NFS within my LAN, I assigned the Ethernet port enp1s0 to the firewalld internal zone and then added NFS-related services to internal as follows:

$ sudo firewall-cmd --zone=internal --change-interface=enp1s0
[sudo] password for archjun: 
success
$ sudo firewall-cmd --permanent --zone=internal --add-service=nfs
success
$ sudo firewall-cmd --permanent --zone=internal --add-service=rpc-bind
success
$ sudo firewall-cmd --permanent --zone=internal --add-service=mountd

Now to apply this to firewalld, we need to reload the firewall rules:

$ sudo firewall-cmd --reload
Error: 'NoneType' object has no attribute 'query_rule'

I verified that the services I added above (nfs, rpc-bind, and mountd) were not yet reflected in firewalld:

$ firewall-cmd --zone=internal --list-all
internal (active)
  interfaces: enp1s0
  sources: 
  services: dhcpv6-client mdns samba-client ssh
  ports: 
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules:

I looked for any firewalld-related errors in the systemd journal journalctl, but found nothing of interest.

Googling for the error above did not turn up any relevant results. I also tried to use the firewalld GUI, firewall-config, and clicked the checkbox for each service to whitelist in Zone internal :

Changing Runtime rules only works for the current session, but you can also define Permanent rules, too. I assumed that changing Runtime rules in the GUI would be immediately applied, but this was not the case; I am not sure if this is a problem with firewall-config in Archlinux or a lack of understanding on my part. When I clicked Options -> Reload Firewalld, I got the exact same error that firewalld-cmd gave me:

'NoneType' object has no attribute 'query_rule'

As a last resort, I invoked

$ sudo systemctl restart firewalld

After restarting systemd's firewalld.service, you can see that the permanent rule changes I made to the internal zone with firewall-cmd finally appear:

$ firewall-cmd --zone=internal --list-all
internal (active)
  interfaces: enp1s0
  sources: 
  services: dhcpv6-client mdns mountd nfs rpc-bind rsyncd samba-client ssh
  ports: 
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules:

*Note: although firewalld calls "rpcbind" rpc-bind, the name of the systemd service is rpcbind.service, not rpc-bind.service.


Update 2015-07-10

On another Archlinux machine using the Openbox Desktop Environment, reloading firewalld rules with firewall-cmd --reload works just fine. The problem I have described in this post occurred on an Archlinux machine using the LXDE desktop environment. Perhaps the problems I experienced are somehow related to LXDE?

References:

https://fedoraproject.org/wiki/User:Renich/HowTo/NFSv4 (with firewalld)

https://fedoraproject.org/wiki/FirewallD#Using_firewall-cmd