Setting up fail2ban on CentOS 7 with firewalld on a Digital Ocean droplet

When running a server in the cloud, it is a good idea to at least secure SSH so that brute forcing attempts will automatically be blocked after a set number of incorrect tries. Having set up fail2ban with firewalld in Archlinux on a laptop, I figured the process would be pretty much the same for CentOS 7 in a Digital Ocean droplet.

As soon as you login into your new Droplet, I recommend that you follow the steps in this DO article about creating a regular user account, enabling sudo, disabling root logins over SSH, etc. You should also read the follow-up article about how to set up firewalld in CentOS 7 (TL;DR sudo systemctl enable firewalld, sudo systemctl start firewalld)

Now you need to install fail2ban. Although it is not in the default CentOS yum repositories, it is available through EPEL which is included in the CentOS Extras repository (which is enabled by default).

sudo yum install epel-release

Starting from RHEL/CentOS 7, iptables and rsyslog have been replaced with firewalld and journalctl (part of systemd), so you must be sure to yum install the following packages:

fail2ban-systemd (to ensure journalctl compatibility)
fail2ban-firewalld
ipset (to ban IP's with firewalld)
fail2ban-server

Now cd into /etc/fail2ban where you will see the systemwide fail2ban config file jail.conf which you can use as a reference but should not edit directly. Instead, in the subdir /etc/fail2ban/jail.d/ you should make your own config file ending with the .conf extension (I named mine local.conf). You will also notice that in this sub-directory there will be two other .conf files named 00-firewalld.conf and 00-systemd.conf which contain the following:

[centjun@juncent7 jail.d]$ cat 00-firewalld.conf 
# This file is part of the fail2ban-firewalld package to configure the use of
# the firewalld actions as the default actions.  You can remove this package
# (along with the empty fail2ban meta-package) if you do not use firewalld
[DEFAULT]
banaction = firewallcmd-ipset

[centjun@juncent7 jail.d]$ cat 00-systemd.conf 
# This file is part of the fail2ban-systemd package to configure the use of
# the systemd journal as the default backend.  You can remove this package
# (along with the empty fail2ban meta-package) if you do not want to use the
# journal backend
[DEFAULT]
backend=systemd

In Archlinux, these config files are not included in the jail.d subdir, so I had to add banaction=... and backend=... in local.conf (in the case of Archlinux, not CentOS7).

The /etc/fail2ban/jail.d/local.conf which I used in Archlinux is as follows:

[DEFAULT]
bantime = 18000
banaction = firewallcmd-ipset
backend = systemd
sender = fail2ban@example.com
destemail = root
ignoreip = 127.0.0.1 192.168.0.0/16
use_dns = no
maxretry = 21

action = %(action_mwl)s

[sshd]
enabled = true

However when I used the settings above in CentOS 7 and then tried to start the fail2ban systemd service with sudo systemctl start fail2ban I got the following errors in journalctl:

Jul 04 14:36:19 juncent7 fail2ban-client[1674]: ERROR  Found no accessible config files for 'action.d/sendmail-whois-lines' under /e...ail2ban
Jul 04 14:36:19 juncent7 fail2ban-client[1674]: ERROR  Error in action definition sendmail-whois-lines[name=sshd, dest="centjun", lo...INPUT"]
Jul 04 14:36:19 juncent7 fail2ban-client[1674]: ERROR  Errors in jail 'sshd'. Skipping...

Since fail2ban uses sendmail as its default mta, I checked to see if sendmail was installed:

[centjun@juncent7 jail.d]$ rpm -q sendmail
package sendmail is not installed

Instead, I found that postfix was installed, so I then added mta = postfix to local.conf, but still got similar errors:

Jul 04 14:45:13 juncent7 fail2ban-client[1715]: ERROR  Found no accessible config files for 'action.d/postfix-whois-lines' under /etc/fail2ban
Jul 04 14:45:13 juncent7 fail2ban-client[1715]: ERROR  Error in action definition postfix-whois-lines[name=sshd, dest="centjun", log...INPUT"]
Jul 04 14:45:13 juncent7 fail2ban-client[1715]: ERROR  Errors in jail 'sshd'. Skipping...

I checked /etc/fail2ban/action.d/ for the files postfix-whois-lines and sendmail-whois-lines but both of the them were nowhere to be found.

Finally I edited the default action in local.conf to just ban the offending IP instead of sending a mail to root:

action = $(action_)s

Now when starting fail2ban.service with systemctl start fail2ban, everything works fine:

[centjun@juncent7 jail.d]$ systemctl status fail2ban
fail2ban.service - Fail2Ban Service
   Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled)
   Active: active (running) since Sat 2015-07-04 15:08:04 SGT; 44min ago
     Docs: man:fail2ban(1)
  Process: 1803 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)
 Main PID: 1806 (fail2ban-server)
   CGroup: /system.slice/fail2ban.service
           └─1806 /usr/bin/python -Es /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b

[centjun@juncent7 jail.d]$ sudo fail2ban-client status
[sudo] password for centjun:
Status
|- Number of jail:      1
`- Jail list:   sshd

Just in case you think it is too much hassle to do minimal server hardening, take a look at just a few of the dozens of unauthorized login attempts recorded by PAM in journalctl:

Jul 04 15:21:42 juncent7 sshd[1828]: reverse mapping checking getaddrinfo for 107.30.65.218.broad.xy.jx.dynamic.163data.com.cn [218...ATTEMPT!
Jul 04 15:21:43 juncent7 sshd[1828]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.65...ser=root
Jul 04 15:21:43 juncent7 sshd[1828]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:21:45 juncent7 sshd[1828]: Failed password for root from 218.65.30.107 port 57804 ssh2
Jul 04 15:21:45 juncent7 sshd[1828]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:21:47 juncent7 sshd[1828]: Failed password for root from 218.65.30.107 port 57804 ssh2
Jul 04 15:21:49 juncent7 sshd[1828]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:21:51 juncent7 sshd[1828]: Failed password for root from 218.65.30.107 port 57804 ssh2
Jul 04 15:21:54 juncent7 sshd[1828]: Received disconnect from 218.65.30.107: 11:  [preauth]
Jul 04 15:21:54 juncent7 sshd[1828]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.65.30.107  user=root
Jul 04 15:21:55 juncent7 sshd[1830]: reverse mapping checking getaddrinfo for 107.30.65.218.broad.xy.jx.dynamic.163data.com.cn [218...ATTEMPT!
Jul 04 15:21:55 juncent7 sshd[1830]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=218.65...ser=root
Jul 04 15:21:55 juncent7 sshd[1830]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:21:58 juncent7 sshd[1830]: Failed password for root from 218.65.30.107 port 53038 ssh2
Jul 04 15:22:03 juncent7 sshd[1830]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:22:04 juncent7 sshd[1830]: Failed password for root from 218.65.30.107 port 53038 ssh2
Jul 04 15:22:05 juncent7 sshd[1830]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Jul 04 15:22:06 juncent7 sshd[1830]: Failed password for root from 218.65.30.107 port 53038 ssh2
Jul 04 15:22:07 juncent7 sshd[1830]: Received disconnect from 218.65.30.107: 11:  [preauth]
...

When I switched to root user PAM informed me of a large number of failed login attempts on the root account:

[centjun@juncent7 jail.d]$ su -
Password:
Last login: Fri Jul  3 23:08:03 SGT 2015 from xxx.xxx.xxx.xxx (redacted) on pts/0
Last failed login: Sat Jul  4 15:22:42 SGT 2015 from 218.65.30.107 on ssh:notty
There were 65 failed login attempts since the last successful login.

This large number of unsuccessful logins occurred within just 1 hour of launching my new droplet!

When I ran a reverse DNS on one of the IP's associated with a failed login, it appeared to be an address from China.


Comments on certain settings in /etc/fail2ban/jail.d/local.conf

ignoreip = 

It is a good idea to put your remote IP (the IP of the computer you are connecting from) into ignoreip above (multiple IP's can be entered in the whitelist, separated with spaces) to avoid getting locked out of your cloud server by an attacker sending packets with a spoofed source header (of course the attacker would have to know your remote IP address).

use_dns = no

It is safer not to use dns to ban hostnames because an attacker can change the PTR of an IP that they control to point to another hostname that could make fail2ban to mistakenly ban valid domains (http://www.fail2ban.org/wiki/index.php/Hostnames_or_IP_Addresses).


Thoughts about Digital Ocean Droplets

I was pleasantly surprised with my first experience using Digital Ocean (DO) cloud instances. The only other cloud server I have used is AWS EC2 (free usage tier) and I think DO compares favorably. Although DO is missing the "enterprisey" features of AWS, I think DO is much easier to use and better-suited for quickly spinning up a server instance in the cloud. Whereas AWS micro instances only give you 8GB of storage and 1 CPU, DO's smallest instance gives you 1 CPU with 30 GB of SSD!

For me, DO's killer feature is the ability to take a snapshot of an instance (once you have shut it down), delete the Droplet, and then later create a new Droplet from the snapshot you created earlier. The reason this is so awesome is that DO snapshots are free (unlike those on AWS EC2, which take up gigabytes of Elastic Storage) and allow you to use cloud instances on an hourly basis if you are so inclined. Here are some good forum posts about using DO on an hourly basis:

https://www.digitalocean.com/community/questions/pricing-monthly-or-hourly

https://www.digitalocean.com/community/questions/restore-snapshot-after-destroying-a-droplet

On AWS EC2, however, snapshots are destroyed along with an image, so you couldn't easily replicate this DO feature over there.

References:

https://fedoraproject.org/wiki/Fail2ban_with_FirewallD
Fedora wiki about setting up fail2ban with firewalld; also explains the meaning of variables in fail2ban config files

https://wiki.gentoo.org/wiki/Fail2ban#Actions
Gentoo wiki about fail2ban

Problem when using GNU Screen 'stuff' command with 40+ open tabs - runaway input to stdin

Once a month I conduct system diagnostics on about 40 machines at an IDC for a Korean food company. Because the commands I need to run (i.e. dmesg | grep -iE "criti|fail|warn|bug|") are the same for all  machines, I use a script to launch GNU Screen with dozens of tabs each ssh'ing into a different server and once connected to all machines, I use

C-a :at "#" stuff "cmd^M"

to send commands to all tabs at once. Without this feature, there is no way I could complete a system checkup on 40+ machines in less than a few hours.

I recently had to install the leap second patch for tzdata packages on 40+ RHEL6.X machines, and wanted to scp them from my local machine to all the servers in one fell swoop. Therefore I entered the following command into GNU Screen:

:at "#" stuff "scp -r archjun@10.200.250.156:/MULTIMEDIA/iso/pkg-update/tzdata^M" .

As this was a new ssh connection, I was prompted whether I wanted to continue connecting:

The authenticity of host '10.200.250.156 (10.200.250.156)' can't be established.
RSA key fingerprint is
...
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.200.250.156' (RSA) to the list of known hosts.
archjun@10.200.250.156's password:

Obviously I wasn't about to type 'yes' into each of 40+ tabs. Therefore I stuffed yes into all 40+ GNU Screen tabs using

:at "#" stuff "yes^M"

All was fine for about 20 tabs, but the remaining tabs entered an input loop stuffing an infinite number of y's to stdin

y
y
y
y
y
y
y
...

In my .screenrc config file I have automatic logging turned on with the two lines

logfile $HOME/Documents/term_sessions/%Y-%m-%d_%0c-%n.log
deflog on

so that a logfile is automatically generated as soon as a new GNU Screen tab is created (hotkey C-a c). I noticed that logfiles were suddenly ballooning to hundreds of megabytes in size.

I manually had to enter C-c into each of these tabs to stop the runaway input. I have never experienced anything like this when stuffing commands to all windows in GNU Screen. Has anyone else encountered similar behavior?

Setting up mcelog to work with systemd

If you regularly observe system logs such as /var/log/messages, dmesg, or journalctl (systemd) you will eventually encounter a Machine Check Event (mce) which warns you that some kind of hardware error has occurred. For example, a common mce is caused when an incorrect bit is flipped in RAM. For server ECC memory, this is less of a problem because such bit errors can be fixed automatically. When encountering servers in the field with uptime greater than 365 days, it is not hard to find mce errors logged here and there. In the case of RHEL 5/6 machines I encounter in the field, mce errors are logged in the file /var/log/mcelog and the mcelog service runs by default.

Recently I noticed that every few days the kernel ring buffer dmesg on my work laptop gives the following error:

[Jun11 23:49] mce: [Hardware Error]: Machine check events logged

However, when I navigate to /var/log/ I cannot see any file named mcelog. Some old posts floating around the Internet recommend redirecting mcelog to some output file, i.e. /usr/sbin/mcelog > mcelog.out but this didn't work for me. Make sure you have the mcelog package (as it is called in Arch) installed . To enable the daemon in systemd, systemctl enable mcelog. When running mcelog on a Linux machine running systemd instead of the old syslog, you need to make some changes to /etc/mcelog/mcelog.conf

What led me astray was the Archwiki page on MCE Handling, which recommends uncommenting the line

syslog = yes

If  you are running systemd you do NOT want the above setting! The problem is that systemd handles system logging through journalctl. You can follow the other suggestions in the Archwiki to run mcelog as a daemon (daemon = yes), but make sure the syslog lines are commented out. Also you need to specify an output log file for mce errors by uncommenting the following in /etc/mcelog/mcelog.conf:

logfile = /var/log/mcelog

Also uncomment the following in /etc/mcelog/mcelog.conf

run-credentials-user = root

Restart the mcelog service

systemctl restart mcelog

Next time a Machine Check Event occurs, it will be written to /var/log/mcelog. Here is some sample output:

Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 5
MISC b8a0000086 ADDR ffb07500
TIME 1434034181 Thu Jun 11 23:49:41 2015
MCG status:
MCi status:
Error overflow
Uncorrected error
MCi_MISC register valid
MCi_ADDR register valid
Processor context corrupt
MCA: corrected filtering (some unreported errors in same region)
Generic CACHE Level-2 Generic Error
STATUS ee0000000040110a MCGSTATUS 0
MCGCAP c07 APICID 0 SOCKETID 0
CPUID Vendor Intel Family 6 Model 69
Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 5
MISC 78a0000086 ADDR ffb07500

This seems to be indicating a memory error in the CPU cache.

Here is my /etc/mcelog/mcelog.conf file:

[SOLVED] Upgrade from Python 2.7.9 to 2.7.10 breaks Brainworkshop

Note: I originally made this post in June 2015, but in Dec. 2015 I found a workaround to get Brainworkshop 4.8.4/4.8.7 to work with Python 2.7.10+. Please refer to the new post at the following link:
http://eatpeppershothot.blogspot.kr/2015/12/enabling-brainworkshop-487-to-work-with.html

=================================================================
Recently Archlinux made python2-2.7.10-1 available in the main Arch repositories. Unfortunately, however, the Brainworkshop N-back training program doesn't work with the latest version of Python 2.

In Gentoo and Arch, the brainworkshop.pyw source file is launched by the startup script /usr/bin/brainworkshop which looks like the following (I have edited it to correspond to my BW data and config files on Dropbox):



After upgrading from Python 2.7.9 to 2.7.10, however, the following error message is generated:

/usr/bin/brainworkshop: line 8:  5212 Segmentation fault      (core dumped) python2 /usr/share/brainworkshop/brainworkshop.pyw --configfile $BRW_CONFIGFILE --statsfile $BRW_STATFILE --datadir $BRW_DATADIR

The actual Python 2 source file is located at /usr/share/brainworkshop/brainworkshop.pyw (.pyw instead of .py so that a separate console window won't be launched). Even launching brainworkshop.pyw directly without any option flags returns Segmentation fault (core dumped). In Archlinux, core dump files are located in /var/lib/systemd/coredump, but rather than trying to analyze a dump of system memory at the time brainworkshop.pyw crashed, I decided to run gdb to see if I could get more informative error messages through a stack trace. Although gdb is normally run on C/C++ compiled executables, gdb (version 7+) will also work with Python if you invoke it as follows:

[archjun@lenovoS310 ~]$ gdb /usr/bin/python2
GNU gdb (GDB) 7.9.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python2...(no debugging symbols found)...done.

Now at the gdb prompt, type run followed by the name of the python program:

(gdb) run /usr/share/brainworkshop/brainworkshop.pyw
Starting program: /usr/bin/python2 /usr/share/brainworkshop/brainworkshop.pyw
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Traceback (most recent call last):
  File "/usr/share/brainworkshop/brainworkshop.pyw", line 1023, in
    window = MyWindow(cfg.WINDOW_WIDTH, cfg.WINDOW_HEIGHT, caption=''.join(caption), style=style, vsync=VSYNC)
  File "/usr/lib/python2.7/site-packages/pyglet/window/xlib/__init__.py", line 163, in __init__
    super(XlibWindow, self).__init__(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pyglet/window/__init__.py", line 494, in __init__
    display = get_platform().get_default_display()
  File "/usr/lib/python2.7/site-packages/pyglet/window/__init__.py", line 1766, in get_default_display
    return pyglet.canvas.get_display()
  File "/usr/lib/python2.7/site-packages/pyglet/canvas/__init__.py", line 82, in get_display
    return Display()
  File "/usr/lib/python2.7/site-packages/pyglet/canvas/xlib.py", line 83, in __init__
    raise NoSuchDisplayException('Cannot connect to "%s"' % name)
pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to "None"
[Inferior 1 (process 26382) exited with code 01]

So it seems like Python's pyglet module is the culprit once again in breaking Brainworkshop. In a previous post from February 2015 (pyglet 1.2 breaks Brainworkshop), I described a way to get Brainworkshop working again by making a few edits to the /usr/share/brainworkshop/brainworkshop.pyw source file. I have played around with changing some of the parameters in line 1024

window = MyWindow(cfg.WINDOW_WIDTH, cfg.WINDOW_HEIGHT, caption=''.join(caption), style=style, vsync=VSYNC)

but I have yet to succeed in getting Brainworkshop to work with Python 2.7.10.

Workaround

Luckily, BW still works with Python 2.7.9, so I downgraded to 2.7.9 with pacman -U /var/cache/pacman/pkg/python2-2.7.9-1-x86_64.pkg.tar.xz (if you don't have this package in your pacman package cache, you can still download it from the Arch Linux Archive) and added python2 to IgnorePkg in /etc/pacman.conf as a temporary workaround.

Update 2015-09-07

python2-pyglet in Archlinux was recently upgraded from 1.2.3-1 to 1.2.4-1. Now when trying to run Brainworkshop with Python 2.7.10 I get the following error in brainworkshop.pyw when debugging with gdb:

[archjun@latitude630 ~]$ gdb /usr/bin/python2
GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python2...(no debugging symbols found)...done.
(gdb) run /usr/share/brainworkshop/brainworkshop.pyw
Starting program: /usr/bin/python2 /usr/share/brainworkshop/brainworkshop.pyw
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7fffeaef1700 (LWP 8948)]
[Thread 0x7fffeaef1700 (LWP 8948) exited]
[New Thread 0x7fffeaef1700 (LWP 8949)]
[New Thread 0x7fffedd79700 (LWP 8950)]

Program received signal SIGSEGV, Segmentation fault.
0x00007fffde270664 in gdk_pixbuf_loader_write () from /usr/lib/libgdk_pixbuf-2.0.so.0

It looks like we are no longer getting errors from pyglet like we did above with pyglet 1.2.3, but now we are having problems with libgdk_pixbuf-2.0.so.0. Let's take a look at this file:

[archjun@latitude630 ~]$ ls -al /usr/lib/libgdk_pixbuf-2.0.so.0
lrwxrwxrwx 1 root root 29 Aug 20 20:32 /usr/lib/libgdk_pixbuf-2.0.so.0 -> libgdk_pixbuf-2.0.so.0.3100.6

Apparently libgdk_pixbuf-2.0.so.0 is a symlink from the file /usr/lib/libgdk_pixbuf-2.0.so.0.3100.6

What package owns libgdk_pixbuf-2.0.so.0.3100.6?

[archjun@latitude630 lib]$ sudo pacman -Qo libgdk_pixbuf-2.0.so.0.3100.6
[sudo] password for archjun: 
/usr/lib/libgdk_pixbuf-2.0.so.0.3100.6 is owned by gdk-pixbuf2 2.31.6-1

[archjun@latitude630 lib]$ sudo pacman -Ss gdk-pixbuf2
extra/gdk-pixbuf2 2.31.6-1 [installed]
    An image loading library

OK~ so gdk-pixbuf2 loads images.

According to gdb, the function that is causing trouble is called gdk_pixbuf_loader_write (). Looking at the Gnome GDK-PixBuf Reference Manual, we can find more detailed information on the C function gdk_pixbuf_loader_write (), but I'm not familiar with C so I am not sure where to start debugging this issue.

A google search for gdk-pixbuf2 gdk_pixbuf_loader_write () segfault returns an interesting (but dated) google code issues page on github that discusses problems with pyglet decoding images using gdk-pixbuf2. One proposed workaround is to try PIL, pyllow or pypng for image decoding instead of gdk-pixbuf2.

Opening /usr/share/brainworkshop/brainworkshop.pyw with a text editor such as vim or emacs quickly reveals that there are no explicit calls to gdk functions in the Python code. We know that pyglet can use gdk-pixbuf2 when it tries to decode images, so I searched for invocations of pyglet.image and found 6 results at lines 1031, 2304, 2327, 4622, 4626, and 4628, respectively:

if sys.platform == 'linux2':
    window.set_icon(pyglet.image.load(resourcepaths['misc']['brain'][0]))
...

self.spr_square = [pyglet.sprite.Sprite(pyglet.image.load(path))
...

self.image_set = [pyglet.sprite.Sprite(pyglet.image.load(path))
...

brain_icon = pyglet.sprite.Sprite(pyglet.image.load(random.choice(resourcepaths['misc']['brain'])))
...

if cfg.BLACK_BACKGROUND:
    brain_graphic = pyglet.sprite.Sprite(pyglet.image.load(random.choice(resourcepaths['misc']['splash-black'])))
else:
    brain_graphic = pyglet.sprite.Sprite(pyglet.image.load(random.choice(resourcepaths['misc']['splash'])))

I should take a look at the Python 2.7.9 to 2.7.10 changelog to see if there are any changes that might affect pyglet image loading...

Issues with Terminator and GNU Screen

One of the first Linux distros I used was Crunchbang Linux based on Ubuntu 9.04. The default terminal was terminator, and I thought it was so cool how you could split terminal windows vertically and horizontally multiple times using the hotkeys C-S-o (horizontal split), C-S-e (vertical split). I have continued using terminator on all my personal machines and even in some VM's where terminator's ability to split a window into multiple sub-windows is very helpful for making the most of limited screen real estate.

When I was a Linux hobbyist, I almost never used ssh or telnet, let alone a serial console cable to connect to other machines, but now as a Linux engineer I often connect remotely to dozens of different machines in one session. Using gnome-terminal with tabs or terminator with split windows has its limits when working with more than a handful of remote machines. I started using GNU Screen as my terminal multiplexer of choice because unlike tmux (which many people prefer to Screen) GNU Screen supports serial console connections with much less fuss than minicom and with much more robustness than putty.

Over the last several months I have launched GNU Screen from within terminator, which does a passable job of supporting screen for simple terminal tasks. The problem, however, is that if you try to use vi/vim in a screen session that is itself launched inside terminator, upon exiting from your vim editing session, GNU Screen's output will only scroll in the top half of the terminator window!

Another weird occurrence when using GNU Screen from within terminator is that after several thousand lines are stored in terminator's scrollback buffer (which I have set to 'unlimited') during a serial console session, Screen will start returning gibberish even though the console speed is set to the correct setting for your hardware.

Check out this totally broken login screen (which is supposed to be a standard /etc/issue banner)

#####################################################################

 #  T␤␋⎽ ⎽≤⎽├␊└ ␋⎽ °⎺⎼ ├␤␊ ┤⎽␊ ⎺° ▒┤├␤⎺⎼␋≥␊␍ ┤⎽␊⎼⎽ ⎺┼┌≤.             #

 #  I┼␍␋┴␋␍┤▒┌⎽ ┤⎽␋┼± ├␤␋⎽ ␌⎺└⎻┤├␊⎼ ⎽≤⎽├␊└ ┬␋├␤⎺┤├ ▒┤├␤⎺⎼␋├≤, ⎺⎼ ␋┼  #

 #  ␊│␌␊⎽⎽ ⎺° ├␤␊␋⎼ ▒┤├␤⎺⎼␋├≤, ▒⎼␊ ⎽┤␉┘␊␌├ ├⎺ ␤▒┴␋┼± ▒┌┌ ⎺° ├␤␊␋⎼    #

 #  ▒␌├␋┴␋├␋␊⎽ ⎺┼ ├␤␋⎽ ⎽≤⎽├␊└ └⎺┼␋├⎺⎼␊␍ ▒┼␍ ⎼␊␌⎺⎼␍␊␍ ␉≤ ⎽≤⎽├␊└       #

 #  ⎻␊⎼⎽⎺┼┼␊┌.                                                       #

 #                                                                   #

 #  I┼ ├␤␊ ␌⎺┤⎼⎽␊ ⎺° └⎺┼␋├⎺⎼␋┼± ␋┼␍␋┴␋␍┤▒┌⎽ ␋└⎻⎼⎺⎻␊⎼┌≤ ┤⎽␋┼± ├␤␋⎽    #

The blocks are control characters. The banner should look something like:

#####################################################################

 #  This system is for the use of authorized users only.             #

 #  Individuals using this computer system without authority, or in  #

 #  excess of their authority, are subject to having all of their    #

 #  activities on this system monitored and recorded by system       #

 #  personnel.

...

This problem can be solved by detaching the screen session with C-a d, exiting the current terminator session, opening an entirely new terminator session and attaching to the detached GNU Screen session with screen -r.

What is interesting is that I have not run into the scrolling bug (in which text only scrolls in the top half of the terminal window.) when using vi/vim within a screen session launched from lxterminal. (Note: scrolling issues after invoking vim in a GNU screen session can be mostly fixed by adding altscreen on to your ~/.screenrc)

Although I still enjoy using terminator for day-to-day CLI tasks on my local machine, for remote sessions I now use GNU Screen with other terminal emulators besides terminator. I have tested GNU Screen and xfce4-terminal and lxterminal and I suspect that screen will also work well with gnome-terminal without the bugs I experienced with terminator.


Postscript 2015-10-11

After several months of using various terminals with GNU Screen over serial console, I have learned that serial console text corruption issue that often occurs after a remote machine reboot when the scrollback buffer is quite full affects all the vte-dependent terminals I've used including terminator, lxterminal, and xfce4-terminal.

ksvalidator - linter for kickstart automated install config files

Unattended installations with anaconda and Kickstart files are a must when installing RHEL/CentOS on multiple machines via PXE. One problem, however, is that you often don't know if you've made a typo or some syntax error within your Kickstart file until the anaconda installer tells you something is wrong with the configuration parameters. Once such an error occurs, you have no choice but to reboot.

This is a big headache when working on servers that have super-long reboot times, like the HP Proliant DL980 Gen 8 and 9 machines that take 10 minutes or more to reboot. Several reboots caused by invalid Kickstart files will easily eat up an hour or two during tight maintenance windows at night.

Thankfully, there is a python-based linter for Kickstart files called ksvalidator which is available from the pykickstart package on RHEL and CentOS or the python2-pykickstart package from AUR for Archlinux.

The most important option flag is -v (--version) which takes the argument RHELX where X is some number denoting the RHEL version, i.e. RHEL5, 6, 7.

If you don't specify a version, it will use the highest current RHEL version by default, which is RHEL7 (as of May 2015).

Here is some sample output:

[archjun@arch pxe]$ ksvalidator ks5_sk_20140701.cfg 

The following problem occurred on line 8 of the kickstart file:

Unknown command: key

The following problem occurred on line 24 of the kickstart file:

Unknown command: interactive

The following problem occurred on line 204 of the kickstart file:

Section %packages does not end with %end

Running ksvalidator on a kickstart file for RHEL5 shows the errors above, but these would only be errors according to the kickstart syntax for RHEL7!

The command above with no option flags is equivalent to executing ksvalidator -v RHEL7 ...

If we re-run ksvalidator on the same file, this time specifying version RHEL5, no syntax errors are found:

[archjun@arch pxe]$ ksvalidator -v RHEL5 ks5_sk_20140701.cfg

This is all well and good, but you should note that ksvalidator cannot catch logic errors, for example, trying to format a partition with an unsupported partition type. For instance, RHEL5.X on Linux kernel 2.6.18-X does not support ext4, but if you try to format a partition as ext4 in a RHEL5.X kickstart file, the linter will not catch the error! ksvalidator also cannot catch the incorrect use of mbr partition table type on a system using UEFI instead of legacy BIOS. In such a case, the kickstart file would have to specify use of gpt partition table.

For these types of mistakes, you must create your own error-checking.

Let's look at an excerpt from a problematic kickstart file for RHEL5 containing the following lines:

...
clearpart --initlabel --all
zerombr # no prompt when deleting all partitions
part /boot --fstype ext4 --size=20482 --ondisk=cciss/c0d0 --asprimary
#part /usr/local --fstype ext3 --size=30720 --ondisk=cciss/c0d0
part /usr --fstype ext4 --size=30720 --ondisk=cciss/c0d0
part /var --fstype ext4 --size=18432 --ondisk=cciss/c0d0
part / --fstype ext5 --size=10240 --ondisk=cciss/c0d0 --asprimary
part swap --size=8192 --ondisk=cciss/c0d0 --asprimary
part /workspace --fstype ext3 --size=2048 --ondisk=cciss/c0d0
firstboot --disable
...

There are several problems above. As mentioned earlier, RHEL5.X does not support ext4, so trying to format /usr and /var/ as ext4 will cause the anaconda installer to terminate. Another problem is the typo ext5 which is a non-existent partition type.

You could do RHEL5.X kickstart filesystem error checking with the following one-liner:

[archjun@arch pxe]$ grep -wi "part" ks5-err-example.cfg | grep -v "ext3"
part /boot --fstype ext4 --size=20482 --ondisk=cciss/c0d0 --asprimary
part /usr --fstype ext4 --size=30720 --ondisk=cciss/c0d0
part /var --fstype ext4 --size=18432 --ondisk=cciss/c0d0
part / --fstype ext5 --size=10240 --ondisk=cciss/c0d0 --asprimary
part swap --size=8192 --ondisk=cciss/c0d0 --asprimary

This returns all the problematic lines containing non-ext3 partitions. It would be great if ksvalidator added functionality for catching some obvious partitioning errors.

Some observations about glibc GHOST vulnerability patching in the field

I currently work as a Linux System Engineer for an open source software service company that provides manpower for various Managed Service Providers (MSP) serving the Korean telecom industry. I spend most of my days out of the office on service calls ranging from basic server installation, system monitoring, to troubleshooting. In the normal course of my work I come into contact with sysadmins with varying levels of Linux familiarity. About 50% of the time the sysadmins I meet are more familiar with Windows or various flavors of Unix like HPUX, AIX, Solaris, etc. Of course, there are some sysadmins who are quite adept at Linux, too.

Thanks to an itinerant work arrangement in which I visit different sites every day, I have some perspective on the diverse ways that IT staff maintain their Linux servers. 2014 and early 2015 have been a busy time for patching servers with bugs found in Bash, OpenSSL and glibc. Although the glibc GHOST vulnerability was announced in Jan 2015, some of our clients still haven't completed patching all of their servers. Several sysadmins have asked me about the proper rpm commands for upgrading glibc-related packages on RHEL or CentOS. I was a bit worried when they said they used commands like

rpm -Uvh --nodeps pkgName

rpm -i --force --nodeps pkgName

as hacky workarounds for dependency errors that popped up when they tried to update glibc.

For the record, at my employer Growin, we recommend the following steps for upgrading glibc.

1) First identify relevant glibc packages installed on your RHEL/CentOS system

rpm -qa | grep -E "glibc|nscd" | grep -v "compat"

This will return a list of glibc-related packages on your system with the exclusion of compat- packages which don't need to be upgraded. For why compat-glibc doesn't need to be patched, see the Redhat Solutions post Is compat-glibc affected by GHOST, glibc vulnerability CVE-2015-0235? (login required). I quote,

The dynamic libraries provided by the compat-glibc package are not vulnerable because they do not provide runtime code...

2) Although there are more than 16 packages related to glibc that you could install, most probably you don't have glibc-debug* or the glibc-utils packages installed on your system. If these packages did not appear in the results from step 1, do not try to install them (doing so can lead to dependency errors)! Prepare only the updated rpm's that you need for an upgrade in a separate directory and then

rpm -Uvh glibc*

should do the trick. If your system has nscd installed, the above command would become:

rpm -Uvh glibc* nscd*


It is also possible to skip step 1 if you use the rpm -F flag (--freshen), which, according to man rpm:

will upgrade packages, but only ones for which an earlier version is installed

Desktop Linux users of Fedora, CentOS, or other rpm-based distros might wonder why sysadmins go to all the trouble of manually upgrading packages using rpm -Uvh pkgName when they could just do a yum update and upgrade all packages at one go.

There are several good reasons for not using yum update in an enterprise environment. First, some servers are only connected to an internal network. Second, custom applications created by developers may be compiled against a certain version of the C libraries in glibc, so doing a yum update runs the risk of breaking applications. Mainly for the second reason, it is not uncommon to see production servers running really old kernels like 2.6.18 (RHEL 5.X), and I have heard horror stories from coworkers about companies that use even older kernels!