Problem when using GNU Screen 'stuff' command with 40+ open tabs - runaway input to stdin

Once a month I conduct system diagnostics on about 40 machines at an IDC for a Korean food company. Because the commands I need to run (i.e. dmesg | grep -iE "criti|fail|warn|bug|") are the same for all  machines, I use a script to launch GNU Screen with dozens of tabs each ssh'ing into a different server and once connected to all machines, I use

C-a :at "#" stuff "cmd^M"

to send commands to all tabs at once. Without this feature, there is no way I could complete a system checkup on 40+ machines in less than a few hours.

I recently had to install the leap second patch for tzdata packages on 40+ RHEL6.X machines, and wanted to scp them from my local machine to all the servers in one fell swoop. Therefore I entered the following command into GNU Screen:

:at "#" stuff "scp -r archjun@10.200.250.156:/MULTIMEDIA/iso/pkg-update/tzdata^M" .

As this was a new ssh connection, I was prompted whether I wanted to continue connecting:

The authenticity of host '10.200.250.156 (10.200.250.156)' can't be established.
RSA key fingerprint is
...
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.200.250.156' (RSA) to the list of known hosts.
archjun@10.200.250.156's password:

Obviously I wasn't about to type 'yes' into each of 40+ tabs. Therefore I stuffed yes into all 40+ GNU Screen tabs using

:at "#" stuff "yes^M"

All was fine for about 20 tabs, but the remaining tabs entered an input loop stuffing an infinite number of y's to stdin

y
y
y
y
y
y
y
...

In my .screenrc config file I have automatic logging turned on with the two lines

logfile $HOME/Documents/term_sessions/%Y-%m-%d_%0c-%n.log
deflog on

so that a logfile is automatically generated as soon as a new GNU Screen tab is created (hotkey C-a c). I noticed that logfiles were suddenly ballooning to hundreds of megabytes in size.

I manually had to enter C-c into each of these tabs to stop the runaway input. I have never experienced anything like this when stuffing commands to all windows in GNU Screen. Has anyone else encountered similar behavior?

Setting up mcelog to work with systemd

If you regularly observe system logs such as /var/log/messages, dmesg, or journalctl (systemd) you will eventually encounter a Machine Check Event (mce) which warns you that some kind of hardware error has occurred. For example, a common mce is caused when an incorrect bit is flipped in RAM. For server ECC memory, this is less of a problem because such bit errors can be fixed automatically. When encountering servers in the field with uptime greater than 365 days, it is not hard to find mce errors logged here and there. In the case of RHEL 5/6 machines I encounter in the field, mce errors are logged in the file /var/log/mcelog and the mcelog service runs by default.

Recently I noticed that every few days the kernel ring buffer dmesg on my work laptop gives the following error:

[Jun11 23:49] mce: [Hardware Error]: Machine check events logged

However, when I navigate to /var/log/ I cannot see any file named mcelog. Some old posts floating around the Internet recommend redirecting mcelog to some output file, i.e. /usr/sbin/mcelog > mcelog.out but this didn't work for me. Make sure you have the mcelog package (as it is called in Arch) installed . To enable the daemon in systemd, systemctl enable mcelog. When running mcelog on a Linux machine running systemd instead of the old syslog, you need to make some changes to /etc/mcelog/mcelog.conf

What led me astray was the Archwiki page on MCE Handling, which recommends uncommenting the line

syslog = yes

If  you are running systemd you do NOT want the above setting! The problem is that systemd handles system logging through journalctl. You can follow the other suggestions in the Archwiki to run mcelog as a daemon (daemon = yes), but make sure the syslog lines are commented out. Also you need to specify an output log file for mce errors by uncommenting the following in /etc/mcelog/mcelog.conf:

logfile = /var/log/mcelog

Also uncomment the following in /etc/mcelog/mcelog.conf

run-credentials-user = root

Restart the mcelog service

systemctl restart mcelog

Next time a Machine Check Event occurs, it will be written to /var/log/mcelog. Here is some sample output:

Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 5
MISC b8a0000086 ADDR ffb07500
TIME 1434034181 Thu Jun 11 23:49:41 2015
MCG status:
MCi status:
Error overflow
Uncorrected error
MCi_MISC register valid
MCi_ADDR register valid
Processor context corrupt
MCA: corrected filtering (some unreported errors in same region)
Generic CACHE Level-2 Generic Error
STATUS ee0000000040110a MCGSTATUS 0
MCGCAP c07 APICID 0 SOCKETID 0
CPUID Vendor Intel Family 6 Model 69
Hardware event. This is not a software error.
MCE 0
CPU 0 BANK 5
MISC 78a0000086 ADDR ffb07500

This seems to be indicating a memory error in the CPU cache.

Here is my /etc/mcelog/mcelog.conf file:

[SOLVED] Upgrade from Python 2.7.9 to 2.7.10 breaks Brainworkshop

Note: I originally made this post in June 2015, but in Dec. 2015 I found a workaround to get Brainworkshop 4.8.4/4.8.7 to work with Python 2.7.10+. Please refer to the new post at the following link:
http://eatpeppershothot.blogspot.kr/2015/12/enabling-brainworkshop-487-to-work-with.html

=================================================================
Recently Archlinux made python2-2.7.10-1 available in the main Arch repositories. Unfortunately, however, the Brainworkshop N-back training program doesn't work with the latest version of Python 2.

In Gentoo and Arch, the brainworkshop.pyw source file is launched by the startup script /usr/bin/brainworkshop which looks like the following (I have edited it to correspond to my BW data and config files on Dropbox):



After upgrading from Python 2.7.9 to 2.7.10, however, the following error message is generated:

/usr/bin/brainworkshop: line 8:  5212 Segmentation fault      (core dumped) python2 /usr/share/brainworkshop/brainworkshop.pyw --configfile $BRW_CONFIGFILE --statsfile $BRW_STATFILE --datadir $BRW_DATADIR

The actual Python 2 source file is located at /usr/share/brainworkshop/brainworkshop.pyw (.pyw instead of .py so that a separate console window won't be launched). Even launching brainworkshop.pyw directly without any option flags returns Segmentation fault (core dumped). In Archlinux, core dump files are located in /var/lib/systemd/coredump, but rather than trying to analyze a dump of system memory at the time brainworkshop.pyw crashed, I decided to run gdb to see if I could get more informative error messages through a stack trace. Although gdb is normally run on C/C++ compiled executables, gdb (version 7+) will also work with Python if you invoke it as follows:

[archjun@lenovoS310 ~]$ gdb /usr/bin/python2
GNU gdb (GDB) 7.9.1
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python2...(no debugging symbols found)...done.

Now at the gdb prompt, type run followed by the name of the python program:

(gdb) run /usr/share/brainworkshop/brainworkshop.pyw
Starting program: /usr/bin/python2 /usr/share/brainworkshop/brainworkshop.pyw
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Traceback (most recent call last):
  File "/usr/share/brainworkshop/brainworkshop.pyw", line 1023, in
    window = MyWindow(cfg.WINDOW_WIDTH, cfg.WINDOW_HEIGHT, caption=''.join(caption), style=style, vsync=VSYNC)
  File "/usr/lib/python2.7/site-packages/pyglet/window/xlib/__init__.py", line 163, in __init__
    super(XlibWindow, self).__init__(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pyglet/window/__init__.py", line 494, in __init__
    display = get_platform().get_default_display()
  File "/usr/lib/python2.7/site-packages/pyglet/window/__init__.py", line 1766, in get_default_display
    return pyglet.canvas.get_display()
  File "/usr/lib/python2.7/site-packages/pyglet/canvas/__init__.py", line 82, in get_display
    return Display()
  File "/usr/lib/python2.7/site-packages/pyglet/canvas/xlib.py", line 83, in __init__
    raise NoSuchDisplayException('Cannot connect to "%s"' % name)
pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to "None"
[Inferior 1 (process 26382) exited with code 01]

So it seems like Python's pyglet module is the culprit once again in breaking Brainworkshop. In a previous post from February 2015 (pyglet 1.2 breaks Brainworkshop), I described a way to get Brainworkshop working again by making a few edits to the /usr/share/brainworkshop/brainworkshop.pyw source file. I have played around with changing some of the parameters in line 1024

window = MyWindow(cfg.WINDOW_WIDTH, cfg.WINDOW_HEIGHT, caption=''.join(caption), style=style, vsync=VSYNC)

but I have yet to succeed in getting Brainworkshop to work with Python 2.7.10.

Workaround

Luckily, BW still works with Python 2.7.9, so I downgraded to 2.7.9 with pacman -U /var/cache/pacman/pkg/python2-2.7.9-1-x86_64.pkg.tar.xz (if you don't have this package in your pacman package cache, you can still download it from the Arch Linux Archive) and added python2 to IgnorePkg in /etc/pacman.conf as a temporary workaround.

Update 2015-09-07

python2-pyglet in Archlinux was recently upgraded from 1.2.3-1 to 1.2.4-1. Now when trying to run Brainworkshop with Python 2.7.10 I get the following error in brainworkshop.pyw when debugging with gdb:

[archjun@latitude630 ~]$ gdb /usr/bin/python2
GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/python2...(no debugging symbols found)...done.
(gdb) run /usr/share/brainworkshop/brainworkshop.pyw
Starting program: /usr/bin/python2 /usr/share/brainworkshop/brainworkshop.pyw
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
[New Thread 0x7fffeaef1700 (LWP 8948)]
[Thread 0x7fffeaef1700 (LWP 8948) exited]
[New Thread 0x7fffeaef1700 (LWP 8949)]
[New Thread 0x7fffedd79700 (LWP 8950)]

Program received signal SIGSEGV, Segmentation fault.
0x00007fffde270664 in gdk_pixbuf_loader_write () from /usr/lib/libgdk_pixbuf-2.0.so.0

It looks like we are no longer getting errors from pyglet like we did above with pyglet 1.2.3, but now we are having problems with libgdk_pixbuf-2.0.so.0. Let's take a look at this file:

[archjun@latitude630 ~]$ ls -al /usr/lib/libgdk_pixbuf-2.0.so.0
lrwxrwxrwx 1 root root 29 Aug 20 20:32 /usr/lib/libgdk_pixbuf-2.0.so.0 -> libgdk_pixbuf-2.0.so.0.3100.6

Apparently libgdk_pixbuf-2.0.so.0 is a symlink from the file /usr/lib/libgdk_pixbuf-2.0.so.0.3100.6

What package owns libgdk_pixbuf-2.0.so.0.3100.6?

[archjun@latitude630 lib]$ sudo pacman -Qo libgdk_pixbuf-2.0.so.0.3100.6
[sudo] password for archjun: 
/usr/lib/libgdk_pixbuf-2.0.so.0.3100.6 is owned by gdk-pixbuf2 2.31.6-1

[archjun@latitude630 lib]$ sudo pacman -Ss gdk-pixbuf2
extra/gdk-pixbuf2 2.31.6-1 [installed]
    An image loading library

OK~ so gdk-pixbuf2 loads images.

According to gdb, the function that is causing trouble is called gdk_pixbuf_loader_write (). Looking at the Gnome GDK-PixBuf Reference Manual, we can find more detailed information on the C function gdk_pixbuf_loader_write (), but I'm not familiar with C so I am not sure where to start debugging this issue.

A google search for gdk-pixbuf2 gdk_pixbuf_loader_write () segfault returns an interesting (but dated) google code issues page on github that discusses problems with pyglet decoding images using gdk-pixbuf2. One proposed workaround is to try PIL, pyllow or pypng for image decoding instead of gdk-pixbuf2.

Opening /usr/share/brainworkshop/brainworkshop.pyw with a text editor such as vim or emacs quickly reveals that there are no explicit calls to gdk functions in the Python code. We know that pyglet can use gdk-pixbuf2 when it tries to decode images, so I searched for invocations of pyglet.image and found 6 results at lines 1031, 2304, 2327, 4622, 4626, and 4628, respectively:

if sys.platform == 'linux2':
    window.set_icon(pyglet.image.load(resourcepaths['misc']['brain'][0]))
...

self.spr_square = [pyglet.sprite.Sprite(pyglet.image.load(path))
...

self.image_set = [pyglet.sprite.Sprite(pyglet.image.load(path))
...

brain_icon = pyglet.sprite.Sprite(pyglet.image.load(random.choice(resourcepaths['misc']['brain'])))
...

if cfg.BLACK_BACKGROUND:
    brain_graphic = pyglet.sprite.Sprite(pyglet.image.load(random.choice(resourcepaths['misc']['splash-black'])))
else:
    brain_graphic = pyglet.sprite.Sprite(pyglet.image.load(random.choice(resourcepaths['misc']['splash'])))

I should take a look at the Python 2.7.9 to 2.7.10 changelog to see if there are any changes that might affect pyglet image loading...