2016년 3월 26일 토요일

Turning off OpenGL debug mode in Pyglet for Brain Workshop

I recently read an article on Infoworld by Serdar Yegulalp about performance optimizations for Python. The following tip caught my eye:

Pyglet, a handy library for creating windowed graphical applications, automatically enables a debug mode, which dramatically impacts performance until it’s explicitly disabled.
I checked the Pyglet docs for proof of this and found the following:

http://pyglet.readthedocs.org/en/latest/programming_guide/options.html

options = {'xsync': True, 'debug_gl_trace': False, 'debug_media': False, 'debug_gl': True, 'debug_font': False, 'font': ('gdiplus', 'win32'), 'audio': ('directsound', 'pulse', 'openal', 'silent'), 'debug_win32': False, 'xlib_fullscreen_override_redirect': False, 'debug_trace_depth': 1, 'debug_lib': False, 'debug_x11': False, 'debug_trace_args': False, 'debug_gl_trace_args': False, 'shadow_window': True, 'debug_texture': False, 'debug_trace_flush': True, 'darwin_cocoa': False, 'vsync': None, 'search_local_libs': True, 'debug_trace': False, 'debug_graphics_batch': False, 'graphics_vbo': True}

Apparently OpenGL debugging is turned on by default!

From the docs:
Global dict of pyglet options. To change an option from its default, you must import pyglet before any sub-packages. For example:

    import pyglet
    pyglet.options['debug_gl'] = False

debug_gl
    If True, all calls to OpenGL functions are checked afterwards for errors using glGetError. This will severely impact performance, but provides useful exceptions at the point of failure. By default, this option is enabled if __debug__ is (i.e., if Python was not run with the -O option). It is disabled by default when pyglet is “frozen” within a py2exe or py2app library archive.

I therefore added the highlighted snippet at line 814 in the Brainworkshop 4.8.7 source inside of a try-except block:

try:
    # workaround for pyglet.gl.ContextException error on certain video cards.
    os.environ["PYGLET_SHADOW_WINDOW"]="0"
    import pyglet

    # disable OpenGL debugging mode which can hurt performance
    pyglet.options['debug_gl'] = False

    from pyglet.gl import *
    if NOVBO: pyglet.options['graphics_vbo'] = False
    from pyglet.window import key
except:
    quit_with_error(_('Error: unable to load pyglet.  If you already installed pyglet, please ensure ctypes is installed.  Please visit %s') % WEB_PYGLET_DOWNLOAD)


I'm not sure how to rigorously measure the changes in execution time, however. Although Brain Workshop feels faster and is using less cpu according to htop, I would like to use something like the Python timeit module to measure a baseline for the original version and then time the edited code.

2016년 3월 19일 토요일

ASUS U36JC Notebook Intel H55 chipset does not support 8GB DDR3 modules

I have happily used a 13.3" ASUS U36JC notebook since 2011. When I bought it I made sure that 8GB DDR3 of total memory was installed (2 x 4GB DDR3). Back then, 8GB was a large amount of memory, but as the years have gone by, I have required more and more memory (Chrome has become a memory hog, and I use virtualization via KVM and virtualbox much more frequently).

Recently at work I have been put on an Openstack project so I want to be able to play with Openstack at home on my personal notebook as well. Although simple all-in-one(AIO) installations will work with just 8GB of RAM, standalone physical installations using the Ubuntu installer for Openstack Kilo require at least 12GB, while installation of the Kolla project (which containerizes all Openstack components within Docker) requires > 8GB to build and run ~80 containers.

I used dmidecode to query the system BIOS for max memory supported and info on the memory modules currently populating the memory banks.

[fedjun@u36jfed23 ~]$ sudo dmidecode -t 16
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.6 present.

Handle 0x0041, DMI type 16, 15 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: None
    Maximum Capacity: 16 GB
    Error Information Handle: Not Provided
    Number Of Devices: 4


DMI type 16 shows the Physical Memory Array. According to the information above, the motherboard has 4 memory slots and can support a total of 16GB of RAM. This means that each memory bank can support up to 4GB RAM. Note that although the Intel H55 chipset can support up to 4 physical memory banks, there are only two physical memory slots offered on the ASUS U36JC. According to the following Superuser forum thread, this is not an unusual occurrence:

http://serverfault.com/questions/137491/dmidecode-showing-more-ram-slots-than-available

But since the mobo can theoretically support up to 16GB DDR3, I decided to make a gamble and purchased two Samsung Electronics 8GB DDR3L PC-12800 memory modules:


The picture above shows the 8GB DDR3L (low-power 1.35V) memory modules installed on my ASUS U36JC notebook. Here is the information on the memory modules provided by dmidecode reading from the BIOS:

[fedjun@u36jfed23 ~]$ sudo dmidecode -t 17
[sudo] password for fedjun:
# dmidecode 3.0
Getting SMBIOS data from sysfs.
SMBIOS 2.6 present.

Handle 0x0042, DMI type 17, 28 bytes
Memory Device
    Array Handle: 0x0041
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 8192 MB
    Form Factor: SODIMM
    Set: None
    Locator: DIMM0
    Bank Locator: BANK 0
    Type: DDR3
    Type Detail: Synchronous
    Speed: 1067 MHz
    Manufacturer: 80CE
    Serial Number: 312BBD50
    Asset Tag: Unknown
    Part Number: M471B1G73DB0-YK0 
    Rank: Unknown

Handle 0x0044, DMI type 17, 28 bytes
Memory Device
    Array Handle: 0x0041
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: DIMM
    Set: None
    Locator: DIMM1
    Bank Locator: BANK 1
    Type: Unknown
    Type Detail: None
    Speed: Unknown
    Manufacturer: [Empty]
    Serial Number: [Empty]
    Asset Tag: Unknown
    Part Number: [Empty]
    Rank: Unknown

Handle 0x0045, DMI type 17, 28 bytes
Memory Device
    Array Handle: 0x0041
    Error Information Handle: Not Provided
    Total Width: 64 bits
    Data Width: 64 bits
    Size: 8192 MB
    Form Factor: SODIMM
    Set: None
    Locator: DIMM2
    Bank Locator: BANK 2
    Type: DDR3
    Type Detail: Synchronous
    Speed: 1067 MHz
    Manufacturer: 80CE
    Serial Number: 312BBD72
    Asset Tag: Unknown
    Part Number: M471B1G73DB0-YK0 
    Rank: Unknown

Handle 0x004A, DMI type 17, 28 bytes
Memory Device
    Array Handle: 0x0041
    Error Information Handle: Not Provided
    Total Width: Unknown
    Data Width: Unknown
    Size: No Module Installed
    Form Factor: DIMM
    Set: None
    Locator: DIMM3
    Bank Locator: BANK 3
    Type: Unknown
    Type Detail: None
    Speed: Unknown
    Manufacturer: [Empty]
    Serial Number: [Empty]
    Asset Tag: Unknown
    Part Number: [Empty]
    Rank: Unknown


DMI type 17 shows the Memory Device. Banks 0 and 2 are populated and Banks 1 and 3 are empty (and do not physically exist on the U36JC Intel H55 chipset motherboard).

BIOS detects the memory modules and Linux will boot; however, once a memory address above 4GB is accessed, the system will reboot or a kernel panic will result. I contacted ASUS and asked if a BIOS upgrade could fix this issue, but ASUS tech support responded that this is a physical limitation of the H55 chipset as implemented on the mobo for the ASUS U36JC.

It seems that the maximum amount of memory supported in each memory bank is 4GB!

2016년 3월 12일 토요일

Uncompressed audio playback using Python 2 pyglet

Python 2's pyglet module is quite popular for making games as it can render graphics files like png and also playback uncompressed audio. pyglet is not capable of playing compressed audio, however; for this task it has traditionally relied on the avbin plugin. Unfortunately, avbin is not under active development anymore and many programs using pyglet require ancient versions of avbin to be installed on your system.

Take, for instance, the dual N-back training game Brain Workshop (BW). Written in Python 2 and pyglet, it uses both uncompressed wav and compressed ogg audio files. For playing ogg files, BW requires avbin 7, which was released in 2008. This is a problem because most Linux distributions don't provide avbin in their default repositories (and even if they did, they would use the latest version avbin 10/11). In another post, I will describe some changes I made to Brain Workshop's python source to remove the dependency on avbin. The trick is to first convert all compressed audio to wav and then to use pyglet's built-in audio playback function.

When attempting to play wav files with pyglet, you must make sure that there are no audio tags encoded in the file. If any tags exist, pyglet will give a WAVEFormatException complaining that the format is invalid. Let's say you convert ogg to wav using ffmpeg:

$ ffmpeg -i Concert.ogg Concert.wav

A file converted in this way will be tagged with the string Lavf56.40.101 in the field Software:


The screenshot above is from Audacity, an open source digital audio workstation. If you select edit metadata tags from the File menu you will be able to see what information, if any, is encoded as metadata. Clear Lavf56.40.101 and re-export the file to wav (Microsoft PCM, 16 bit) and you should be able to play the file with pyglet:


import pyglet
source = pyglet.media.load('Concert.wav')
source.play()

2016년 3월 5일 토요일

Canon Pixma MG2490 as CUPS print server on Fedora 23

A machine with Fedora 23 installed will act as a print server for both Linux and Windows clients.

What is really nice is that using ipp, CUPS can now share printers over the network to Windows clients without messing with Samba configuration!

At first, I naively thought that simply installing cups and xsane would be suffcient:

sudo dnf install cups xsane

However these packages and their dependencies are not sufficient for getting the MG2490 to print or scan.

Initially I also tried installing the package cnijfilter-mg2400-series.rpm.tar.gz but again, this package alone is not sufficient for setting up the printer/scanner. I therefore removed this package and was able to get the printer/scanner working just fine with the pure open-source drivers.

Referring to /var/log/dnf.log Here are all the packages I installed on Fedora 23 server for the Canon Pixma MG2490:

cups
gutenprint
gutenprint-libs
gutenprint-foomatic
gutenprint-cups

foomatic
foomatic-db
system-config-printer
(optional, but recommended)
sane
sane-backends-drivers-scanners
sane-backends
xsane (optional, but recommended)

Once everything is installed, start the cups systemd service:

systemctl start cups

Make sure the MG2490 is connected by USB cable to your CUPS print server and then run system-config-printer as root. Select "Add" and choose printer and your MG2490 should appear in a list. Choose the "USB MG2490" entry. The gutenprint-foomatic package includes a postscript PPD file for the Canon MG2400 series, so your printer should appear in the list under Canon printers





Since you will be sharing this printer on the network, in the printer properties make sure it is shared for all users. One of the dialog boxes will also offer to set up rules for CUPS and IPP in firewalld (how convenient!).
If you don't use system-config-printer, however, you will have to add several rules to firewalld manually using the gui firewall-config or on the command line with firewall-cmd.

In your default zone, open up port 631/tcp (for CUPS remote administration http interface) and also allow ipp-client traffic.There might be other settings required, but these are the ones that occur off the top of my head (I just used system-config-printer because I'm lazy).

Now from the server machine, open a browser and navigate to http://localhost:631, login to Printer Administration with root:password
 for your server, and print a test page. You can also print a test page from system-config-printer.

Other Linux clients can find the CUPS server if you enter it's IP address in the system-config-printer "Network Printer -> Find Network Printer" dialog box.

To add the Canon MG2490 for Windows clients, click on Control Panel -> Add New Printer, and select Network Printer.

In the address bar, enter the IPP address of your printer listed in CUPS. You can find this address in the CUPS web admin page on port 631 or through system-config-printer. In my case, the IPP address is:

ipp://192.168.30.6:631/printers/Canon_MG2400_series
But in the "Add Network Printer" dialog box on Windows clients, change this to http:

http://192.168.30.6:631/printers/Canon_MG2400_series

Note that the Windows client must have windows drivers for the MG2490 installed. Windows users will be prompted to select the printer driver. They should select the driver named "MG 2400 series". In the case of Windows 10, however, it will automatically detect the printer and install all drivers for the printer and scanner automatically! Once the printer is set up, have the Windows user print a test page. In my experience the initial test page took > 1 minute to print, but subsequent print jobs from Windows clients were processed much more quickly (< 30 sec).

It is such a relief to be able to set up a Linux print server for Windows clients without messing with Samba shares. Thank you, Linux + IPP!







2016년 2월 27일 토요일

Internet connection sharing through a computer with two NIC's

Recently a new server room was built at the office and now all the network and server gear has been moved there. I used to use wireless connections on my notebooks out of convenience, but now that the wireless router is inside the server room instead of in the middle of an open floor plan office, the wireless signal is too weak.
My notebooks are connected to the internal network, however. Most of the test servers have dual NIC's so on a server with Fedora 23 installed, I first enabled port forwarding manually:
# check if port forwarding is enabled (it isn't)
[fedgro@fx8350no2 sysctl.d]$ sudo sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
# enable it
[fedgro@fx8350no2 sysctl.d]$ sudo sysctl net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1
# verify that port forwarding has been enabled:
[fedgro@fx8350no2 sysctl.d]$ sudo sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

The change above will only apply until the next reboot, so to make the changes permanent, I want to edit sysctl.conf

In recent versions of Fedora using systemd, this conf file is no longer monolithic and is generated at boot by systemd from files in /usr/lib/sysctl.d/ (distribution default kernel settings) and in /etc/sysctl.d/ (custom kernel settings)
Here is the content of each directory:

[fedgro@fx8350no2 sysctl.d]$ cd /usr/lib/sysctl.d/
[fedgro@fx8350no2 sysctl.d]$ ls
00-system.conf              50-coredump.conf  60-libvirtd.conf
10-default-yama-scope.conf  50-default.conf
[fedgro@fx8350no2 sysctl.d]$ cd /etc/sysctl.d/
[fedgro@fx8350no2 sysctl.d]$ ls
50-libreswan.conf  99-sysctl.conf
Under /etc/sysctl.d I created a custom file 98-ipv4-forward.conf containing the single line

net.ipv4.ip_forward=1
Now that ipv4 port forwarding has been enabled, it is time to set up IP masquerading in the firewall. Fedora 23 uses firewalld instead of iptables, so to make the appropriate settings you can use the GUI firewall-config or the cli tool firewall-cmd.

You can see that firewalld is running:

[fedgro@fx8350no2 sysctl.d]$ systemctl status firewalld
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2016-02-22 10:13:28 KST; 23h ago
 Main PID: 2132 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─2132 /usr/bin/python3 -Es /usr/sbin/firewalld --nofork --nopid

Feb 22 10:13:28 fx8350no2 systemd[1]: Starting firewalld - dynamic firewal.....
Feb 22 10:13:28 fx8350no2 systemd[1]: Started firewalld - dynamic firewall...n.
Hint: Some lines were ellipsized, use -l to show in full.

Now I use firewall-cmd below to set up masquerading:
# Check the default firewall zone
# (FedoraServer is the default for F23)
[fedgro@fx8350no2 sysctl.d]$ sudo firewall-cmd --get-default

FedoraServer

Check the firewall zone for the NIC connected to a router providing access to the Internet:

[fedgro@fx8350no2 sysctl.d]$ sudo firewall-cmd --get-zone-of-interface=enp5s0
FedoraServer
 

Add IP masquerading to the default zone:

[fedgro@fx8350no2 sysctl.d]$ sudo firewall-cmd --zone=FedoraServer --add-masquerade
success

[fedgro@fx8350no2 sysctl.d]$ sudo firewall-cmd --permanent --zone=FedoraServer --add-masquerade
success

Verify changes have been made to FedoraServer zone:
[fedgro@fx8350no2 sysctl.d]$ firewall-cmd --zone=FedoraServer --list-all
FedoraServer (default, active)
  interfaces: enp5s0
  sources:
  services: cockpit dhcpv6-client http https ipp ipp-client samba-client sane ssh
  ports: 80/tcp 631/tcp
  protocols:
  masquerade: yes
  forward-ports:
  icmp-blocks:
  rich rules:

Now from my notebook connected to the internal network, I simply add the following route (to the server with dual NIC's):

sudo ip r add default via 192.168.95.145 dev enp1s0

where 192.168.95.145 is the IP of the NIC connected to the internal network on the dual-NIC server, and where enp1s0 is the ethernet port on my notebook.

And now the notebook connected to the internal network can talk to the outside world. Let's test on Google's DNS server 8.8.8.8:

[archjun@pinkS310 ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=51 time=36.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=51 time=35.3 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=51 time=36.1 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=51 time=36.6 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=51 time=35.9 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 35.342/36.157/36.718/0.496 ms
[archjun@pinkS310 ~]$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (192.168.95.145)  0.299 ms  0.278 ms  0.279 ms
 2  192.168.30.1 (192.168.30.1)  0.657 ms  0.695 ms  0.743 ms
 3  REDACTED (REDACTED)  11.346 ms  11.357 ms  11.349 ms
 4  REDACTED (REDACTED)  9.661 ms  13.730 ms  13.750 ms
 5  REDACTED (REDACTED)  9.541 ms  9.571 ms  9.566 ms
 6  112.189.28.97 (112.189.28.97)  8.167 ms  6.349 ms 112.189.29.97 (112.189.29.97)  15.201 ms
 7  112.174.103.225 (112.174.103.225)  10.270 ms 112.174.103.213 (112.174.103.213)  8.211 ms 112.174.119.205 (112.174.119.205)  6.721 ms
 8  112.174.48.162 (112.174.48.162)  6.676 ms  6.688 ms 112.174.8.42 (112.174.8.42)  6.898 ms
 9  112.174.83.218 (112.174.83.218)  6.854 ms 112.174.83.34 (112.174.83.34)  6.857 ms 112.174.84.122 (112.174.84.122)  6.957 ms
10  72.14.194.106 (72.14.194.106)  36.267 ms  36.246 ms  36.211 ms
11  216.239.54.13 (216.239.54.13)  36.528 ms 216.239.50.241 (216.239.50.241)  36.149 ms 216.239.54.13 (216.239.54.13)  41.606 ms
12  209.85.254.17 (209.85.254.17)  34.989 ms 209.85.246.89 (209.85.246.89)  37.053 ms 209.85.255.241 (209.85.255.241)  36.305 ms
13  google-public-dns-a.google.com (8.8.8.8)  36.593 ms  36.592 ms  42.129 ms

You can see that the first hop in the traceroute is to 192.168.95.145, the server on the internal network which has 2 NIC's. 1 NIC is connected to the internal network while one NIC is connected to the external network. It forwards the packets from the internal 192.168.95.0/24 network to the external network on subnet 192.168.30.0/24 which has access to the Internet through a router.

(the second link provides some useful commands for firewall-cmd)

2016년 2월 13일 토요일

Setup ibus-hangul in Fedora 23

Fedora 23 XFCE 데스크탑 판을 최근에 설치 했는데 한글 입력 방식 설치하기 위해

sudo dnf install ibus-hangul

명령어를 실행했습니다. 의존 패키지 ibus까지 다 설치 되고

ibus-daemon -rdx &

명령을 터미널에 실행하여 ibus 데몬은 배경에서 돌아가게 했습니다. ibus 설정에서 추가 언어 한글을 세팅 했는데도 한글 모드로 전환할 때 한글 입력이 안 됐습니다.

pidof ibus-daemon 하여 pid 알아낸 다음 sudo kill -15 xxx 해서 ibus-daemon을 멈추게 하고 문제를 추려내기 위해 ibus 데몬을 -v (또는 --verbose) 옵션으로 다시 실행했습니다. 구지 pid 킬 안 해도 -r (또는 --replace) 옵션으로 ibus-daemon을 실행시키면 기존에 돌고 있는 ibus 데몬 프로세스를 죽이고 새로운 인스턴스를 실행한답니다.

ibus-daemon -rvx

명령 실행하니까 vala와 gtk 경고나 에러가 나타났습니다. 혹시나 해서 xfce4-vala (제가 쓰는 DE는 xfce4이기 때문), ibus-pygtk2ibus-qt를 설치했습니다. X 윈도우 세션에서 로그아웃 하기 전에 달라진 게 없었지만 GUI환경에서 로그아웃 로긴 한 다음에 한글 입력이 잘 됐습니다!

다양한 앱 그리고 Java 프로그램에서까지 ibus-hangul 입력을 원하시면 제가 옛날 올린 글을 참조 하시길 바랍니다 (.bashrc 설정과 Java 앱 실행 스크립트 예제):

http://eatpeppershothot.blogspot.kr/2014/03/enabling-ibus-support-in-omegat.html

-------------------------------------------------------------

To get ibus-hangul working in Fedora 23, keep in mind that it is not sufficient to just dnf install ibus-hangul . There are a couple of additional packages you need to install:

ibus-pygtk2
ibus-qt

In my recollection, ibus-gtk2 and ibus-gtk3 are installed by default together with ibus-hangul, but if they aren't, make sure to install them as well.

Once this is done, log out of your X windows session and log back in (or just reboot) and you should find that Korean and Chinese (hanja) input works fine. Keep in mind that ibus-daemon needs to be running and that you have added Korean as an additional language in the ibus preferences.

Manually Installing xfce4 on Fedora 23 Server

I recently found myself needing to launch a VM image in qcow2 format using KVM. The VM required various additional settings such as bridged networking (using one of the host's ethernet ports) and attachment of virtual storage (/dev/vdXY). Unfortunately, since I am not familiar with the libvirt cli tool virsh, I had to rely on the nice GUI interface of virt-manager. The server machine I was working on did not have X windows installed, so I assumed that simply invoking

dnf install @xfce-desktop

would be sufficient. After installing all the packages and dependencies for xfce4, when I tried to launch X windows with startxfce4, I got the following error:

xinit: unable to run server "X": No such file or directory

The problem is caused by xfce4 packages not having X windows as a dependency!

To manually install all the X windows packages in Fedora 23, invoke the following:

dnf group install base-x

This will install all the all the xorg-x11-[blah] packages necessary for X windows to start.