Make sure to disable NetworkManager before manually assigning IP addresses to network interfaces

This might be obvious, but I am writing this post as a note to myself in the future: stop NetworkManager before you do network configuration work in Linux!

I had the following happen to me several weeks ago. I was sent to install RHEL on several machines during a server room maintenance window from 2:00~5:00 am. Strangely, however, PXE installation kept freezing in the middle of the installation. tail -f /var/log/messages revealed that NetworkManager was deleting the IP address I had assigned to the PXE server (followed by messages that ntpd was removing the IP from its records). I once again manually added an IP with

ip addr add 192.168.10.100/24 broadcast 192.168.10.255 dev eth0

which enabled the PXE install over http to continue once again. But every 5 minutes,  NetworkManager would again delete the IP, requiring me to manually add it once more. Once I stopped the NM service altogether, random IP address deletions no longer occurred. As my PXE server is a CentOS 6.5 VM that uses SysVinit, to stop the NetworkManager service I use:

service NetworkManager stop

but for those of you on distros using systemd you would use

systemctl stop NetworkManager

When adding an IP, I usually just specify addr/CIDR and broadcast address (which has worked just fine for me), but this tutorial from the Archlinux docs recommends also adding routing information.

Postscript 2015.12.04
Nowadays I am working with Openstack at work, and NetworkManager and openstack are incompatible. After the openstack installation completes using scripts from packstack or devstack, you will be warned if the NetworkManager daemon is running. For machines intended as openstack nodes, it is a good idea to just disable NetworkManager (or any other network connection management daemon like wicd, connman, etc.) entirely:

systemctl disable NetworkManager
systemctl stop NetworkManager

Troubleshooting a failure of ntpd.service at system startup

Several months ago, I noticed that journalctl was containing messages about ntpd.service failing. systemctl status ntpd also confirmed that systemd failed to load ntpd. A quick and dirty hack (that doesn't solve the underlying problem) is to just run sudo ntpd -qgd to manually load the ntp daemon (and update system time even if there is a difference of over 1000s between local time and ntp server time). This has the effect of making a one-time change to the system clock after ntpd queries the Network Time Protocol servers defined in /etc/ntp.conf

Today I had some free time so I decided to take a closer look at the problem. I discovered several issues:

1. Manually starting ntpd daemon conflicts with starting systemd ntp.service

I know it sounds like common sense, but at times I seem to lack this resource. This problem is characterized by the following error message in journalctl:
...
unable to bind to wildcard address :: - another process may be running - EXITING

By checking running processes, we can see that, sure enough, ntpd is already running:

[archjun@arch ~]$ ps aux | grep ntp
root      1699  0.0  0.3 105200 14588 ?        SLs  10:47   0:00 ntpd
archjun  28055  0.0  0.0  11908  2276 pts/2    S+   11:00   0:00 grep ntp

So problem #1 was solved by doing a kill -15 on pid 1699 shown above.


2. Create user ntp

Invoking systemctl start ntpd still didn't work, however. journalctl -f (equivalent of tail -f /var/log/messages for non-systemd machines) showed the following error:

Nov 18 11:02:59 arch ntpd[1241]: Cannot find user `ntp'
Nov 18 11:02:59 arch systemd[1]: ntpd.service: main process exited, code=exited, status=255/n/a

That's strange. Despite re-installing the ntp package through pacman, user ntp was not created (checked with cat /etc/passwd |grep ntp), although group ntp was created (verified with cat /etc/group |grep ntp).

I tried to create user ntp with a simple useradd ntp, but my system complained that there was already a group with the same name. I thus added user ntp and added them to group ntp all in the same command:

useradd ntp -g ntp

Now when I run systemctl start ntpd everything looks fine when checked with systemctl status ntpd and in journalctl:

Nov 18 11:07:43 arch systemd[1]: Starting Network Time Service...
Nov 18 11:07:43 arch ntpd[11063]: ntpd 4.2.7p465@1.2483-o Sun Sep  7 07:03:04 UTC 2014 (1): Starting
Nov 18 11:07:43 arch ntpd[11063]: Command line: /usr/bin/ntpd -g -u ntp:ntp
Nov 18 11:07:43 arch systemd[1]: Started Network Time Service.
Nov 18 11:07:43 arch ntpd[11064]: proto: precision = 1.047 usec (-20)
Nov 18 11:07:43 arch ntpd[11064]: Listen and drop on 0 v6wildcard [::]:123
Nov 18 11:07:43 arch ntpd[11064]: Listen and drop on 1 v4wildcard 0.0.0.0:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 2 lo 127.0.0.1:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 3 wlp12s0 192.168.0.9:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 4 lo [::1]:123
Nov 18 11:07:43 arch ntpd[11064]: Listen normally on 5 wlp12s0 [fe80::21f:3cff:fe46:6467%3]:123
Nov 18 11:07:43 arch ntpd[11064]: Listening on routing socket on fd #22 for interface updates

Finally, a helpful thread I referred to from the Archlinux forums:

https://bbs.archlinux.org/viewtopic.php?id=155120

Setting up OpenDaylight Hydrogen VM - Ubuntu 14.04

On Wednesday and Thursday (Nov. 5 & 6) last week, I participated in my first Hackathon -- Cisco Codefest 2014 -- which was held at the POSCO Engineering and Construction Building main hall 4F in Songdo, South Korea. The track my team chose for the competition was OpenDaylight, a Software Defined Networking (SDN) framework led by the Linux Foundation, Redhat, Cisco, and other partners. This framework enables network devices to be virtualized (NFV - Network Function Virtualization) which opens up the possibility of regular PC's with multiple PCI network cards being able to act as cheap switches, for example, among other possibilities.

Although this contest was a coding competition, our team spent a lot of time doing Linux sysadmin work to get the OpenDaylight VM's into a usable state. We also spent a significant chunk of time learning how to use mininet, the network simulator bundled with the OpenDaylight VM, and setting up network visualization (DLUX - Daylight User Interface) provided by Karaf from the ODL Helium container. This post will walk you through the process of customizing the Ubuntu 14.04 VM available for download from the ODL Hydrogen downloads page (you can navigate here from the main OpenDaylight Downloads page).


Virtualbox Settings for Ubuntu 14.04 VM

As of 2014-11-11 the download link for the Ubuntu 14.04 VM appears as odl-test-desktop-3.ova, but once you actually import it into Virtualbox, the name of the VM appears as odl-test-desktop-2, which is the name of the Ubuntu 13.04 VM available at the second download link in the middle. Here's a screenshot of the page:

I thought I had downloaded the wrong VM, but after verifying with uname -a that the guest OS was running kernel 3.13, I was sure I was running Ubuntu 14.04 (instead of 13.04, the version in the other two VM's available for download). Note that all the Ubuntu ODL images are 64-bit, so if you're running a 32-bit version of say, Windows 7 (or Linux, for that matter), you will not be able to run the VM's even if you have Virtualbox installed (my Codefest teammate actually went out and bought a new laptop because his Core i5 with just 2GB RAM just wasn't up to the task of running the heavy Ubuntu 14.04 VM's).

First, after importing the VM you should first tweak some settings for speed. The download page does mention  "disable 3D acceleration otherwise left menu may not show," (the Unity bar on the left-hand margin) but in my case, leaving 3D acceleration on in the VM display settings also caused problems with my host machine display, making me unable to minimize the VM full-screen window. So you definitely want to make sure that 3D acceleration box is not checked in the following window from the Virtualbox Manager:

Second, from the "System" tab above, you should reduce the number of CPU cores allocated to the VM from 4(!) to something more manageable for your system. Likewise with memory; the memory initially allocated to the VM is 4096MB (4 GB), but I found that for simple topologies 2 CPU cores and 2GB works fine.

Third, from the "Network" tab above, make Adapter 1 attached to "Bridged Adapter" and choose the name of your host machine's network interface. Adapter 1 will provide your VM with Internet access from your host and will also give the VM an IP within the same subnet as your host making ssh possible. You should also enable Adapter 2 as a "Host-only" interface that you can use exclusively for a mininet network. If you do not have any host-only interfaces enabled, click "File" -> "Preferences" -> "Network" from the Virtualbox Manager and select the "Host-only Networks" tab. Click the '+' icon to add one interface (which appears as vboxnet0 in Linux hosts).

Fourth, the VM will not be able to go full screen until you intstall the Virtualbox Guest Additions into the guest OS. Refer to this previous post on my blog for instructions on how to do this in a Linux VM. It would be nice if the people at the ODL foundation actually installed the Virtualbox Guest Additions into the Ubuntu 14.04 VM before exporting it as an .ova file...

Setting up the Application Environment

Fifth, seriously consider installing a lighter Desktop Environment (DE) like XFCE or LXDE. Unity with compiz is installed by default as the Ubuntu 14.04 DE, but its 3D bling, compositing, drop shadows, etc. are superfluous to the job of running network simulations for SDN. In fact, once you launch Karaf (from ODL Helium) and mininet you will find that even powerful systems will lag (my teammate's new machine, a brand-new Core i7 laptop with 256GB SSD and 8GB of RAM ran sluggishly with Unity enabled while running mininet et al). If you run top within the VM while using Unity, it is not uncommon to see compiz taking up 20%+ of CPU.

Don't worry -- installing a new DE does not require the removal of Unity and compiz! Once the new DE is installed through apt-get, you can select what DE to use from the login screen. To install either XFCE or LXDE for Ubuntu, enter one of the following:

sudo apt-get install --no-install-recommends xubuntu-desktop

sudo apt-get install --no-install-recommends lubuntu-desktop

It would also be a good idea to turn off automatic updates... this is a VM, after all, so you should only do point updates for packages that really need it.

Sixth, set up Wireshark. Although it is pre-installed in the VM, running it as the regular user mininet will not allow you to listen to network traffic on any of the network interfaces. That doesn't mean you should run Wireshark as root, however (doing so is a security risk). Instead, run dpkg-reconfigure wireshark-common from the CLI. This will enable regular users to listen on network interfaces as long as they are members of the newly-created user group wireshark. To add user mininet to group wireshark:

sudo usermod -a -G wireshark mininet

For changes to take effect, you must log out and log in once again.

Seventh, install Karaf from ODL Helium. From the main ODL download page, click the link for either the zip or tar archive and extract the files into a folder in the ODL Hydrogen VM. Once you have extracted the files, navigate to the folder and enter the bin directory and invoke karaf as follows:

mininet@mininet-vm:~/distribution-karaf-0.2.0-Helium/bin\> ./karaf

karaf will work with any JVM running Java 1.7 or higher. Fortunately, the Ubuntu VM comes with OpenJDK 7 runtime already installed. Once karaf is installed and running, from the karaf command prompt enter the following to install packages necessary for DLUX:

opendaylight-user@root>feature:install odl-restconf odl-l2switch-switch odl-mdsal-apidocs odl-dlux-core

Now when you create a network in mininet and generate traffic between hosts, you will be able to see the topology by opening a browser within the VM and navigating to localhost:8181/dlux/index.html, logging in with admin:admin, and clicking on "Topology" from the left-hand menu within the browser page body. (Note: If you launch mininet with default settings, it will try to create a network controller on localhost:6633, but this conflicts with the controller created by karaf. To avoid this conflict, you need to specify the IP for a remote controller and a port that doesn't conflict with 6633. In our case, we will use port 6653.)


Example of mininet and karaf in action with topology visualization

Open two terminals within the VM and start karaf in one of them. Once karaf is fully loaded (this can take up to 60 seconds), we will launch mininet with a remote controller residing on the host-only IP for our VM, 192.168.56.101:

mininet@mininet-vm:~\> sudo mn --mac --switch=ovsk,protocols=OpenFlow13 --controller=remote,ip=192.168.56.101,port=6653 --topo=tree,2

We have created a tree topology 2 levels deep (switch 1 is the root node, switches 2 and 3 branch from s1) using an open virtual switch with openflow 1.3

Note that we have changed the controller port to 6653 so that it will not conflict with the karaf controller. Within the mininet command environment, we invoked pingall -c 1 (ping all hosts 1 time) to generate network traffic (if there is no traffic, DLUX will not show any flows). Now navigating to localhost:8181 and navigating to Topology should show the following:

Now your Ubuntu 14.04 VM for ODL Hydrogen should be ready for you to experiment with SDN!

Time-lapse recording of meditation sessions with motion, ffmpeg, and bash script

Background

I have been doing insight meditation for the past several months, but have a problem with being consistent and disciplined enough to actually meditate when I feel tired or too busy. I have used Beeminder (here are my automatically tracked goals) to good effect to keep me on track for certain goals that can be automatically monitored, i.e. RescueTime, Fitbit step count, Dual N-Back practice sessions, number of cards reviewed in Mnemosyne (similar to Anki but easier to use), etc.

For more free-form goals that require human oversight, StickK is a good tool. Both Beeminder and StickK are commitment devices that help people actually stick to their goals by making them put money on the line; if you go off track (determined by a human referee in the case of StickK, and by a computer in the case of Beeminder), your credit card gets charged some amount (that exponentially increases in Beeminder).

I track weekly pushups and meditation sessions using StickK. In the case of pushups, I prop up my smartphone so that it will record me doing one set of X pushups. The video is automatically uploaded to G+ from where I share a link to the video with my StickK referee. For 25 pushups, the videos are usually about 60 seconds long. But this will not work for meditation, considering the fact that a meditation session can last anywhere from 10 to 30 minutes. The battery life on many smartphones is pretty terrible, and even if you could take a half-hour video of yourself sitting on a mat meditating, who the hell would watch it? I want to save my StickK referee from such torture as well as save memory card space and battery life on my smartphone. Solution: use Linux!

The Tools

motion

motion is a web-cam utility that begins taking snapshots when motion is detected in front of the camera. It is included in the package repositories of many Linux distros, and I use the version from the Archlinux community repository. motion is commonly used in DIY CCTV projects using the Raspberry Pi, however, I will use it to take time-lapse photos of my meditation sessions. By default, motion will start taking pictures whenever it senses motion, but we need to change this default behavior so that it will take a photo every N seconds. To do this, you need to edit /etc/motion/motion.conf as follows:

First make sure that motion daemon mode is turned off

############################################################
# Daemon
############################################################

# Start in daemon (background) mode and release terminal (default: off)
daemon off

Although the comment above claims the default is off, in Archlinux the default is actually on.

############################################################

# Snapshots (Traditional Periodic Webcam File Output)

############################################################

# Make automated snapshot every N seconds (default: 0 = disabled)

snapshot_interval n

In our case, we set n to 6.

Since we will be taking time-lapse photos, we need to turn off the feature that takes pictures when motion is detected:

############################################################

# Image File Output

############################################################

# Output 'normal' pictures when motion is detected (default: on)

...

# Can be used as preview shot for the corresponding movie.

output_normal off

After installing the motion package in Archlinux, you also have to edit the permissions on /var/run/motion so that it is writable by the regular user. Something like the following should do the trick:

sudo chown username:username /var/run/motion

If your /usr/local directory is not already writable by the regular user, you will also need to recursively change the permissions on this directory as well because motion outputs all its image files to the path /usr/local/apache2/htdocs/cam1 (at least this is true on Archlinux). You can recursively change ownership of directories and subdirectories using the -R option in chown.

There are several other config changes you might want to make to /etc/motion/motion.conf.

If you don't want motion to create partial preview videos every X frames as well as a final video (I think it's a waste of space), you need to edit the following section:

# Use ffmpeg to encode mpeg movies in realtime (default: off)

ffmpeg_cap_new off

ffmpeg

Once you have finished taking time-lapse photos in motion, all the images will be contained in /usr/local/apache2/htdocs/cam1

Now you need to use ffmpeg to render these time-lapse images into a video (without audio). In our video, we want each image to be shown for 1 second each and the video should be 25 frames per second. We can render such a video as follows:

ffmpeg -framerate 1/1 -pattern_type glob -i "*.jpg" -r 25 filename.mp4

Note: -framerate flag ensures that each input image appears for 1 second (to make each image appear for n seconds, set this flag to 1/n)

When using the -i flag (input), if you wish to use wildcards (to render all jpg files in a directory, for example) you need to preface the -i with the -pattern_type flag glob option

The -r flag sets the frame rate for the output video file.

bash script

You don't want to have to remember the ffmpeg invocation above every time you want to record a time-lapse video. Below is a script I made to launch motion, render the images, and then cleanup the output directory once video rendering is complete:

My original script didn't have any error trapping before running rm *.jpg and I bitterly came to regret this when one day ffmpeg exited with an error; despite no .mp4 video being rendered, my silly script deleted all jpg files. The unsafe lines were as follows:
...
ffmpeg -framerate 1/1 -pattern_type glob -i "*.jpg" -r 25 $DATE.mp4
rm *.jpg

The problem with these lines is that even if ffmpeg exits with an error, the script will not stop; the rm command will still be executed! After learning my lesson the hard way, I read up on error handling from William Shotts' wonderful bash tutorial. I also highly recommend his book, The Linux Command Line.