PXE netboot installation differences between Ubuntu and RHEL

PXE netboot is the preferred method for fully automated/unattended installations of Linux to many machines at once. I have written several posts about PXE netboot for Linux, this one (PXE for UEFI and Legacy BIOS) being the most recent. In the RHEL world, fine-grained system settings are defined in Kickstart files. While Ubuntu supports a subset of Kickstart commands, it is better to use D-I (Debian Installer) preseed files to automate PXE netboot installs of Debian/Ubuntu.

Below is a partial list of the differences I have observed in PXE netboot for Ubuntu and RHEL:

Remote monitoring of PXE installation

RHEL supports monitoring remote installs via vnc or vnc reverse-connect, but Ubuntu does not support vnc. Ubuntu does allow remote installation via ssh but only for manual installs; You cannot monitor a fully automated Ubuntu install remotely.

To enable vnc for a PXE install on RHEL 7.0, for example, you would add the following options to the APPEND line in your PXE default config file menu entry (for Legacy BIOS)

LABEL RHEL7.0
  MENU LABEL Boot RHEL 7.0 (ks install)
  KERNEL images/rhel7.0/vmlinuz
  INITRD images/rhel7.0/initrd.img
  APPEND ip=dhcp inst.repo=http://192.168.95.97:8080 inst.vnc ksdevice=link inst.ks=http://192.168.95.97:8000/rhel7.0-ks.cfg

If you want to enable vnc reverse connect you would add something like inst.vncconnect=1.2.3.4:5500 after inst.vnc on the APPEND line.

In the case of Ubuntu you would enable D-I preseed from your PXE default config as follows:

LABEL xenial16.04_netinst-Netconsole
  MENU LABEL Boot Ubuntu 16.04 (manual netinst over ssh)
  KERNEL images/xenial_x64/linux
  INITRD  images/xenial_x64/initrd.gz
  APPEND ip=dhcp auto=true priority=critical locale=en_US.UTF-8 kdb-chooser/method=us netcfg/choose_interface=auto url=http://192.168.95.97:8000/netconsole.cfg

It is important that you specify auto=true for automated install and priority=critical so that D-I will not ask the user for information (except for absolutely critical cases). You can specify the location of the D-I preseed file over http (among other methods) with url=

Then the preseed file must contain instructions to start sshd so that you can make an ssh connection to the Debian Installer and control the installation manually. Here is my D-I preseed file for Ubuntu PXE install over ssh:

# D-I preseed file to enable remote installs of Ubuntu/Debian
# over SSH

#d-i debian-installer/locale string ko_KR.UTF-8
d-i debian-installer/language string en
d-i debian-installer/country string KR
d-i debian-installer/locale string en_US.UTF-8
d-i keyboard-configuration/xkb-keymap select us
d-i debconf/priority                   select critical
d-i auto-install/enabled               boolean true
d-i netcfg/choose_interface            select auto
d-i netcfg/get_hostname                string unassigned-hostname

### Network console
# Use the following settings if you wish to make use of the network-console
# component for remote installation over SSH. This only makes sense if you
# intend to perform the remainder of the installation manually.
d-i anna/choose_modules string network-console
d-i network-console/password           password foofoo
d-i network-console/password-again     password foofoo
d-i preseed/early_command string anna-install network-console


Kickstart syntax support
RHEL and its variants natively support Kickstart syntax for automated installs (although the syntax varies between versions of RHEL; I strongly recommend checking your syntax with ksvalidator) but Ubuntu only supports a subset of Kickstart commands. For example, installing to multiple disks with something like

part /boot --fstype=ext4    --ondisk=sda --size=512
part /     --fstype=ext4    --ondisk=sda --size=1 --grow
part /var  --fstype=ext4    --ondisk=sdb --size=1 --grow

is NOT supported in Ubuntu-flavored kickstart files. You can only install to a single disk when using Ubuntu kickstart syntax. There are many more such gotchas so it's a good idea to stick with the Debian/Ubuntu native automated install solution D-I preseed.


DNS
For all PXE installs, you need a DHCP server to assign IP's to netboot clients. But a fully-automated Ubuntu PXE D-I preseed installation also requires that you have a DNS server running. If you do not have a name server running on the subnet for PXE netboot, D-I will stop and ask you to enter the address of the name server or press ENTER for no name server.

Since I use dnsmasq, I made sure that the following was commented out in /etc/dnsmasq.conf so that DNS would run:

# Listen on this specific port instead of the standard DNS port
# (53). Setting this to zero completely disables DNS function,
# leaving only DHCP and/or TFTP.
#port=5353
#port=0

I am sure there are many more differences, but these are the ones I've encountered so far. If you know of more, please let me know in the comments.

Openstack with SELINUX

Normally before installing Openstack (be it Devstack, RDO, or some other flavor) I set SELINUX to permissive. Yesterday, however, I was at a client IDC and they requested that SELINUX be set to enabled before installing RDO 6 Juno.

This was the first time I had received such a request. I therefore edited /etc/selinux/config and set SELINUX=enforcing and then made selinux relabel the entire filesystem with fixfiles relabel (and said y to deleting all the files in /tmp). After the relabeling, you must reboot the system.

However after enabling SELINUX, the system integration company working on the project complained that they were getting permission denied errors for the kvm kernel module. I was able to get things working again after editing /etc/libvirt/qemu.conf and uncommenting the line:

security_driver = "selinux"

and then restarting libvirtd with systemctl restart libvirtd

In hindsight, I think the client might have run into the following issue when trying to install Redhat Distribution of Openstack (RDO) with SELINUX set to disabled:

PackStack fails if SELinux is disabled

The solution is to enable SELinux in permissive mode (if there is a reason not to have it in enforcing mode).

References:

https://www.centos.org/docs/5/html/5.2/Deployment_Guide/sec-sel-fsrelabel.html

https://www.rdoproject.org/install/selinux/

Google Cloud Platform billing shock (or don't forget to delete networking resources, too!)

I have played with public cloud services on AWS, Digital Ocean and recently, Google Cloud Platform (GCP). GCP is has a user-friendly, clean interface. One thing I really like is that estimated usage charges appear on-screen when you launch a compute instance.

I followed one of the GCP Kubernetes tutorials from the following link:

https://cloud.google.com/container-engine/docs/tutorials

I created a Kubernetes cluster composed of several docker containers and a load balancer to split traffic among containers running a hello world node.js app. Once you are done with the tutorial you should of course delete the resources you were using. For some reason, however, I only deleted the compute resources I used in the tutorial (Kubernetes cluster and constituent containers) but forgot to delete the load balancer. As a result, I was surprised when I got my GCP bill for June 2016:

The item Network Load Balancing Forwarding Rule Minimum Service Charge in APAC indicates that the load balancer ran for 696 hours or 4 weeks and one day! This cost me over $17 USD.

As soon as I finished the Kubernetes tutorial I made sure to delete the Kubernetes cluster, and as a result I only used 0.65 hours of compute resources costing me just 4 cents. Unfortunately, however, I neglected to delete the external load balancer (doh!) ...

The step in which you are supposed to delete the Kubernetes pod and LB are detailed here:

http://kubernetes.io/docs/hellonode/#thats-it-time-to-tear-it-down

That’s it for the demo! So you don’t leave this all running and incur charges, let’s learn how to tear things down.

Delete the Deployment (which also deletes the running pods) and Service (which also deletes your external load balancer):

kubectl delete service,deployment hello-node

I think I missed the load balancer because I deleted resources through the GCP web dashboard instead of from the command line.

To delete networking resources from the dashboard, click the icon at top left that looks like 三 and select Networking.

Then from the Networking screen, select Load Balancing:

If you have any existing load balancers they will appear in this screen and you can delete them. Hopefully other people will be able to learn from my dumb mistake and make sure to delete the networking resources in addition to their compute resources when they are done using GCP.

Problems compiling motion 3.2.12 on Fedora 24

Note: On July 17, 2016 Leigh Scott created a new rpmfusion package motion-3.3.0.trunkREV561-2.fc24.x86_64 for motion webcam which contains a patch for ffmpeg-3. You can also check out the rpmfusion/motion repository on github to see the changes which must be made to ffmpeg.c and configure.in for motion to properly compile.

https://github.com/rpmfusion/motion/blob/master/api-update_libav10.patch
https://github.com/rpmfusion/motion/blob/master/api-update_ffmpeg-2.9.patch

============================================

I recently upgraded from f23 to f24 as soon as the latter was released. There are quite a few packages from the RPM Fusion community repo which are not yet available on F24 that are available on F23.

When compiling motion the first issue I encountered after running

./configure

make

------------------

fatal error: linux/videodev.h: No such file or directory

compilation terminated.

I fixed this problem by installing fedora package libv4l-devel and v4l-utils-devel-tools

Then I created the following symlink:

sudo ln -s /usr/include/libv4l1-videodev.h   /usr/include/linux/videodev.h 

I tried to compile again:

./configure

make clean

make

The next problem, however, is that motion looks for PIX_FMT_YUV420P in ffmpeg's pixfmt.h header file provided by ffmpeg-3.0.2-1.fc24.x86_64 from RPM Fusion but this variable has been renamed to AV_PIX_FMT_YUV420P 

I need to find which files to edit with new variable names in order to compile motion... 

References:

https://wuliangx.wordpress.com/2015/03/31/installing-motion-3-2-12-on-banana-pi/

http://unix.stackexchange.com/questions/165791/installing-kino-fails-with-missing-file-linux-videodev-h

https://github.com/gpac/gpac/issues/285

http://www.lavrsen.dk/foswiki/bin/view/Motion/MotionGuideInstallation#Make_Install

http://www.lavrsen.dk/foswiki/bin/view/Motion/WebHome

irssi 0.8.18+ SASL / SSL authentication config

As of v0.8.18, irssi irc client has native support for SASL (Simple Authentication and Security Layer). Because irssi < 0.8.18 didn't have SASL support, the old workaround was to copy the Perl script cap_sasl.pl to ~/.irssi/scripts/autorun/ . My old irssi config file contained something like the following:

Freenode = {
    type = "IRC";
    max_kicks = "1";
    max_msgs = "4";
    max_whois = "1";
    sasl_mechanism = "plain";
    sasl_username = "archjun";
    sasl_password = "myPlainTextPW";
  };

Hardcoding my Freenode password into the irssi config was dumb because this file managed in my dotfiles repository on github so everyone could see the password. I later realized my mistake and had to remove the file from my git repo, rewrite the git history and remove refs to the deleted file:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch fileName

Beware that the command above will DELETE fileName despite the --cached flag. Before you run this command make sure to backup the original file somewhere.

I decided not to use regular passwords for irc authentication on Freenode and to use SSL passwordless auth instead. There is a great guide for this on the Archlinux wiki:

https://wiki.archlinux.org/index.php/irssi#SSL_Connection

After following the steps now my Freenode entry in ~/.irssi/config looks like this:

...
  {
    address = "chat.freenode.net";
    chatnet = "Freenode";
    port = "6697";
    use_ssl = "yes";
    ssl_cert = "~/.irssi/irssi.pem";
    ssl_pass = "";
    ssl_verify = "yes";
    ssl_capath = "/etc/ssl/certs";
  },
...

*Note 2016-12-13: You should no longer specify ssl_capath when defining servers in the irssi config file. Reference: https://github.com/NixOS/nixpkgs/issues/16651

The above was automatically generated by irssi when I invoked /save after registering my SSL key with Freenode NickServ.

Looking at this syntax, I think it might be possible to replace the plain-text sasl_password in my old irssi config file with something like:

sasl_password = "~/.irssi/mypw.txt"

And then add mypw.txt to .gitignore to avoid accidentally including my password in a public git repo.

Now when I /connect Freenode with SSL cert auth enabled, my nick is authenticated automatically. Very convenient!

Opening Ports for Openstack in Firewalld

Last week I made a post about opening ports in Ubuntu's ufw firewall when using Devstack (Openstack upstream). Today I will show you how to do the same thing in firewalld dynamic firewall which is now the default in RHEL 7+ and Fedora.

The ports to be opened are the same, but you must also enable two additional services in firewalld, namely http and vnc-server. If you don't enable the former, you will be unable to access Horizon web UI, and if you don't enable the latter, you will not be able to see the console through Horizon when you launch an instance on Nova compute.

In the case of ufw, however, http port 80 was opened by default and vnc was enabled by simply opening 6080/tcp.

I wrote a Bash script to open the necessary ports for Openstack in firewalld. I have tested it on Openstack Kilo running on F23. You can find the script at the following link:

https://gitlab.com/gojun077/openstack-conf/blob/67f98aa4b93ab268e386028ec0e764547d0a1bb2/firewalld_openstack_rdo.sh

#!/bin/bash
# firewalld_openstack_rdo.sh
# Created by Jun Go gojun077@gmail.com
# Last Updated 2016-06-07

# Script that will permanently open ports needed by
# Redhat Distribution of Openstack (RDO) in Firewalld

# Tested with Openstack Kilo RDO 7

# This script should be run as root

# DEFAULT FIREWALLD ZONE
DZONE=FedoraServer

#################
#   NETWORK
#   IFACES
#################
EXT0=br-ex
INT0=br-enp5s0

#################
#   TCP PORTS
#################
AMQP=5672
CEILOM=8777
CINDER=8776
GLANCE=9292
GLANCEREG=9191
NEUTRON=9696
NOVNCPROX=6080
#NOVAEC2=8773
#NOVAMETA=8775
#NOVAISCSI=3260
#NOVAREDIS=6379
#NOVAS3=3333
SHEEPDOG=7000

#################
#   UDP PORTS
#################
CEILUDP=4952
OVSNEUTRONVXLAN=4789

TCPPORTS=($AMQP
   $CEILOM
   $CINDER
   $GLANCE
   $GLANCEREG
   $NEUTRON
   $NOVNCPROX
   $SHEEPDOG
  )

UDPPORTS=($CEILUDP
   $OVSNEUTRONVXLAN
   )

# ADD NETWORK IFACES TO DEFAULT ZONE
firewall-cmd --permanent --zone=$DZONE --add-interface=$EXT0
firewall-cmd --permanent --zone=$DZONE --add-interface=$INT0

# ENABLE SERVICES REQ'D FOR OPENSTACK
# Horizon (http)
firewall-cmd --permanent --zone=$DZONE --add-service=http
# vnc-server (for some reason, enabling TCP 6080 is not enough)
firewall-cmd --permanent --zone=$DZONE --add-service=vnc-server

for i in ${TCPPORTS[*]}; do
  firewall-cmd --permanent --zone=$DZONE --add-port="$i"/tcp
done

for j in ${UDPPORTS[*]}; do
  firewall-cmd --permanent --zone=$DZONE --add-port="$j"/udp
done

# Apply permanent rules as the current runtime config
firewall-cmd --reload

# List Default Zone Firewall Info (along with ports & svcs)
firewall-cmd --list-all

Openstack - List of ports which must be opened in the firewall

I must confess that when testing new Openstack releases on a variety of different Linux distros (RHEL, Fedora, Ubuntu) I often disable the firewall in the interests of expediency. While this might be OK for internal testing in a lab environment or while preparing a Proof of Concept (PoC) for a client, this is definitely a bad habit that is unacceptable for a production environment.

I recently did a Devstack install on Ubuntu 15.10 to test upstream Openstack compatibility with the Sheepdog distributed storage backend. This time I left the firewall running (ufw, aka uncomplicated firewall for Ubuntu) and opened the ports necessary for Openstack to run.

Before I present the list of ports which must be opened, note that you can find all these port numbers in the conf files for Glance, Cinder, Nova, Keystone, etc. in /etc/glance/glance-api.conf, /etc/nova/nova.conf, /etc/cinder/cinder.conf, and so on.

In the conf files, many ports will be commented out. For example, it is possible to connect Nova Compute with Amazon EC2 so it can launch instances from AWS. To do so, you would have to open TCP port 8773 in your firewall on your compute node, but this is commented out by default in nova.conf.

Here is the list of ports I have compiled. All ports are TCP unless specified otherwise:

AMQP/RabbitMQ: 5672 (5671 if rabbitmq uses SSL auth)
Ceilometer: 8777
Ceilometer: udp_port=4952
Cinder: sheepdog_store_port=7000
Cinder: 8776
Glance: 9292
Glance glance-api.conf: registry_port=9191
Neutron: 9696
Nova novncproxy: 6080
Nova ec2_port: 8773 (commented out by default)
Nova metadata: 8775 (commented out by default)
Nova iSCSI target: 3260 (commented out by default)
Nova nova.virt.xenapi.image.bittorrent: 6881~6891 (commented out by default)
Nova redis host: 6379 (commented out by default)
Neutron ovs_neutron_plugin.ini: vxlan_udp_port=4789
Nova s3_port=3333 (commented out by default)
...
(there are more, but those ports are optional)

I have written a simple bash script that can be used to open the necessary ports in ufw. You can find the script at the following URL:

https://gitlab.com/gojun077/openstack-conf/blob/master/ufw_openstack.sh

#!/bin/bash
# ufw_openstack.sh
# Created by Jun Go gojun077@gmail.com
# Last Updated 2016-05-25

# Script that will open ports needed by Openstack in
# UFW Firewall

# This script should be run as root

#################
#   TCP PORTS
#################
AMQP=5672
CEILOM=8777
CINDER=8776
GLANCE=9292
GLANCEREG=9191
NEUTRON=9696
NOVNCPROX=6080
#NOVAEC2=8773
#NOVAMETA=8775
#NOVAISCSI=3260
#NOVAREDIS=6379
#NOVAS3=3333
SHEEPDOG=7000

#################
#   UDP PORTS
#################
CEILUDP=4952
OVSNEUTRONVXLAN=4789

TCPPORTS=($AMQP
   $CEILOM
   $CINDER
   $GLANCE
   $GLANCEREG
   $NEUTRON
   $NOVNCPROX
   $SHEEPDOG
  )

UDPPORTS=($CEILUDP
   $OVSNEUTRONVXLAN
   )

for i in ${TCPPORTS[*]}; do
  ufw allow "$i"/tcp
done

for j in ${UDPPORTS[*]}; do
  ufw allow "$j"/udp
done

# List Open Ports
ufw status