2014년 9월 19일 금요일

RHEL/CentOS PXE install using dnsmasq and darkhttpd (instead of dhcp + httpd + xinetd + tftp-server)

In my new job, one of my responsibilities is going to data centers and installing Linux on many machines at once. Using physical media like DVDs or LiveUSBs one at a time on each machine would take too long, so system engineers often use network boot, aka PXE to send installation images over the network to several machines at once.

The Old Way
Most of the online tutorials (this one, for example) for installing Red Hat Enterprise Linux or CentOS through PXE specify the following steps:

1. Install required packages
    a. dhcp server package

    b. httpd/apache

    c. xinetd

    d. syslinux

    e. tftp-server

2. Disconnect router from Internet (active DHCP processes conflict with dhcpd
    assigning IP's to PXE clients; The PXE server will be connected to a
    router with DHCP turned off or a simple hub. The PXE server will run its own DHCP server
    for assigning IP's to PXE clients)

3. Edit/Create config files
    a. edit /etc/sysconfig/dhcpd

    b. edit /etc/xinetd.d/tftp

    c. edit /etc/dhcp/dhcpd.conf (this step is quite involved)

    d. create apache conf file for PXE /etc/httpd/conf.d/pxeboot.conf

4. Setup PXE
    a. copy PXE boot images (menu.c32, memdisk, etc.) from syslinux directory (in Archlinux, this directory is /usr/lib/syslinux/bios/) to /var/lib/tftpboot

    b. create a mount point for the installation .iso under /var/lib/tftpboot and
        mount the image

    c. create a config file for the PXE server under /var/lib/tftpboot... ie
        something like /var/lib/tftpboot/pxelinux.cfg/default
        (where default is the config file)

5. Start/Restart Services
    a. restart xinetd

    b. restart httpd

    c. restart dhcpd

6. PXE Client setup
    a. Physical machine: enter BIOS and give network/PXE boot first priority

    b. VM: Enable network booting in the VM Manager, change the network interface to
                bridge with the wired interface (i.e. eth0 or enp1s0 et al)

7. Installation


In the above method, there are a lot of config files (for dhcp, tftp, xinet, pxe, apache) that must be manually tweaked to set up PXE booting. This opens up room for error and can be very frustrating when you don't know exactly where you have gone wrong among the many cookbook steps above. Thankfully, there is a better way to do things.


The Simpler Way
I'm a big fan of Archlinux, and the Arch tutorial on using PXE for installing the Archlinux iso was my inspiration for using dnsmasq and darkhttpd to install RHEL/CentOS iso images with PXE. Unfortunately, you cannot blindly follow the instructions from the fine Arch tutorial when installing RHEL.

And why not? In the case of Archiso, PXE images are included within the .iso itself under /arch/boot/syslinux/ but in the case of RHEL/CentOS iso images, although there is a pxe subdirectory at /images/pxeboot/, it only contains Linux kernel images (initrd.img, vmlinuz), not the PXE images necessary for network booting.

Not to fear, we can still use dnsmasq and dispense with configuring xinetd, dhcp, tftp, etc. separately.

1. Install required packages
    a. dnsmasq (dns, dhcp, and tftp server all-in-one)

    b. darkhttpd (http server - much lighter and simpler to set up than httpd)

    c. syslinux (for boot images used in pxe)

2. Disconnect router from Internet (active DHCP processes conflict with dhcpd
    assigning IP's to PXE clients; The PXE server machine will be connected to a
    router with DHCP turned off or a simple hub. dnsmasq will act as the DHCP server
    for assigning IP's to PXE clients)

3. Setup PXE
    a. copy all files from /usr/lib/syslinux/bios to a local directory
        that will be used as the tftp-root (for the purposes of this tutorial
        we will call this directory /usr/local/tftpboot/pxe Make sure this directory
        is writable by your regular user)

    b. create a mountpoint for the installation .iso - the mountpoint doesn't have to be
        located under the ../tftpboot directory as in the previous method. In this tutorial
        the iso mountpoint will be /mnt/distroIso

    c. From the mounted RHEL/CentOS installation .iso, navigate to /images/pxeboot
        and copy the Linux kernel images vmlinuz and initrd.img to a sub-
        directory of /usr/local/tftpboot/pxe, something like
        /usr/local/tftpboot/pxe/images/centos7_64

    d. create a config file for the PXE server under /usr/local/tftpboot/pxe
        Try to make it something distro-specific, i.e. centos7_64_vnc.cfg or
        rhel6dot4_64_vnc.cfg

4. Edit dnsmasq.conf
    a. edit /etc/dnsmasq.conf

5. Start/Restart Services
    a. start dnsmasq: systemctl start dnsmasq (systemd syntax)

    b. start darkhttpd as root: sudo darkhttpd /usr/local/tftpboot/pxe --no-keepalive
        (this will share the pxe images on port 80 of the PXE server's IP address)

    c. start darkhttpd as user: darkhttpd /mnt/path-to-iso-mountpoint/
        (this will share the mounted iso on port 8080 of the PXE server's IP address)

6. PXE Client setup
    a. Physical machine: enter BIOS and set network/PXE boot order to first place

    b. VM: Enable network booting in the VM Manager, change the network interface to
                bridge with wired interface (i.e. eth0 or enp1s0)

7. Installation

The astute reader will notice that instead of having to edit 5 config files (two for dhcp, and one each for xinet.d/tftp, apache, and the pxe server) the new method only requires 2 config files to be edited!

This saves time and avoids the chance of typing errors. The rest of this tutorial will give a detailed walk-through of how to create the distro-specific pxe server config file in step 3d above as well as what settings need to be made in step 4 when editing /etc/dnsmasq.conf


Creating a distro-specific config file for the PXE server

For the purposes of this tutorial we will assume the following:

  • we are installing CentOS7 64-bit from the installation DVD iso image
  • the CentOS7 iso image has been mounted at /mnt/distroIso
  • the tftpboot directory is /usr/local/tftpboot/pxe and is writable by the regular user
  • the Linux kernel images from the iso are located at /usr/local/tftpboot/pxe/images/centos7_64
  • our PXE server will use the IP 192.168.10.100 (this can be manually set through ifconfig or ip addr add...)
Here is a sample pxe server config file for installing CentOS7 over the network using http.


The pxe images needed for network boot in /usr/local/tftpboot/pxe are shared over http on port 80 by darkhttpd (run as root), while the iso installation files are shared on port 8080 by a second instance of darkhttpd (run as regular user) specified in the config file above after inst.repo=

*Note that for RHEL7/CentOS7 inst.repo= or method= has been deprecated in favor of just repo= (But avoid using the syntax repo= in Kickstart files for RHEL 5.X, as this will cause your KS installation to fail)

The config file above will enable a simple pxe install, but when installing on multiple machines at once, it would be helpful connect to each installation instance through vncviewer so we can manage several installs at once. Below are pxe server config files that enable these options; the first enables manually connecting to each instance with VNC, while the second enables Kickstart automated install and VNC reverse connect so you don't have to manually connect to each installation instance with vncviewer 123.456.789:1


*Note: To get vnc reverse connect to automatically connect to vncviewer, you must launch the viewer in a separate terminal with the flag listen:

vncviewer -listen

and then reverse connections should automatically appear as they come in. However, when I tried this on my Archlinux machine, only the first connection opened automatically; all others I had to connect to manually following the IP given by the client terminal. I wonder if it isn't related to this Redhat bug report.


Postscript 2014-10-8: tigervnc up to version 1.1.0-8 from the RHEL/CentOS 6.X repos allows vncviewer in listen mode to connect to multiple vnc reverse connections, but versions higher than this (for example, tigervnc 1.2.80 and above from RHEL/CentOS 7 and other distros) allow vncviewer in listen mode to connect to only one incoming connection at a time! This makes automatic pxe installs of multiple machines much more cumbersome, as we must manually connect to each installation instance with vncviewer 123.456.789:1 My current workaround is use a VM running CentOS 6.5 as my PXE server for installation jobs, as there is no limit on the number of vnc reverse connections that can be used in tigervnc 1.1

Slight changes to pxe server config files can also enable pxe network installation over ftp or nfs instead of http. I will cover these variations in subsequent posts.

Settings for /etc/dnsmasq.conf

Here is the link to a sample dnsmasq.conf configured for the pxe server config file centos7_64_vnc.cfg which appears above.

The entries which should be uncommented and defined are as follows:
    - port=0 (disable DNS; but DNS must be enabled for auto installs of Ubuntu)
    - inferface=eth0 (or enp1s0 etc, YMMV)
    - bind-interfaces
    - dhcp-range=(range of ip addresses to assign to clients)
    - dhcp-option-force=209,path/to/pxe_server_cfg_file
      (relative path from tftp-root)
    - dhcp-boot=lpxelinux.0 (or gpxelinux.0 or pxelinux.0, depending on your
      version of syslinux)
    - enable-tftp (enable dnsmasq's built-in tftp server)
    - tftp-root=/path/to/pxe/boot/images (in our case
   /usr/local/tftpboot/pxe)

Questions and comments are welcome, especially if you have a better way of doing things!

Postscript 2016-02-06
This old post only covers Legacy BIOS PXE netboot with dnsmasq. Please refer to a newer post from 2016 that explains how to setup dnsmasq for UEFI PXE netboot:

http://eatpeppershothot.blogspot.kr/2016/02/uefi-and-legacy-bios-pxe-netboot.html

2014년 9월 18일 목요일

Bash solution for ShortCutFoo binary email puzzle

In a previous post last week, I introduced a Python solution for converting an email in binary to ASCII. In my day job, I'm a Linux Engineer, so I should be able to write a Bash script to do the same thing. Here is what I came up with:


There are two different versions that parse a LF (linefeed) delimited textfile containing all the 8-character binary strings from the original email. To convert from space-delimited to LF-delimited text, I simply pasted the text into an Emacs buffer and ran M-x replace-text and replaced [SPACE] with [RETURN] (which has to be entered in escaped form as Ctrl-q Ctrl-j in Emacs). bintext.txt is available here.

The first version is a bit more roundabout than the second because it converts from binary to decimal to hexadecimal to ASCII (4 steps).

The second version converts from binary to octal to ASCII (3 steps).

2014년 9월 15일 월요일

PXE network booting doesn't work in Virtualbox 4.3.16 [SOLVED]

I have been using Virtualbox 4.3.14 in Archlinux to test PXE installation using .iso images for CentOS7 and RHEL6.4

Everything works well in Vbox 4.3.14, but in 4.3.16 after the pxe image is loaded and the pxe menu appears, the vmlinuz kernel image and initrd start loading after the user presses ENTER. Installation proceeds but then Virtualbox suddenly exits the VM and goes back to the Vbox manager screen. The VM has the status 'aborted'.

I searched the Virtualbox Bugtracker for similar issues, but didn't find anything related to pxe in version 4.3.16.

Searching /var/cache/pacman/pkg luckily I found the old versions of 4.3.14 so I was able to downgrade with pacman -U pkgname

In the Vbox manager when I right-click on the VM and select "show log", I can't find any error messages that hint at why the pxe install failed.

The relevant part of VM log starting from LAN boot (PXE):

00:00:03.258185 Guest Log: BIOS: Booting from LAN...
00:00:03.266122 Display::handleDisplayResize(): uScreenId = 0, pvVRAM=0000000000000000 w=720 h=400 bpp=0 cbLine=0x0, flags=0x1
00:00:03.266135 UIFrameBuffer::RequestResize: Screen=0, Format=0, BitsPerPixel=0, BytesPerLine=0, Size=720x400, Sending to async-handler..
00:00:03.266187 UIFrameBufferQImage::resizeEvent: Format=0, BitsPerPixel=0, BytesPerLine=0, Size=720x400
00:00:03.266202 UIFrameBufferQImage::resizeEvent: Resizing to FALLBACK buffer due to format is invalid..
00:00:22.342112 Guest Log: BIOS: KBD: unsupported int 16h function 03
00:00:22.342514 Guest Log: BIOS: AX=0305 BX=0000 CX=0000 DX=0000 
00:00:22.345938 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=81
00:00:22.346524 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=81
00:00:22.347124 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=82
00:00:22.347686 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=82
00:00:22.348285 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=83
00:00:22.348895 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=83
00:00:22.349495 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=84
00:00:22.350077 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=84
00:00:22.350689 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=85
00:00:22.351249 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=85
00:00:22.352083 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=86
00:00:22.352649 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=86
00:00:22.353258 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=87
00:00:22.353821 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=87
00:00:22.354422 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=88
00:00:22.354986 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=88
00:00:22.355595 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=89
00:00:22.356167 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=89
00:00:22.356782 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8a
00:00:22.357349 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8a
00:00:22.357950 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8b
00:00:22.358532 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8b
00:00:22.359132 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8c
00:00:22.359694 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8c
00:00:22.360318 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8d
00:00:22.360880 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8d
00:00:22.361478 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8e
00:00:22.362122 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8e
00:00:22.362765 Guest Log: int13_harddisk_ext: function 41, unmapped device for ELDL=8f
00:00:22.363332 Guest Log: int13_harddisk: function 02, unmapped device for ELDL=8f
00:00:23.009453 PIT: mode=2 count=0x4a9 (1193) - 1000.15 Hz (ch=0)
00:00:23.671500 PIT: mode=0 count=0x10000 (65536) - 18.20 Hz (ch=0)
00:00:30.647365 EHCI: Hardware reset
00:00:30.649658 EHCI: Hardware reset
00:00:30.649717 EHCI: USB Operational

The last entry is that USB is operational, there are no error messages...For the time being I'm just going to use Virtualbox 4.3.14 and file a new bug report. Has anyone else had similar problems with 4.3.16?

Postscript 2014-10-8
In Virtualbox 4.3.16 r95972 on CentOS 7 (in Archlinux I believe this is 4.3.16-4) PXE network booting within VM's works just fine.

I suspect that the problems I had with Virtualbox 4.3.16 in Archlinux were due to some configuration problems with virtualbox dkms kernel modules.

2014년 9월 5일 금요일

Quick Solution to a binary-to-ASCII puzzle in a ShortCutFoo Marketing Email

This morning I received the following email in my inbox:

This Week in Bits!

"All the news that's fit to 01110000 01110010 01101001 01101110 01110100"
01010111 01100101 01101100 01101100 00101100 00100000 01001001 00100000 01100100 01101111 01101110 00100111 01110100 00100000 01101011 01101110 01101111 01110111 00100000 01110111 01101000 01111001 00100000 01110111 01100101 00100000 01110100 01101000 01101111 01110101 01100111 01101000 01110100 00100000 01110100 01101000 01101001 01110011 00100000 01110111 01100001 01110011 00100000 01100001 00100000 01100111 01101111 01101111 01100100 00100000 01101001 01100100 01100101 01100001 00101110 00100000 00100000 01001111 01110010 00100000 01110111 01101000 01101111 00100000 01110111 01101111 01110101 01101100 01100100 00100000 01110010 01100101 01100001 01100100 00100000 01101001 01110100 00101110 00100000 00100000 01000010 01110101 01110100 00100000 01111001 01101111 01110101 00100000 01100100 01101001 01100100 00101110 00100000 00100000 01000010 01100101 01100011 01100001 01110101 01110011 01100101 00100000 01111001 01101111 01110101 00100111 01110010 01100101 00100000 01100001 01110111 01100101 01110011 01101111 01101101 01100101 00101110 00100000 00100000 01001001 00100111 01100100 00100000 01110100 01101111 01110100 01100001 01101100 01101100 01111001 00100000 01100010 01110101 01111001 00100000 01111001 01101111 01110101 00100000 01100001 00100000 01100010 01100101 01100101 01110010 00100000 01101001 01100110 00100000 01111001 01101111 01110101 00100000 01110111 01100101 01110010 01100101 00100000 01101000 01100101 01110010 01100101 00101110 00100000 01000001 01110011 01110011 01110101 01101101 01101001 01101110 01100111 00100000 01110100 01101000 01100001 01110100 00100000 01100010 01100101 01100101 01110010 00100000 01110111 01100001 01110011 00100000 01100001 01110100 00100000 01100001 00100000 01110010 01100101 01100001 01110011 01101111 01101110 01100001 01100010 01101100 01100101 00100000 01110000 01110010 01101001 01100011 01100101 00101110 00100000 01010100 01101000 01100101 01110010 01100101 00100111 01110011 00100000 01100001 00100000 01110000 01101100 01100001 01100011 01100101 00100000 01101001 01101110 00100000 01101101 01111001 00100000 01101110 01100101 01101001 01100111 01101000 01100010 01101111 01110010 01101000 01101111 01101111 01100100 00100000 01110100 01101000 01100001 01110100 00100000 01110011 01100101 01101100 01101100 01110011 00100000 01101110 01101111 01110010 01101101 01100001 01101100 00100000 01110011 01101001 01111010 01100101 01100100 00100000 01100010 01100101 01100101 01110010 01110011 00100000 01100110 01101111 01110010 00100000 00110001 00110010 00100000 01100100 01101111 01101100 01101100 01100001 01110010 00101110 00100000 00100000 00110001 00110010 00100000 01100100 01101111 01101100 01101100 01100001 01110010 01110011 00100001 00100000 00100000 01001001 01110100 00100111 01110011 00100000 01101110 01110101 01110100 01110011 00100001 00100000 00100000 01000001 01101110 01111001 01110111 01100001 01111001 00101110 00100000 00101110 00100000 00101110

Python has nice built-in functions for dealing with numbers in various bases. For example, bin() will convert a decimal number into a binary representation prefixed with 0b. Although ASCII was originally a 7-bit format, nowadays the leftmost or 8th bit (in the \(2^8\) position) is customarily set to 0, so all our strings of binary numbers will have a total of eight characters - the first character from the left is 0, while the remaining 7 characters represent 7-bit binary.

I first wrote a quick-and-dirty version to see what the message above was saying. Then I sat down and wrote a more formal version encapsulating code into the functions cvBinToASCII() and cvASCIItoBin() complete with doctests.

Sending a marketing email in binary is a great idea on ShortCutFoo's part for selling services to coders and those interested in programming!

2014년 9월 2일 화요일

My new job: Linux System Engineer 리눅스 시스템 엔지니어로 다시 태어나다!

I never would have imagined 5 years ago when I first starting tinkering with Linux as a hobby that I would end up working as a Linux Engineer in IDC's (Internet Data Centers). Over most of the past 15 years, I have been a free agent wearing the different hats of college prep tutor / college admissions counselor for Korean kids hoping to study abroad in the U.S., as well as doing freelance translation and interpretation work.

At some point during the past couple of years, I came to the realization that I actually enjoyed experimenting with Linux and programming more than I enjoyed the work I did for a living. Thanks to a basic level of Linux know-how gained from troubleshooting my own desktop Linux installations, an IT service company in Teheran-ro, Korea's "Silicon Valley" has decided to take me on as a junior system engineer.

On my first day on the job I accompanied a senior engineer on a service call to an IDC run by one of the big 3 telecom firms in Korea. My only Linux experience thus far was with consumer hardware, but I was greeted by this:


This is a terminal for an HP Proliant DL580 4U rackmounted server with 256 GB of RAM connected to a SAN (Storage Area Network) consisting of dozens of terabytes. You can see from the screen that the server is running RHEL 5, which we upgraded to a more recent 6.X version. Two more pictures:



I had never used the command ethtool -p before, which causes the PCI network interface light to blink so you can identify which cable is attached to which network interface, i.e. eth0 ~ eth15

I also had never seen PXEBOOT to make a new installation before, so today was a learning experience. I have so much to catch up on over the weeks ahead... I am currently preparing for my LFCS exam (similar to the RHCSA, but allows a choice of CentOS, OpenSUSE, or Ubuntu). While I'm in my training period I have to be careful not to blow up any servers through careless mistakes; racks full of high-end servers are worth more than my entire annual pay!

But it's pretty cool to realize that the entire South Korean mobile phone network depends on Linux to operate!