2014년 9월 19일 금요일

RHEL/CentOS PXE install using dnsmasq and darkhttpd (instead of dhcp + httpd + xinetd + tftp-server)

In my new job, one of my responsibilities is going to data centers and installing Linux on many machines at once. Using physical media like DVDs or LiveUSBs one at a time on each machine would take too long, so system engineers often use network boot, aka PXE to send installation images over the network to several machines at once.

The Old Way
Most of the online tutorials (this one, for example) for installing Red Hat Enterprise Linux or CentOS through PXE specify the following steps:

1. Install required packages
    a. dhcp server package

    b. httpd/apache

    c. xinetd

    d. syslinux

    e. tftp-server

2. Disconnect router from Internet (active DHCP processes conflict with dhcpd
    assigning IP's to PXE clients; The PXE server will be connected to a
    router with DHCP turned off or a simple hub. The PXE server will run its own DHCP server
    for assigning IP's to PXE clients)

3. Edit/Create config files
    a. edit /etc/sysconfig/dhcpd

    b. edit /etc/xinetd.d/tftp

    c. edit /etc/dhcp/dhcpd.conf (this step is quite involved)

    d. create apache conf file for PXE /etc/httpd/conf.d/pxeboot.conf

4. Setup PXE
    a. copy PXE boot images (menu.c32, memdisk, etc.) from syslinux directory (in Archlinux, this directory is /usr/lib/syslinux/bios/) to /var/lib/tftpboot

    b. create a mount point for the installation .iso under /var/lib/tftpboot and
        mount the image

    c. create a config file for the PXE server under /var/lib/tftpboot... ie
        something like /var/lib/tftpboot/pxelinux.cfg/default
        (where default is the config file)

5. Start/Restart Services
    a. restart xinetd

    b. restart httpd

    c. restart dhcpd

6. PXE Client setup
    a. Physical machine: enter BIOS and give network/PXE boot first priority

    b. VM: Enable network booting in the VM Manager, change the network interface to
                bridge with the wired interface (i.e. eth0 or enp1s0 et al)

7. Installation


In the above method, there are a lot of config files (for dhcp, tftp, xinet, pxe, apache) that must be manually tweaked to set up PXE booting. This opens up room for error and can be very frustrating when you don't know exactly where you have gone wrong among the many cookbook steps above. Thankfully, there is a better way to do things.


The Simpler Way
I'm a big fan of Archlinux, and the Arch tutorial on using PXE for installing the Archlinux iso was my inspiration for using dnsmasq and darkhttpd to install RHEL/CentOS iso images with PXE. Unfortunately, you cannot blindly follow the instructions from the fine Arch tutorial when installing RHEL.

And why not? In the case of Archiso, PXE images are included within the .iso itself under /arch/boot/syslinux/ but in the case of RHEL/CentOS iso images, although there is a pxe subdirectory at /images/pxeboot/, it only contains Linux kernel images (initrd.img, vmlinuz), not the PXE images necessary for network booting.

Not to fear, we can still use dnsmasq and dispense with configuring xinetd, dhcp, tftp, etc. separately.

1. Install required packages
    a. dnsmasq (dns, dhcp, and tftp server all-in-one)

    b. darkhttpd (http server - much lighter and simpler to set up than httpd)

    c. syslinux (for boot images used in pxe)

2. Disconnect router from Internet (active DHCP processes conflict with dhcpd
    assigning IP's to PXE clients; The PXE server machine will be connected to a
    router with DHCP turned off or a simple hub. dnsmasq will act as the DHCP server
    for assigning IP's to PXE clients)

3. Setup PXE
    a. copy all files from /usr/lib/syslinux/bios to a local directory
        that will be used as the tftp-root (for the purposes of this tutorial
        we will call this directory /usr/local/tftpboot/pxe Make sure this directory
        is writable by your regular user)

    b. create a mountpoint for the installation .iso - the mountpoint doesn't have to be
        located under the ../tftpboot directory as in the previous method. In this tutorial
        the iso mountpoint will be /mnt/distroIso

    c. From the mounted RHEL/CentOS installation .iso, navigate to /images/pxeboot
        and copy the Linux kernel images vmlinuz and initrd.img to a sub-
        directory of /usr/local/tftpboot/pxe, something like
        /usr/local/tftpboot/pxe/images/centos7_64

    d. create a config file for the PXE server under /usr/local/tftpboot/pxe
        Try to make it something distro-specific, i.e. centos7_64_vnc.cfg or
        rhel6dot4_64_vnc.cfg

4. Edit dnsmasq.conf
    a. edit /etc/dnsmasq.conf

5. Start/Restart Services
    a. start dnsmasq: systemctl start dnsmasq (systemd syntax)

    b. start darkhttpd as root: sudo darkhttpd /usr/local/tftpboot/pxe --no-keepalive
        (this will share the pxe images on port 80 of the PXE server's IP address)

    c. start darkhttpd as user: darkhttpd /mnt/path-to-iso-mountpoint/
        (this will share the mounted iso on port 8080 of the PXE server's IP address)

6. PXE Client setup
    a. Physical machine: enter BIOS and set network/PXE boot order to first place

    b. VM: Enable network booting in the VM Manager, change the network interface to
                bridge with wired interface (i.e. eth0 or enp1s0)

7. Installation

The astute reader will notice that instead of having to edit 5 config files (two for dhcp, and one each for xinet.d/tftp, apache, and the pxe server) the new method only requires 2 config files to be edited!

This saves time and avoids the chance of typing errors. The rest of this tutorial will give a detailed walk-through of how to create the distro-specific pxe server config file in step 3d above as well as what settings need to be made in step 4 when editing /etc/dnsmasq.conf


Creating a distro-specific config file for the PXE server

For the purposes of this tutorial we will assume the following:

  • we are installing CentOS7 64-bit from the installation DVD iso image
  • the CentOS7 iso image has been mounted at /mnt/distroIso
  • the tftpboot directory is /usr/local/tftpboot/pxe and is writable by the regular user
  • the Linux kernel images from the iso are located at /usr/local/tftpboot/pxe/images/centos7_64
  • our PXE server will use the IP 192.168.10.100 (this can be manually set through ifconfig or ip addr add...)
Here is a sample pxe server config file for installing CentOS7 over the network using http.


The pxe images needed for network boot in /usr/local/tftpboot/pxe are shared over http on port 80 by darkhttpd (run as root), while the iso installation files are shared on port 8080 by a second instance of darkhttpd (run as regular user) specified in the config file above after inst.repo=

*Note that for RHEL7/CentOS7 inst.repo= or method= has been deprecated in favor of just repo= (But avoid using the syntax repo= in Kickstart files for RHEL 5.X, as this will cause your KS installation to fail)

The config file above will enable a simple pxe install, but when installing on multiple machines at once, it would be helpful connect to each installation instance through vncviewer so we can manage several installs at once. Below are pxe server config files that enable these options; the first enables manually connecting to each instance with VNC, while the second enables Kickstart automated install and VNC reverse connect so you don't have to manually connect to each installation instance with vncviewer 123.456.789:1


*Note: To get vnc reverse connect to automatically connect to vncviewer, you must launch the viewer in a separate terminal with the flag listen:

vncviewer -listen

and then reverse connections should automatically appear as they come in. However, when I tried this on my Archlinux machine, only the first connection opened automatically; all others I had to connect to manually following the IP given by the client terminal. I wonder if it isn't related to this Redhat bug report.


Postscript 2014-10-8: tigervnc up to version 1.1.0-8 from the RHEL/CentOS 6.X repos allows vncviewer in listen mode to connect to multiple vnc reverse connections, but versions higher than this (for example, tigervnc 1.2.80 and above from RHEL/CentOS 7 and other distros) allow vncviewer in listen mode to connect to only one incoming connection at a time! This makes automatic pxe installs of multiple machines much more cumbersome, as we must manually connect to each installation instance with vncviewer 123.456.789:1 My current workaround is use a VM running CentOS 6.5 as my PXE server for installation jobs, as there is no limit on the number of vnc reverse connections that can be used in tigervnc 1.1

Slight changes to pxe server config files can also enable pxe network installation over ftp or nfs instead of http. I will cover these variations in subsequent posts.

Settings for /etc/dnsmasq.conf

Here is the link to a sample dnsmasq.conf configured for the pxe server config file centos7_64_vnc.cfg which appears above.

The entries which should be uncommented and defined are as follows:
    - port=0 (disable DNS; but DNS must be enabled for auto installs of Ubuntu)
    - inferface=eth0 (or enp1s0 etc, YMMV)
    - bind-interfaces
    - dhcp-range=(range of ip addresses to assign to clients)
    - dhcp-option-force=209,path/to/pxe_server_cfg_file
      (relative path from tftp-root)
    - dhcp-boot=lpxelinux.0 (or gpxelinux.0 or pxelinux.0, depending on your
      version of syslinux)
    - enable-tftp (enable dnsmasq's built-in tftp server)
    - tftp-root=/path/to/pxe/boot/images (in our case
   /usr/local/tftpboot/pxe)

Questions and comments are welcome, especially if you have a better way of doing things!

Postscript 2016-02-06
This old post only covers Legacy BIOS PXE netboot with dnsmasq. Please refer to a newer post from 2016 that explains how to setup dnsmasq for UEFI PXE netboot:

http://eatpeppershothot.blogspot.kr/2016/02/uefi-and-legacy-bios-pxe-netboot.html