레이블이 darkhttpd인 게시물을 표시합니다. 모든 게시물 표시
레이블이 darkhttpd인 게시물을 표시합니다. 모든 게시물 표시

2015년 10월 18일 일요일

Transferring files without ssh: netcat, darkhttpd, and Python

ssh is indispensable when working on remote machines, but to my surprise (and frustration) many of the big telecoms in Korea have started disabling sshd on most machines due to security audit recommendations. This is ridiculous when you consider that the audits don't flag the rampant use of telnet (which sends all traffic in cleartext) for managing machines on the internal network. At some sites that disable sshd, sysadmins are using old-fashioned ftp in place of sftp or vsftpd!

To do my work, whether it's applying patches or setting up Apache, at a minimum I need to be able to transfer files between machines. When you aren't given access to ssh (which also means no scp or vsftpd), oftentimes netcat (nc), darkhttpd or Python's built-in webservers will do nicely for file transfers.

Netcat

Lots of old-school sysadmins are familiar with using netcat to transfer files between machines, and there are many tutorials on the Internet. Here is an example of using netcat (both GNU netcat/nc and BSD nc will work with each other) to transfer a file to a machine running firewalld dynamic firewall.

By default, firewalld will keep all ports closed except those necessary for web browsing (port 80 http or 443 https) and certain user-defined services like nfs, ssh, etc. The remote machine in this example is on my local network and has sshd and rpcbind (for NFS) running. firewalld will ignore a regular ping scan from nmap, but if we run nmap -Pn hostname, we can see a list of open ports (-Pn Treat all hosts as online -- skip host discovery).

[archjun@latitude630 playground]$ nmap -Pn 192.168.10.57

Starting Nmap 6.47 ( http://nmap.org ) at 2015-10-16 23:07 KST
Nmap scan report for 192.168.10.57
Host is up (0.85s latency).
Not shown: 996 filtered ports
PORT     STATE  SERVICE
22/tcp   open   ssh
111/tcp  open   rpcbind
873/tcp  closed rsync
2049/tcp open   nfs

Before transferring files to a remote machine using netcat, first I need to temporarily open a port for netcat to use. Let's use tcp port 4444:

[archjun@d257 playground]$ sudo firewall-cmd --zone=internal --add-port=4444/tcp
[sudo] password for archjun: 
success

btw, firewalld has the concept of zones which have different security policies. Network interfaces can be placed The internal zone applies to the local network only. The available zones are:

[archjun@d257 playground]$ firewall-cmd --get-zones
block dmz drop external home internal public trusted work

Running nmap from latitude630  on the remote machine d257 now shows tcp port 4444:

[archjun@latitude630 playground]$ nmap -Pn 192.168.10.57

Starting Nmap 6.47 ( http://nmap.org ) at 2015-10-16 23:21 KST
Nmap scan report for 192.168.10.57
Host is up (0.61s latency).
Not shown: 994 filtered ports
PORT     STATE  SERVICE
22/tcp   open   ssh
111/tcp  open   rpcbind
873/tcp  closed rsync
2049/tcp open   nfs
4444/tcp closed krb524

Now from the remote machine d257 I will start BSD nc and tell it to listen on tcp port 4444 and to redirect all traffic to the file rcv_nc_test.txt:

[archjun@d257 playground]$ nc -l 4444 > rcv_nc_test.txt

From the sending machine, I will start GNU netcat/nc and tell it to send the file test_new_vimrc which contains the following text:

abc
#aabcde
hopefully no more temp files generated in editing path..
One more try... hopefully no more .un~ files will be generated
in the edited file's PATH

[archjun@latitude630 playground]$ nc 192.168.10.57 4444 < test_new_vimrc

nc on both the receiver and sender will not give any indication that the transfer is complete, so on the receiving end, I sent Ctrl-C to terminate the netcat session. Let's see if the content of the text file from latitude630 was sent to rcv_nc_test.txt on d257:

[archjun@d257 playground]$ ls
1  3    foo-replace      orig1      orig2  play-w-bash-functions.sh  test_sed_replace.txt
2  foo  netcat_file.svg  orig1.old  orig3  rcv_nc_test.txt

[archjun@d257 playground]$ cat rcv_nc_test.txt 
abc
#aabcde
hopefully no more temp files generated in editing path..
One more try... hopefully no more .un~ files will be generated
in the edited file's PATH

The content of test_new_vimrc from latitude630 was successfully redirected to rcv_nc_test.txt on d257! nc can also transfer binary files just fine.


Built-in Webservers in Python 2 and Python 3

Nowadays on most Linux installations (RHEL/CentOS, Ubuntu) I work on in the field, python 2 is installed by default. Python 2 comes with its own webserver module called SimpleHTTPServer. When it is invoked from the command line, it will by default serve up the current directory over http on port 8000. Since the remote machine is running firewalld, I first have to open tcp port 8000:

[archjun@d257 bin]$ sudo firewall-cmd --zone=internal --add-port=8000/tcp
[sudo] password for archjun: 
success

[archjun@d257 playground]$ python2 -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
192.168.10.63 - - [16/Oct/2015 23:18:36] "GET / HTTP/1.1" 200 -
192.168.10.63 - - [16/Oct/2015 23:18:36] "GET /favicon.ico HTTP/1.1" 404 -
192.168.10.63 - - [16/Oct/2015 23:18:41] "GET /foo-replace HTTP/1.1" 200 -

Note that in Archlinux python 2 must be invoked with python2. Since 2014, python 3 has been the default python in Arch. In Python 3, invoking the built-in webserver is a bit different. The module name is http.server:

[archjun@d257 playground]$ python -m http.server
Serving HTTP on 0.0.0.0 port 8000 ...
192.168.10.63 - - [16/Oct/2015 23:31:54] "GET /test_sed_replace.txt HTTP/1.1" 200 -
192.168.10.63 - - [16/Oct/2015 23:31:58] "GET /play-w-bash-functions.sh HTTP/1.1" 200 -

Now if I navigate to 192.168.10.57:8000 from another machine on the local network, I get the following HTML page listing the contents of ~/playground:


Much easier than configuring apache/httpd, isn't it? Although the page reads, "Directory listing for /" it is actually serving up ~/playground from host d257. Output is the same for SimpleHTTPServer and http.server. If you want to change the port number, simply add the port you wish to use after the module name, i.e. python -m http.server 8080. Note that if you want to use a port below 1024, you must invoke the webserver as root (but be aware of the security risks).


Darkhttpd

I am a big fan of darkhttpd. I normally use it in a PXE server setup with tftpboot and dnsmasq to serve up files over http as I detailed in this post. While python's built-in webserver is fine for serving up a few files in a pinch, darkhttpd will handle tens of GB of transfers without any hiccups, as I can attest to when installing Linux on multiple machines from a PXE server sending files over 1 gigabit Ethernet.

By default, darkhttpd will share the specified directory on tcp port 8080, but you can specify a different port with the --port option:

[archjun@d257 playground]$ darkhttpd . --port 8000
darkhttpd/1.11, copyright (c) 2003-2015 Emil Mikulic.
listening on: http://0.0.0.0:8000/
1445006481 192.168.10.63 "GET /" 200 1024 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"
1445006504 192.168.10.63 "GET /netcat_file.svg" 200 1463 "http://192.168.10.57:8000/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.71 Safari/537.36"

Navigating to 192.168.10.57:8000 from another machine shows the following HTML page:


btw if you launch darkhttpd as root, it will share the specified directory on tcp port 80.


Conclusion

If you ever find yourself on a locked-down machine without ssh, give netcat, python webservers and darkhttpd a try. In the examples above, opening ports using firewalld is quite easy compared to editing and reloading static iptables rules. I am so glad that RHEL 7.x uses firewalld by default!

2014년 10월 7일 화요일

RHEL/CentOS PXE Network Install Boot Using dnsmasq, darkhttpd, and vsftpd

  This is a followup to my previous post on PXE network install of RHEL/CentOS in which the installation files from the .iso are sent to clients over http from the PXE server. In this post, I will show how you can send installation files from the .iso over ftp instead of http.

Basically the method is the same as that described previously except that the vsftpd server will share a mounted iso to anonymous over ftp.

 Just like in the previous post, you will need to ensure that you have dnsmasq, darkhttpd, and syslinux installed. In addition you will need to install vsftpd.


Differences between PXE Install over http vs. ftp

1. Only one instance of darkhttpd is required
    When sending .iso installation files by http, we used two instances of darkhttpd -- one to share the PXE
    boot images from /usr/local/tftpboot/pxe on 192.168.10.100:80, and another to share
    the .iso installation files from /mnt/distroIso on 192.168.10.100:8080.

    In the ftp, method, however, we only need to run one instance of darkhttpd sharing
    /usr/local/tftpboot/pxe on 192.168.10.100:80

2. Share the mounted iso over ftp, not http
    We must edit /etc/vsftpd.conf such that the iso mount point will also be the directory used for
    anonymous login.


/etc/vsftpd.conf Settings

Make sure that the following entries are set in /etc/vsftpd.conf :

  1. anonymous_enable=YES
  2. no_anon_password=YES
  3. anon_root=/mnt/distroIso/ (you can customize this mountpoint as needed)
  4. dirmessage_enable=YES
  5. xferlog_enable=YES
  6. connect_from_port_20=YES
  7. listen=YES (listen on IPv4 sockets)


Sample PXE cfg file for network boot using ftp


Note that after repo= above, the protocol is ftp:// and the user is anonymous. Once the PXE menu appears on the server console and you press ENTER, the installer will give you an IP address for manually connecting to the installation instance using the command vncviewer 123.456.789:1

To make each installation client automatically reverse connect to a listening instance of vncviewer (vncviewer -listen), in the kernel parameters line after vnc add the statement vncconnect=192.168.XXX.XXX:5500 where the specified IP address corresponds to the machine on which vncviewer is running in listen mode (note that this address can be different from the IP address of your PXE server).

Steps

Note: Some of these steps will overlap with those from my previous PXE tutorial for http.

1. Install required packages
    a. dnsmasq (integrated dns, dhcp, and tftp server)

    b. darkhttpd (http server which we will use to serve up PXE boot images)

    c. syslinux (for boot images used in pxe)

    d. vsftpd (ftp server using ssh which we will use to serve up installation files from the .iso)

2. Disconnect router from Internet (active DHCP processes conflict with dhcpd/dnsmasq
    assigning IP's to PXE clients; Our PXE server machine will be connected to a
    router (preferably with DHCP turned off) or a simple hub and we will run our own DHCP server
    for assigning IP's to PXE clients)

3. Setup PXE
    a. copy all files from /usr/lib/syslinux/bios to a local directory
        that will be used as the tftp-root (for the purposes of this tutorial
        we will call this directory /usr/local/tftpboot/pxe Make sure this directory
        is writable by your regular user)

    b. create a mountpoint for the installation .iso - the mountpoint doesn't have to be
        located under the ../tftpboot directory as in the previous method. A common
        mountpoint would be something like /mnt/distroIso

    c. From the mounted RHEL/CentOS installation .iso, navigate to /images/pxeboot
        and copy the Linux kernel images vmlinuz and initrd.img to a sub-
        directory of /usr/local/tftpboot/pxe, something like
        /usr/local/tftpboot/pxe/images/centos7_64

    d. create a config file for the PXE server under /usr/local/tftpboot/pxe
        Try to make it something distro-specific, i.e. centos7_64_vnc_ftp.cfg

4. Edit dnsmasq.conf
    a. edit /etc/dnsmasq.conf (refer to the previous post on setting up dnsmasq.conf; note
        that dhcp-option-force=209,path/to/pxe_server_cfg_file should point to the PXE config file
        defined above in step 3d)

5. Start/Restart Services
    a. start dnsmasq as root: sudo systemctl start dnsmasq (systemd syntax)

    b. start darkhttpd as rootsudo darkhttpd /usr/local/tftpboot/pxe --no-keepalive
        (this will share the pxe images on port 80 of our wired IP address)

    c. start vsftpd: sudo systemctl start vsftpd
        (this will share the mounted iso on ftp port 20 from the PXE server's IP)

6. PXE Client setup
    a. Physical machine: enter BIOS and set network/PXE boot order to first place

    b. VM: Enable network booting in the VM Manager, change the network interface to
                bridge with wired interface (i.e. eth0 or enp1s0)

7. Installation


Postscript 2016-02-06
This old post only covers Legacy BIOS PXE netboot with dnsmasq. Please refer to a newer post from 2016 that explains how to setup dnsmasq for UEFI PXE netboot:

http://eatpeppershothot.blogspot.kr/2016/02/uefi-and-legacy-bios-pxe-netboot.html

2014년 9월 19일 금요일

RHEL/CentOS PXE install using dnsmasq and darkhttpd (instead of dhcp + httpd + xinetd + tftp-server)

In my new job, one of my responsibilities is going to data centers and installing Linux on many machines at once. Using physical media like DVDs or LiveUSBs one at a time on each machine would take too long, so system engineers often use network boot, aka PXE to send installation images over the network to several machines at once.

The Old Way
Most of the online tutorials (this one, for example) for installing Red Hat Enterprise Linux or CentOS through PXE specify the following steps:

1. Install required packages
    a. dhcp server package

    b. httpd/apache

    c. xinetd

    d. syslinux

    e. tftp-server

2. Disconnect router from Internet (active DHCP processes conflict with dhcpd
    assigning IP's to PXE clients; The PXE server will be connected to a
    router with DHCP turned off or a simple hub. The PXE server will run its own DHCP server
    for assigning IP's to PXE clients)

3. Edit/Create config files
    a. edit /etc/sysconfig/dhcpd

    b. edit /etc/xinetd.d/tftp

    c. edit /etc/dhcp/dhcpd.conf (this step is quite involved)

    d. create apache conf file for PXE /etc/httpd/conf.d/pxeboot.conf

4. Setup PXE
    a. copy PXE boot images (menu.c32, memdisk, etc.) from syslinux directory (in Archlinux, this directory is /usr/lib/syslinux/bios/) to /var/lib/tftpboot

    b. create a mount point for the installation .iso under /var/lib/tftpboot and
        mount the image

    c. create a config file for the PXE server under /var/lib/tftpboot... ie
        something like /var/lib/tftpboot/pxelinux.cfg/default
        (where default is the config file)

5. Start/Restart Services
    a. restart xinetd

    b. restart httpd

    c. restart dhcpd

6. PXE Client setup
    a. Physical machine: enter BIOS and give network/PXE boot first priority

    b. VM: Enable network booting in the VM Manager, change the network interface to
                bridge with the wired interface (i.e. eth0 or enp1s0 et al)

7. Installation


In the above method, there are a lot of config files (for dhcp, tftp, xinet, pxe, apache) that must be manually tweaked to set up PXE booting. This opens up room for error and can be very frustrating when you don't know exactly where you have gone wrong among the many cookbook steps above. Thankfully, there is a better way to do things.


The Simpler Way
I'm a big fan of Archlinux, and the Arch tutorial on using PXE for installing the Archlinux iso was my inspiration for using dnsmasq and darkhttpd to install RHEL/CentOS iso images with PXE. Unfortunately, you cannot blindly follow the instructions from the fine Arch tutorial when installing RHEL.

And why not? In the case of Archiso, PXE images are included within the .iso itself under /arch/boot/syslinux/ but in the case of RHEL/CentOS iso images, although there is a pxe subdirectory at /images/pxeboot/, it only contains Linux kernel images (initrd.img, vmlinuz), not the PXE images necessary for network booting.

Not to fear, we can still use dnsmasq and dispense with configuring xinetd, dhcp, tftp, etc. separately.

1. Install required packages
    a. dnsmasq (dns, dhcp, and tftp server all-in-one)

    b. darkhttpd (http server - much lighter and simpler to set up than httpd)

    c. syslinux (for boot images used in pxe)

2. Disconnect router from Internet (active DHCP processes conflict with dhcpd
    assigning IP's to PXE clients; The PXE server machine will be connected to a
    router with DHCP turned off or a simple hub. dnsmasq will act as the DHCP server
    for assigning IP's to PXE clients)

3. Setup PXE
    a. copy all files from /usr/lib/syslinux/bios to a local directory
        that will be used as the tftp-root (for the purposes of this tutorial
        we will call this directory /usr/local/tftpboot/pxe Make sure this directory
        is writable by your regular user)

    b. create a mountpoint for the installation .iso - the mountpoint doesn't have to be
        located under the ../tftpboot directory as in the previous method. In this tutorial
        the iso mountpoint will be /mnt/distroIso

    c. From the mounted RHEL/CentOS installation .iso, navigate to /images/pxeboot
        and copy the Linux kernel images vmlinuz and initrd.img to a sub-
        directory of /usr/local/tftpboot/pxe, something like
        /usr/local/tftpboot/pxe/images/centos7_64

    d. create a config file for the PXE server under /usr/local/tftpboot/pxe
        Try to make it something distro-specific, i.e. centos7_64_vnc.cfg or
        rhel6dot4_64_vnc.cfg

4. Edit dnsmasq.conf
    a. edit /etc/dnsmasq.conf

5. Start/Restart Services
    a. start dnsmasq: systemctl start dnsmasq (systemd syntax)

    b. start darkhttpd as root: sudo darkhttpd /usr/local/tftpboot/pxe --no-keepalive
        (this will share the pxe images on port 80 of the PXE server's IP address)

    c. start darkhttpd as user: darkhttpd /mnt/path-to-iso-mountpoint/
        (this will share the mounted iso on port 8080 of the PXE server's IP address)

6. PXE Client setup
    a. Physical machine: enter BIOS and set network/PXE boot order to first place

    b. VM: Enable network booting in the VM Manager, change the network interface to
                bridge with wired interface (i.e. eth0 or enp1s0)

7. Installation

The astute reader will notice that instead of having to edit 5 config files (two for dhcp, and one each for xinet.d/tftp, apache, and the pxe server) the new method only requires 2 config files to be edited!

This saves time and avoids the chance of typing errors. The rest of this tutorial will give a detailed walk-through of how to create the distro-specific pxe server config file in step 3d above as well as what settings need to be made in step 4 when editing /etc/dnsmasq.conf


Creating a distro-specific config file for the PXE server

For the purposes of this tutorial we will assume the following:

  • we are installing CentOS7 64-bit from the installation DVD iso image
  • the CentOS7 iso image has been mounted at /mnt/distroIso
  • the tftpboot directory is /usr/local/tftpboot/pxe and is writable by the regular user
  • the Linux kernel images from the iso are located at /usr/local/tftpboot/pxe/images/centos7_64
  • our PXE server will use the IP 192.168.10.100 (this can be manually set through ifconfig or ip addr add...)
Here is a sample pxe server config file for installing CentOS7 over the network using http.


The pxe images needed for network boot in /usr/local/tftpboot/pxe are shared over http on port 80 by darkhttpd (run as root), while the iso installation files are shared on port 8080 by a second instance of darkhttpd (run as regular user) specified in the config file above after inst.repo=

*Note that for RHEL7/CentOS7 inst.repo= or method= has been deprecated in favor of just repo= (But avoid using the syntax repo= in Kickstart files for RHEL 5.X, as this will cause your KS installation to fail)

The config file above will enable a simple pxe install, but when installing on multiple machines at once, it would be helpful connect to each installation instance through vncviewer so we can manage several installs at once. Below are pxe server config files that enable these options; the first enables manually connecting to each instance with VNC, while the second enables Kickstart automated install and VNC reverse connect so you don't have to manually connect to each installation instance with vncviewer 123.456.789:1


*Note: To get vnc reverse connect to automatically connect to vncviewer, you must launch the viewer in a separate terminal with the flag listen:

vncviewer -listen

and then reverse connections should automatically appear as they come in. However, when I tried this on my Archlinux machine, only the first connection opened automatically; all others I had to connect to manually following the IP given by the client terminal. I wonder if it isn't related to this Redhat bug report.


Postscript 2014-10-8: tigervnc up to version 1.1.0-8 from the RHEL/CentOS 6.X repos allows vncviewer in listen mode to connect to multiple vnc reverse connections, but versions higher than this (for example, tigervnc 1.2.80 and above from RHEL/CentOS 7 and other distros) allow vncviewer in listen mode to connect to only one incoming connection at a time! This makes automatic pxe installs of multiple machines much more cumbersome, as we must manually connect to each installation instance with vncviewer 123.456.789:1 My current workaround is use a VM running CentOS 6.5 as my PXE server for installation jobs, as there is no limit on the number of vnc reverse connections that can be used in tigervnc 1.1

Slight changes to pxe server config files can also enable pxe network installation over ftp or nfs instead of http. I will cover these variations in subsequent posts.

Settings for /etc/dnsmasq.conf

Here is the link to a sample dnsmasq.conf configured for the pxe server config file centos7_64_vnc.cfg which appears above.

The entries which should be uncommented and defined are as follows:
    - port=0 (disable DNS; but DNS must be enabled for auto installs of Ubuntu)
    - inferface=eth0 (or enp1s0 etc, YMMV)
    - bind-interfaces
    - dhcp-range=(range of ip addresses to assign to clients)
    - dhcp-option-force=209,path/to/pxe_server_cfg_file
      (relative path from tftp-root)
    - dhcp-boot=lpxelinux.0 (or gpxelinux.0 or pxelinux.0, depending on your
      version of syslinux)
    - enable-tftp (enable dnsmasq's built-in tftp server)
    - tftp-root=/path/to/pxe/boot/images (in our case
   /usr/local/tftpboot/pxe)

Questions and comments are welcome, especially if you have a better way of doing things!

Postscript 2016-02-06
This old post only covers Legacy BIOS PXE netboot with dnsmasq. Please refer to a newer post from 2016 that explains how to setup dnsmasq for UEFI PXE netboot:

http://eatpeppershothot.blogspot.kr/2016/02/uefi-and-legacy-bios-pxe-netboot.html