Using Systemrescuecd LiveUSB over serial console

Recently I found myself without any RHEL DVD's or liveUSB's when encountering a server that failed with disk errors on reboot. At work I mostly deal with servers using Redhat Enterprise Linux versions anywhere from 4.X(!!) to 7.X, so I used to carry around a CD case containing RHEL v.4~7 DVD's. This is a hassle because some servers don't have optical drives and it's inconvenient to carry around a bunch of CD/DVD's.

I therefore thought it would be a good idea to create liveUSB's of RHEL so I could boot into Redhat rescue mode when servers fail to boot or have disk errors, but creating live USB media from RHEL iso images isn't straightforward.

For RHEL 7.X, Redhat recommends using dd to write the iso file onto a USB memory stick, but this method doesn't seem to work when used with DVD iso files for RHEL5. Unfortunately, the server that failed to boot was running RHEL 5.8. Luckily, I happened to have a 2GB USB stick in my backpack that had SystemRescueCD v. 4.5.1 installed as a liveUSB (as of Aug 2015, the most recent version is 4.5.4).

SystemRescueCD is a very handy liveCD/USB distro for repairing filesystems, editing partitions, recovering MS Windows passwords, etc. on host systems with boot, storage, or other problems.

I had used this utility distro on personal machines (netbooks, laptops, desktop) at home, but on this particular day, I was working on an ATCA MPCBL0030 (Intel) blade with no separate video output. Also no KVM switch was available on the rack, so the only way to interact with the server was over serial console.

In order to boot Linux and see all output over serial console instead of on a monitor, it is necessary to edit the kernel parameters in the boot loader. In the case of RHEL, you would press 'e' at the GRUB menu on the selected kernel and then add the following at the end of the line kernel= (assuming a baud rate of 57600 bps):

console=tty0 console=ttyS0,57600

Since SystemRescueCD version 0.26, serial console support is compiled into the linux kernel, so adding serial console options to the end of the kernel line in the bootloader works just fine. The kernel parameter syntax will differ slightly from that used above for RHEL, however.

Once the server booted into the SystemRescueCD liveUSB menu, I selected option C) Standard 64-bit kernel (rescue64) with more choice which is the 5th from bottom:

I then highlighted option 5. SystemRescueCd with a console in 800x600 and pressed [TAB] to edit the kernel parameters:

At the bottom in white text, you can see the kernel parameters

.linux rescue64 video=800x600 initrd=initram.igz

Note: the _ in the screenshot above is the cursor, not an underbar character!

Remove the parameter video=800x600 and append console=ttyS0,baudRate text to the kernel line so that it looks like the following:

.linux rescue64 initrd=initram.igz console=ttyS0,115200 text

After editing the kernel parameters, press [ENTER] and SysRecCd will boot the Linux kernel with the options specified. You should be able to observe the boot messages over serial console using your communication program of choice (i.e. screen, putty, minicom).

You can see that the serial console kernel parameters used for SystemRescueCD differ from those used in RHEL. RHEL requires two consoles to be specified, console=tty0 console=ttyS0,baudRate but I couldn't get serial console output working in SysRecCD when using the RHEL syntax.

I used a serial console baud rate of 115200 bps because that's what the MPCBL0030 supports.

SystemRescueCD includes almost all the CLI disk utilities you can think of as well as GUI tools like gparted. Of course, since I was working over serial console, I didn't want to enter runlevel 5 and launch X11.

From the command line, I then checked if the server's disks and partitions were being recognized with fdisk and tried to repair the ext3 filesystem using fsck. As the liveUSB was detected as /dev/sda, the local disk on the server appeared as /dev/sdb. Inititally fsck -a /dev/sdb1(2,3...) failed, but e2fsck -f -y /dev/sdb1(2,3 ...) succeeded in correcting the errors in the ext3 filesystem caused by inode corruption.

After a reboot, the GRUB menu finally appeared and RHEL 5.8 booted without any problems. I highly-recommend carrying around a liveUSB containing SysRecCD instead of RHEL media for rescue mode!