2015년 8월 23일 일요일

Administering a Linux system through HP integrated Lights Out (iLO) port

I recently had to work on some HP Proliant BL460c Gen 8 blade servers contained in a 7000c chassis. It is interesting to note that each blade does not have its own network ports; all the ports are provided by the 7000c chassis. Here is what 16 blades installed into a 7000c chassis look like (front view, rear view):



There was a KVM switch connected to the chassis, but for some reason keyboard input wasn't being properly passed on to the machines. Fortunately, an HP hardware engineer was on hand to show me how to connect to individual blades using the iLO port on the 7000c chassis.

The HP iLO port on Proliant servers looks just like a regular Ethernet port but is always labeled iLO to distinguish it from a other network ports. You can assign a fixed IP to the iLO port in the BIOS or even assign an IP if you are running a DHCP server on your laptop which is connected to the iLO port. In this particular case, the HP engineer had already entered BIOS and assigned the IP 192.168.0.100 to the iLO port, so I simply connected my laptop to the iLO port with a cat6 Ethernet cable.

The default iLO user:pass was admin:hpinvent and connection to iLO can be made via telnet, ssh, and http. I connected via ssh as follows:

[archjun@lenovoS310 ~]$ ssh admin@192.168.0.100

-----------------------------------------------------------------------------
WARNING: This is a private system.  Do not attempt to login unless you are an
authorized user.  Any authorized or unauthorized access and use may be moni-
tored and can result in criminal or civil prosecution under applicable law.
-----------------------------------------------------------------------------
Firmware Version: 3.71
Built: 12/07/2012 @ 13:26
OA Bay Number:  1 
OA Role:        Active 
admin@192.168.0.100's password: 






HP BladeSystem Onboard Administrator
(C) Copyright 2006-2012 Hewlett-Packard Development Company, L.P.


Type 'HELP' to display a list of valid commands.
Type 'HELP ' to display detailed information about a specific command.

Type 'HELP HELP' to display more detailed information about the help system.

As this was my first time working with HP iLO, I referred to the HP BladeSystem Onboard Administrator Command Line Interface User Guide. The chassis had 16 BL460c servers installed, so I first needed to obtain a list of machines to connect to.

OA-10604BA59737> SHOW SERVER LIST

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 ILOSGH3280A6K                 192.168.0.200   OK       On      Off 
  2 ILOSGH3280A66                 192.168.0.201   OK       On      Off 
  3 ILOTW36NP1543                 192.168.0.202   OK       On      Off 
  4 ILOSGH3280A5V                 192.168.0.203   OK       On      Off 
  5 ILOSGH3280A6P                 192.168.0.204   OK       On      Off 
  6 ILOSGH3280A5P                 192.168.0.205   OK       On      Off 
  7 ILOSGH3280A6V                 192.168.0.206   OK       On      Off 
  8 ILOSGH33013W5                 192.168.0.207   OK       On      Off 
  9 ILOSGH3280A5H                 192.168.0.208   OK       On      Off 
 10 ILOSGH33013W7                 192.168.0.209   OK       On      Off 
 11 ILOSGH3280A7C                 192.168.0.210   OK       On      Off 
 12 ILOSGH3280A5C                 192.168.0.211   OK       On       *  
 13 ILOSGH3280A6S                 192.168.0.212   OK       On      Off 
 14 ILOSGH3280A7A                 192.168.0.213   OK       On      Off 
 15 ILOSGH3280A5X                 192.168.0.214   OK       On      Off 
 16 ILOSGH3280A62                 192.168.0.215   OK       On      Off 
Totals: 16 server blades installed, 16 powered on.

 * = Blade UID is blinking and a critical operation is being performed on the blade (firmware update in progress or remote console in use).

According to the command reference I linked to above, it is possible to connect to a specific blade using the following command syntax:

CONNECT SERVER [SERIAL] : Opens a Text Console session to the iLO
specified. If the optional argument SERIAL is specified a Virtual Serial Port
session is started.

I wanted to connect to blade #12, so I entered the following:

OA-10604BA59737> connect server 12

Connecting to bay 12 ...
User:OAtmp-admin-55B87738 logged-in to ILOSGH3280A5C.(192.168.0.211 / fe80::da9d:67ff:fe67:6a65)
iLO 4 Standard Blade Edition 1.22 at  Apr 19 2013
Server Name: SLDJ-AGCF_002B
Server Power: On




</>hpiLO->


After typing help from the hpiLO prompt, I noticed the following CLI commands of interest:

...
HP CLI Commands:

POWER    : Control server power.
UID      : Control Unit-ID light.
NMI      : Generate an NMI.
VM       : Virtual media commands.
LANGUAGE : Command to set or get default language
VSP      : Invoke virtual serial port.
TEXTCONS : Invoke Remote Text Console.

TEXTCONS looks promising...

</>hpiLO-> TEXTCONS

status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND ERROR-UNSPECIFIED
Wed Jul 29 18:06:07 2015

iLO Advanced License required for this functionality.

This was a disappointment. According to the HP iLO v4 user's guide p.212, it is still possible to get a remote console even if you don't have an iLO Advanced license by using a Virtual Serial Port (VSP) and connecting to a linux installation over serial console. This would require, of course, that you edit the KERNEL= line in /boot/grub/grub.conf and append serial console settings like console=ttyS0,115200 for example.

Fortunately, I learned from the HP hardware engineer that iLO has a nice web management interface. I pointed my browser at the IP address assigned to the iLO port (192.168.0.100) and after entering the login credentials admin:hpinvent, I was able to access individual blades via remote console by clicking the button Integrated Remote Console. The downside is that the Integrated Remote Console only works on Windows clients with Internet Explorer (which supports ActiveX). I had to load a Windows VM with bridged network in order to use the Integrated Remote Console session.

The Integrated Remote Console session for Windows clients also allows you to mount USB sticks or .iso images locally and have them accessible to the remote machine. If you don't have a Windows machine or VM handy, on Linux you can still get a remote terminal through the iLO web management interface by clicking Remote Console (not Integrated Remote Console, which is Windows-only) but this requires that you install the Iced Tea web plugin before hand, as Remote Console requires Java web plugins.

One thing that is inconvenient about both types of remote consoles accessible through the iLO web interface is that the sessions are not text-based TTY or PTY sessions, but emulated graphical sessions! This means that you cannot copy-and-paste anything to and from these screens, which is a big inconvenience. If you absolutely need a pure text remote console, you will either have to buy an iLO Advanced License or enable the Virtual Serial Port so you can connect to your Linux installation over serial console through the iLO port.

2015년 8월 16일 일요일

Using Systemrescuecd LiveUSB over serial console

Recently I found myself without any RHEL DVD's or liveUSB's when encountering a server that failed with disk errors on reboot. At work I mostly deal with servers using Redhat Enterprise Linux versions anywhere from 4.X(!!) to 7.X, so I used to carry around a CD case containing RHEL v.4~7 DVD's. This is a hassle because some servers don't have optical drives and it's inconvenient to carry around a bunch of CD/DVD's.

I therefore thought it would be a good idea to create liveUSB's of RHEL so I could boot into Redhat rescue mode when servers fail to boot or have disk errors, but creating live USB media from RHEL iso images isn't straightforward.

For RHEL 7.X, Redhat recommends using dd to write the iso file onto a USB memory stick, but this method doesn't seem to work when used with DVD iso files for RHEL5. Unfortunately, the server that failed to boot was running RHEL 5.8. Luckily, I happened to have a 2GB USB stick in my backpack that had SystemRescueCD v. 4.5.1 installed as a liveUSB (as of Aug 2015, the most recent version is 4.5.4).

SystemRescueCD is a very handy liveCD/USB distro for repairing filesystems, editing partitions, recovering MS Windows passwords, etc. on host systems with boot, storage, or other problems.

I had used this utility distro on personal machines (netbooks, laptops, desktop) at home, but on this particular day, I was working on an ATCA MPCBL0030 (Intel) blade with no separate video output. Also no KVM switch was available on the rack, so the only way to interact with the server was over serial console.

In order to boot Linux and see all output over serial console instead of on a monitor, it is necessary to edit the kernel parameters in the boot loader. In the case of RHEL, you would press 'e' at the GRUB menu on the selected kernel and then add the following at the end of the line kernel= (assuming a baud rate of 57600 bps):

console=tty0 console=ttyS0,57600

Since SystemRescueCD version 0.26, serial console support is compiled into the linux kernel, so adding serial console options to the end of the kernel line in the bootloader works just fine. The kernel parameter syntax will differ slightly from that used above for RHEL, however.

Once the server booted into the SystemRescueCD liveUSB menu, I selected option C) Standard 64-bit kernel (rescue64) with more choice which is the 5th from bottom:



I then highlighted option 5. SystemRescueCd with a console in 800x600 and pressed [TAB] to edit the kernel parameters:



At the bottom in white text, you can see the kernel parameters

.linux rescue64 video=800x600 initrd=initram.igz

Note: the _ in the screenshot above is the cursor, not an underbar character!

Remove the parameter video=800x600 and append console=ttyS0,baudRate text to the kernel line so that it looks like the following:

.linux rescue64 initrd=initram.igz console=ttyS0,115200 text

After editing the kernel parameters, press [ENTER] and SysRecCd will boot the Linux kernel with the options specified. You should be able to observe the boot messages over serial console using your communication program of choice (i.e. screen, putty, minicom).

You can see that the serial console kernel parameters used for SystemRescueCD differ from those used in RHEL. RHEL requires two consoles to be specified, console=tty0 console=ttyS0,baudRate but I couldn't get serial console output working in SysRecCD when using the RHEL syntax.

I used a serial console baud rate of 115200 bps because that's what the MPCBL0030 supports.

SystemRescueCD includes almost all the CLI disk utilities you can think of as well as GUI tools like gparted. Of course, since I was working over serial console, I didn't want to enter runlevel 5 and launch X11.

From the command line, I then checked if the server's disks and partitions were being recognized with fdisk and tried to repair the ext3 filesystem using fsck. As the liveUSB was detected as /dev/sda, the local disk on the server appeared as /dev/sdb. Inititally fsck -a /dev/sdb1(2,3...) failed, but e2fsck -f -y /dev/sdb1(2,3 ...) succeeded in correcting the errors in the ext3 filesystem caused by inode corruption.

After a reboot, the GRUB menu finally appeared and RHEL 5.8 booted without any problems. I highly-recommend carrying around a liveUSB containing SysRecCD instead of RHEL media for rescue mode!

2015년 8월 8일 토요일

Korean Employment IQ Test Question - Alchemists transforming lead to gold

My wife is an interpreter and translator for the Korean-English language pair. She recently had a first-round interview with a Korean government agency and had to take an IQ test. Although the position she interviewed for requires fluency in English as well as technical knowledge of railroads, the test she was given was not job-oriented at all. Out of 70 questions, 40 required high school math and logic while the remainder were personality and integrity-related questions.

Today all the major corporations (Korean chaebol) and many government agencies in Korea administer their own aptitude tests to job applicants. These tests go by the names SSAT (Samsung Aptitude Test), NHAT (Nonghyup Aptitude Test - Nonghyup is the Korean government-run agricultural cooperative), etc.

After taking the test, my wife asked me about the following question. I have changed the wording of the question but maintained the essence of the problem to make it more presentable:

April and John are alchemists. April can transform 1 kg of lead into gold in 12 days, while John can transform 1 kg of lead into gold in 24 days. If both April and John work together, what is their effective transmutation rate for converting 1 kg of lead into gold?

(a) 8 days  (b) 9 days  (c) 11 days  (d) 12 days


As she was pressed for time, my wife just guessed 9 days, but she asked me to elaborate when I told her the answer should be 8 days.

Answer

Let's add a new quantity to the problem to make it more concrete and solvable. Let's assume that there are 48 kilograms of lead total. If April works alone, she can transform 48 kg of lead into gold in

48 kg x 12 days/kg = 576 days

If John works alone, he can transform 48 kg of lead into gold in

48 kg x 24 days/kg = 1152 days

We could ask ourselves how many days will it take them to transform all 48 kg of lead into gold if April and John work together.

We can chart out a 12-day cycle and keep a running count of the lead transmuted into gold.

Day 12: 1 kg Au (April) TOTAL: 1 kg
Day 24: 1 kg Au (April), 1 kg Au (John) TOTAL: 3 kg
Day 36: 1 kg Au (April) TOTAL: 4 kg
...

We can see that every 24 days, April and John can complete the transmutation of 3 kilos of lead to gold. This means that in 24 x 16 = 384 days they could transform 48 kilos of Pb to Au. Also note that

24 days / 3 kg = 8 days / 1 kg

which means that 1 kg of gold every 8 days is their effective (but not actual) rate of transmutation when working together. You should not take this to suggest, however, that every 8 days a one-kilo lump of gold will be ready and waiting! April and John will still be transmuting lead to gold on a 12-day cycle.


Musings about Employment in Korea

What this question has to do with being an interpreter/translator for a railway project mystifies me. I have read on the Internet that employment tests with questions unrelated to job requirements were ruled illegal in the US in the Supreme Court case Griggs vs. Duke Power Co. Even if such a ruling were made in South Korea, I don't think it would ever be enforced, considering the immense power that employers have. With the exception of some heavily-unionized sectors of the Korean economy (domestic financial institutions, tier-one auto manufacturers, ports and shipping) for the most part labor laws are not enforced.

2015년 8월 2일 일요일

RHEL/CentOS downgrading to a previously-installed kernel without using yum

Suppose you did a kernel upgrade and the current version is now 2.6.18-308.el5 (stock kernel for RHEL 5.8) but you want to downgrade to 2.6.18-194.el5 (stock kernel for RHEL 5.5) which is still installed on the system and appears in grub.conf.

1. Edit /boot/grub/grub.conf and make sure that the default entry points to the previous kernel (probably default=1, but YMMV)

2. Reboot the system. You should now boot into the previous kernel version. After logging in, remove the higher kernel version from the system:

rpm -e kernel-2.6.18-308.el5 kernel-devel-2.6.18-308.el5

Now all traces of the higher-version kernel will be gone and it will not appear in grub.conf

What if you want to do a kernel downgrade but the previous kernel version is not installed?

If you have a RedHat subscription, first download the required kernel packages from the RedHat website. In the case of RHEL 5.X, all you need are the kernel and kernel-devel packages, but for RHEL6.X  I usually need 4 packages - kernel-firmware, bfa-firmware (for Brocade Fibre Channel Host Bus Adapter), kernel, and kernel-devel

In this second example, let's assume kernel 2.6.18-308.el5 is the current kernel version but that I would like to install 2.6.18-194.el5 and make it the default kernel.

1. rpm -ivh kernel-2.6.18-194.el5 kernel-devel-2.6.18-194.el5

This will install kernel-2.6.18-194 and make it the default in /boot/grub/grub.conf

You can verify that the older kernel has been installed by invoking cat /boot/grub/grub.conf and checking that the older kernel appears in the GRUB menu. If you so desire, after rebooting you can manually remove the other kernel with rpm -e