2015년 8월 23일 일요일

Administering a Linux system through HP integrated Lights Out (iLO) port

I recently had to work on some HP Proliant BL460c Gen 8 blade servers contained in a 7000c chassis. It is interesting to note that each blade does not have its own network ports; all the ports are provided by the 7000c chassis. Here is what 16 blades installed into a 7000c chassis look like (front view, rear view):



There was a KVM switch connected to the chassis, but for some reason keyboard input wasn't being properly passed on to the machines. Fortunately, an HP hardware engineer was on hand to show me how to connect to individual blades using the iLO port on the 7000c chassis.

The HP iLO port on Proliant servers looks just like a regular Ethernet port but is always labeled iLO to distinguish it from a other network ports. You can assign a fixed IP to the iLO port in the BIOS or even assign an IP if you are running a DHCP server on your laptop which is connected to the iLO port. In this particular case, the HP engineer had already entered BIOS and assigned the IP 192.168.0.100 to the iLO port, so I simply connected my laptop to the iLO port with a cat6 Ethernet cable.

The default iLO user:pass was admin:hpinvent and connection to iLO can be made via telnet, ssh, and http. I connected via ssh as follows:

[archjun@lenovoS310 ~]$ ssh admin@192.168.0.100

-----------------------------------------------------------------------------
WARNING: This is a private system.  Do not attempt to login unless you are an
authorized user.  Any authorized or unauthorized access and use may be moni-
tored and can result in criminal or civil prosecution under applicable law.
-----------------------------------------------------------------------------
Firmware Version: 3.71
Built: 12/07/2012 @ 13:26
OA Bay Number:  1 
OA Role:        Active 
admin@192.168.0.100's password: 






HP BladeSystem Onboard Administrator
(C) Copyright 2006-2012 Hewlett-Packard Development Company, L.P.


Type 'HELP' to display a list of valid commands.
Type 'HELP ' to display detailed information about a specific command.

Type 'HELP HELP' to display more detailed information about the help system.

As this was my first time working with HP iLO, I referred to the HP BladeSystem Onboard Administrator Command Line Interface User Guide. The chassis had 16 BL460c servers installed, so I first needed to obtain a list of machines to connect to.

OA-10604BA59737> SHOW SERVER LIST

Bay iLO Name                      iLO IP Address  Status   Power   UID Partner
--- ----------------------------- --------------- -------- ------- --- -------
  1 ILOSGH3280A6K                 192.168.0.200   OK       On      Off 
  2 ILOSGH3280A66                 192.168.0.201   OK       On      Off 
  3 ILOTW36NP1543                 192.168.0.202   OK       On      Off 
  4 ILOSGH3280A5V                 192.168.0.203   OK       On      Off 
  5 ILOSGH3280A6P                 192.168.0.204   OK       On      Off 
  6 ILOSGH3280A5P                 192.168.0.205   OK       On      Off 
  7 ILOSGH3280A6V                 192.168.0.206   OK       On      Off 
  8 ILOSGH33013W5                 192.168.0.207   OK       On      Off 
  9 ILOSGH3280A5H                 192.168.0.208   OK       On      Off 
 10 ILOSGH33013W7                 192.168.0.209   OK       On      Off 
 11 ILOSGH3280A7C                 192.168.0.210   OK       On      Off 
 12 ILOSGH3280A5C                 192.168.0.211   OK       On       *  
 13 ILOSGH3280A6S                 192.168.0.212   OK       On      Off 
 14 ILOSGH3280A7A                 192.168.0.213   OK       On      Off 
 15 ILOSGH3280A5X                 192.168.0.214   OK       On      Off 
 16 ILOSGH3280A62                 192.168.0.215   OK       On      Off 
Totals: 16 server blades installed, 16 powered on.

 * = Blade UID is blinking and a critical operation is being performed on the blade (firmware update in progress or remote console in use).

According to the command reference I linked to above, it is possible to connect to a specific blade using the following command syntax:

CONNECT SERVER [SERIAL] : Opens a Text Console session to the iLO
specified. If the optional argument SERIAL is specified a Virtual Serial Port
session is started.

I wanted to connect to blade #12, so I entered the following:

OA-10604BA59737> connect server 12

Connecting to bay 12 ...
User:OAtmp-admin-55B87738 logged-in to ILOSGH3280A5C.(192.168.0.211 / fe80::da9d:67ff:fe67:6a65)
iLO 4 Standard Blade Edition 1.22 at  Apr 19 2013
Server Name: SLDJ-AGCF_002B
Server Power: On




</>hpiLO->


After typing help from the hpiLO prompt, I noticed the following CLI commands of interest:

...
HP CLI Commands:

POWER    : Control server power.
UID      : Control Unit-ID light.
NMI      : Generate an NMI.
VM       : Virtual media commands.
LANGUAGE : Command to set or get default language
VSP      : Invoke virtual serial port.
TEXTCONS : Invoke Remote Text Console.

TEXTCONS looks promising...

</>hpiLO-> TEXTCONS

status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND ERROR-UNSPECIFIED
Wed Jul 29 18:06:07 2015

iLO Advanced License required for this functionality.

This was a disappointment. According to the HP iLO v4 user's guide p.212, it is still possible to get a remote console even if you don't have an iLO Advanced license by using a Virtual Serial Port (VSP) and connecting to a linux installation over serial console. This would require, of course, that you edit the KERNEL= line in /boot/grub/grub.conf and append serial console settings like console=ttyS0,115200 for example.

Fortunately, I learned from the HP hardware engineer that iLO has a nice web management interface. I pointed my browser at the IP address assigned to the iLO port (192.168.0.100) and after entering the login credentials admin:hpinvent, I was able to access individual blades via remote console by clicking the button Integrated Remote Console. The downside is that the Integrated Remote Console only works on Windows clients with Internet Explorer (which supports ActiveX). I had to load a Windows VM with bridged network in order to use the Integrated Remote Console session.

The Integrated Remote Console session for Windows clients also allows you to mount USB sticks or .iso images locally and have them accessible to the remote machine. If you don't have a Windows machine or VM handy, on Linux you can still get a remote terminal through the iLO web management interface by clicking Remote Console (not Integrated Remote Console, which is Windows-only) but this requires that you install the Iced Tea web plugin before hand, as Remote Console requires Java web plugins.

One thing that is inconvenient about both types of remote consoles accessible through the iLO web interface is that the sessions are not text-based TTY or PTY sessions, but emulated graphical sessions! This means that you cannot copy-and-paste anything to and from these screens, which is a big inconvenience. If you absolutely need a pure text remote console, you will either have to buy an iLO Advanced License or enable the Virtual Serial Port so you can connect to your Linux installation over serial console through the iLO port.