Howto setup IPMI under Linux (Debian / Sarge) on the Intel SR2300 Server Chassis (Intel Server Board SE7501WV2)

$Revision: 1.11 $ $Author: tim $

Introduction

This document describes how to setup Debian / Sarge to take advantage of the management features of the Intel SR2300, this chassis uses the Intel Server Board SE7501WV2, but nearly all of this is also relevant to other related Intel server motherboards (such as the SE7501BR2, and the SE7501HG2), a lot of it will be relevant to other boards which implement IPMI v1.5, or later.

At the time of writing (July 2004), the Linux IPMI support is quite mature, but I found that information was on the sparse side, and getting a working system together seemed to require a lot of googling, reading of think technical documentation, and stabbing in the dark. Hence this document - the purpose of which is to allow Semantico staff to recreate the IPMI-based installation which I carried out during July, but which will hopefully be helpful to others as well.

What is IPMI, and why should I care?

The original motivation for setting up IPMI for me was to make use of Serial Over LAN - this allows you to deploy these servers in a remote location, make only power, and Ethernet connections to each server, and yet still get nearly all of the benefits of expensive KVM, or other remote control systems - such as those built around serial concentrators, with: IPMI stands for Intelligent Platform Management Interface and is an open standard for machine health, and control (including remote control), and is implemented by many hardware vendors - Intel is one of the originators, and early adopters of the standard.  Here are some useful things that IPMI can do on the SR2300 with Linux:
If you would like to know more, then this document from the 2003 Linux Symposium provides more detail.  IPMI is a large standard, with a slight whiff of committee about it, so I'm just going to consider what I think are the most useful bits, and the implementation which the SE7501WV2 makes (this is supposed to be about a single type of server, after all).  Note that IPMI seems to have more than its fair share of TLAs.

How IPMI works, and jargon

It is useful to know a bit about how IPMI does its stuff - so I'll give an overview, and try to bust some weird IPMI/Intel jargon.  There is a second autonomous computer on the motherboard (or baseboard, in IPMI's politically correct / obfusicated language), this is a very simple, low power-consumption device, which should operate as long as power is connected to the machine (including when the majority of the server is powered down) - in IPMI speak, this computer is called the BMC - the Baseboard Management Controller - it uses its own firmware, which is independent of the system BIOS.  On the SE7501WV2, and particularly on the SR2300 the BMC is connected to:

This means that you can talk to the BMC from the server itself under Linux, or from a remote machine via the network (if configured).  The IPMI standard allows for other interfaces as well.

Install

Getting the Software


The packages and tools that I used to gain access to IPMI functionality are:

  1. Kernel space tools:
    1. Risking losing serial data from the console output, if you have not configured it for RTS/CTS hardware serial flow control - this happens if the BMC cannot send serial data over the LAN quickly enough, and fills up its buffers, thus dropping data.
    2. Having the kernel block (very bad - especially during a reboot) whilst timing out sending kernel output to the serial port, if the serial over LAN session is down
  3. User space tools:

Setup

Setting up IPMI

ipmitool + OpenIPMI

I will assume here that you will want local access to the BMC from Linux, using the OpenIPMI drivers, the advantages of doing this, over using Intel's bootable CD are:
The disadvantages are:
Once you have what you feel is a suitable kernel installed, you will want to load the appropriate modules, on my machine these are:

ipmi_si_drv
ipmi_devintf

The kernel should say something like this (check /var/log/kern.log):

IPMI System Interface driver version v30, KCS version v30, SMIC version v30, BT version v30
ipmi_si: Found SMBIOS-specified state machine at I/O address 0xca2
 IPMI kcs interface initialized

If you aren't using devfs, ensure that you have an /dev/ipmi0 device for ipmitool to talk to:

# mknod -m 0600 /dev/ipmi0 c 254 0

Note, that as far as I know, the IPMI device is most likely to end up at device major number 254, but that it will take devices from the 240-254 block, which according to linux/Documentation/devices.txt is "Allocated for local/experimental use. For devices not assigned official numbers, these ranges should be used in order to avoid conflicting with future assignments." I believe that, this is because device numbers are no longer being "officially" assigned, in preparation for the introduction of fully dynamic device number allocation. So, if you have another driver which uses character device numbers from this block, and this other driver gets there first, then ipmi will end up at c 253 0 or lower...

To check where your ipmidev has ended up,

# cat /proc/devices


Testing ipmitool + OpenIPMI

You should now be able to speak to the BMC using ipmitool locally, e.g.

# ipmitool -I open chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : previous
Last Power Event     : ac-failed
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false

If this works, you may want to try out the following, and/or have a look through the manual page to see what else you can do -

# ipmitool -I open sdr list
# ipmitool -I open sel list
# ipmitool -I open chassis identify 1
# ipmitool -I open chassis identify 0

ipmitool + IPMI over LAN

As shipped, the BMC in the WV2 boards doesn't listen on the LAN interfaces - it must be configured to do so, there are at least two ways of doing this, the ones I know about are:

Testing ipmitool + IPMI over LAN

You will now want to test IPMI over LAN, two things are worth pointing out at this stage:
Install ipmitool on another machine (this machine needn't  have OpenIPMI, or an IPMI equipped board), then, from that machines try the following:

$ IPMI_PASSWORD=<your password here> ipmitool -I lan -H <target hostname, or IP address> -E chassis status

This should give you similar output to running the command locally.  Now for something more interesting (well, I think it is more interesting, anyway), shutdown the target machine (e.g. # shutdown -h now), you should still be able to run the previous command, except amongst the output you should see:

System Power         : off

Now you can do this:

$ IPMI_PASSWORD=<your password here> ipmitool -I lan -H <target hostname, or IP address> -E chassis power on

The machine should power up, and boot automatically.  You can also use "power reset" to affect a remote hardware reset of the target machine.

Setting up Serial Over LAN (SOL)

If you like, you can set up the BMC on these Intel boards to talk to the main PC via its serial port B (the rj45 port), and relay the input, and output to another machine, over the LAN interface.

The IPMI v2.0 specification specifies RTS/CTS flow control should be used between the main PC, and the BMC during serial communication - this is necessary because the BMC needs a way to tell the main PC to stop sending it data, in the case that it is about to run out of buffer space - because it hasn't been able to send data to the SOL client quickly enough.  As previously mentioned, the Linux kernel, currently has a problem with RTS/CTS on serial consoles (although serial log-ins are unaffected, since the console output seems to be independant of the settings that the getty sets on the same port).

At the time of writing, the BMC code on the SE7501WV2 implements IPMI v1.5 - and IPMI v1.5 does not define SOL support, so the SOL implementation on these boards is proprietary, and Intel is not currently releasing details except under an NDA for some strange reason(boo, hiss).  The SOL support in IPMI v2.0 is believed to be based on the Intel implementation, so it may be possible to reverse engineer the implementation, but in the mean time you must use Intel's "dpccli / dpcproxy" programs to use the SOL functionality, unfortunately at the time of writing, these:

But they are all that can be used at the moment, and they work well enough to be useful. 

A Quick Note About Security

Although the dpcproxy program must be spoken to using telnet (if you need any security at all on the LAN, then I recommend making the dpcproxy bind to the loopback interface only, and ssh <dpcproxyserver> telnet localhost 623), the SOL session itself (between dpcproxy, and the BMC) is encrypted by default (although Intel gives no details of the encryption), so passwords typed over the SOL session are (probably) not trivially interceptable.

IPMI includes some security, but I have not looked into what strengh of security ipmitool is able to use with the SE7501 boards. Any answers welcome..

The Recipe

The basic setup procedure is as follows..

On the machine(s) from which you will manage the other servers:
On the target machine(s), you will need to set up serial console support - this is covered in detail in http://www.tldp.org/HOWTO/Remote-Serial-Console-HOWTO/ but I will include a brief recipe here:

Enabling and Testing SOL

If you like, you can test the above configuration, using a real serial null-modem cable, and a terminal program such as "minicom", or "gkermit", in either case, you should then do the following:

On the management machine (you could also do the first step on the target machine, using the OpenIPMI interface)

# IPMI_PASSWORD=<your password here> ipmitool -I lan -H <target hostname, or IP address> -E chassis sol setup

Note that the ipmitool "sol" command is likely to be renamed when IPMI v2.0 sol support is added to the program.  I have read documentation which hints that it might be possible to set SOL support to "always on" using the Intel bootable CD - using the ipmitool "sol setup" command, the SOL session can be interupted by some actions (e.g. local keyboard activity, as simulated by some kvms), but I have not verified this.

# telnet localhost 623
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
Server: <your server name>
Username:
Password: *********
Login successful


dpccli> console

myservername login:

You should then be able to log in as root, and reboot the machine, following the entire boot process on the serial console.  To get out of the SOL session, you need to send the sequence "~." to the dpcproxy program.  This clashes nicely with the ssh escape sequence ("<cr>~." tells ssh to terminate an ssh session - so that you may need to remember to type "~~." if you are typing the sequence after a new line), and also means that you cannot type the tilde character on the console (at least I have been unable to figure out how to, and there is no man page).  Nice one Intel.

Other defects include the fact that you cannot send a serial break (for sysrq), and that it quite often seems to exit, and get confused when there is a lot of I/O going on.

Using the Serial BIOS

The serial BIOS interface is a bit brain damaged in that it does not recognise the "F11", and "F12" key escape codes that most terminal programs send, instead you can send "Esc-!", and "Esc-@" (yes very logical, as long as the '@' key is normally typed using 'Shift-2' - as on US keyboards, not miles away from the '2' key, as on many non-US keyboards).  These escapes from HP, and Dell serial BIOS' may or may not be useful:

Defined As     F1     F2     F3     F4     F5     F6     F7     F8     F9     F10    F11    F12
Keyboard Entry <ESC>1 <ESC>2 <ESC>3 <ESC>4 <ESC>5 <ESC>6 <ESC>7 <ESC>8 <ESC>9 <ESC>0 <ESC>! <ESC>@

Defined As     Home   End    Insert Delete PageUp PageDn
Keyboard Entry <ESC>h <ESC>k <ESC>+ <ESC>- <ESC>? <ESC>/

Use the <ESC><Ctrl><M> key sequence for <Ctrl><M>

Use the <ESC><Ctrl><H> key sequence for <Ctrl><H>

Use the <ESC><Ctrl><I> key sequence for <Ctrl><I>

Use the <ESC><Ctrl><J> key sequence for <Ctrl><J>

Use the <ESC><X><X> key sequence for <Alt><x>, where x is any letter key, and X is the upper case of that key

Adding Remote Logging of the Serial Output

Interactive use of the SOL consoles is very useful, but the addition of unattended logging of the output is even more useful - it can be used to catch kernel panics, and pre-cleaned kernel output, in the case that a box is compromised, two pick to examples.

The conserver program can carry out this kind of logging for other types of serial consoles, and with the addition of the "solsession" expect script from the IPMI_on_Debian_files/ directory - it can be made to speak to the Intel dpcproxy program. An example conserver configuration is also provided.

Conserver will also automatically restart dead dpcproxy connections, and continue to log output during interactive use (unless you tell it not to).

Setting up the IPMI watchdog

In this context, a watchdog is a device which has the job of reseting a computer system, if it thinks the software running on the system has hung, or is otherwide not operating as it should. A watch dog is often implemented as follows.. The software running on the computer system must carry out a regular task (such as writing a character to a device file, on Linux) in order to reassure the watchdog that everything is as it should be - if the software fails to carry out the task, the watchdog assumes that the computer has hung, and will reset it.

A watchdog is usually implemented so that it is likely to survive problems that might otherwise take out the operating system (a partial exception to this is the Linux "softdog" watchdog module, which runs in the kernel - it is still useful if the kernel is partially, but not entirely knackered). The IPMI standard includes a watchdog, and the OpenIPMI Linux drivers include a module which provides an implementation the Linux watchdog interface, which is backed by the IPMI BMC - such that it will reset (reset is the default behaviour) the computer if the watchdog device is not attended to in a timely fashion.

A simple recipe for setting up the IPMI watchdog on Debian/Sarge is presented below:

Note that the default time-out for the ipmi_watchdog module is 10 seconds, this is also the default write interval for the watchdog daemon which is included in the Debian "watchdog" package, so make sure you change one of them, otherwise there will be no margin for error at all (in the example, I've upped the kernel module time-out).

TODO

The IPMI system event log will currently (AFAIK) fill up, and stop being appended to (or maybe old entries will be nuked, I am not sure, as it hasn't happened yet..). Ipmitool will allow you to clear the log, so the cron jobs should probably be modified to do this, and archive old entries under /var/log (so that e.g. logrotate cat take care of them). Contributions welcome!

Credits

The original version of this document was written by Tim Small for a client of WPAD Ltd. - Semantico Ltd. Semantico support the publishing this document as part of their backing of open source software. Thankyou, Semantico.

Thanks also to:

Feedback

Please send feedback, patches (e.g. remote modem support), etc. to tim@buttersideup.com