Serial Console Server for the Poor I
The serial port is still the way to access network components out of band. It is slow, but reliable, and remarkably well standardized. It does not have technical whiz-bangs that can fail when one needs things to just work. That makes it the natural way to access critical infrastructure and still being sure that this access vector still works when most other things are down.
Every communication link has two sides, so there is a market for devices with a network link and a bigger number of serial ports to connect the actual devices to. Commercial vendors have a broad choice of serial console servers. Most of them, especially the small products with five to ten ports, are quite expensive, so I have been investigating how do build a serial console server with el cheapo hardware.
USB comes to mind here, of course.
When the UMTS card is plugged in, the ttyUSBx device nodes appear in the order of plugging in the devices. When the system is booted, the order is not very predictable, so one needs - again - udev to give predictable device node names.
udevadm's info command is an important tool to obtain the data set of an USB device which is available to match a device and to act appropriately when it is connected. udevadm info info sorts the devices into a tree and shows the relationship of the devices, which device is "behind" which other device from the system's point of view. udevadm info's output starts at the leaf of the device tree and shows the information for each node up the tree until its root is reached.
will give you the information you need to write an udev rule for your serial interface.udevadm info --name=/dev/ttyUSB0 --attribute-walk
One pitfall is that one can only match against the leaf and _one_ other node more up the tree. So you cannot do matches like "the pl2303-based device connected to port 3 on hub 2-3 will be designated USBserial3". If you try this, your rule will silently be ignored. This is a major turn-off in udev's implementation. I didn't try it on sid, but etch definetely shows this behavior. So one can only match like "the tty on the USB subsystem connected to port 3 on hub 2-3 will be designated USBserial3", but that's probably as good as it'll get.
The cause for my hub having only seven ports clearly shows itself when one looks at output of udevadm info of the fully equipped USB hub: The adapters are connected to ports 1, 2, 3, 4.1, 4.2, 4.3 and 4.4. The seven-port hub is actually two four-port hubs in a single case, with the second one connected to the first one's port 4. This reflects itself, of course, in the udev rules necessary to address the adapters by the hub port they're plugged in:
These rules will only work for a similary structured hub that is directly connected to the notebook. If there are more hubs in between the "final" hub and the host, the address matched against in the KERNELS matches are going to have a different structure. In the test setup, the first two parts of the address needed to be wildcarded since these addresses keep changing when the host is booted or the hub plugged out and in. So I guess that these udev rules are probably going to fail miserably when more hubs and/or USB-to-serial adaptors are connected.SUBSYSTEM=="tty", SUBSYSTEMS=="usb", DRIVERS=="usb", \\ KERNELS=="*-*.1", \\ SYMLINK="USBserial1" SUBSYSTEM=="tty", SUBSYSTEMS=="usb", DRIVERS=="usb", \\ KERNELS=="*-*.2", \\ SYMLINK="USBserial2" SUBSYSTEM=="tty", SUBSYSTEMS=="usb", DRIVERS=="usb", \\ KERNELS=="*-*.3", \\ SYMLINK="USBserial3" SUBSYSTEM=="tty", SUBSYSTEMS=="usb", DRIVERS=="usb", \\ KERNELS=="*-*.4.1", \\ SYMLINK="USBserial4" SUBSYSTEM=="tty", SUBSYSTEMS=="usb", DRIVERS=="usb", \\ KERNELS=="*-*.4.2", \\ SYMLINK="USBserial5" SUBSYSTEM=="tty", SUBSYSTEMS=="usb", DRIVERS=="usb", \\ KERNELS=="*-*.4.3", \\ SYMLINK="USBserial6" SUBSYSTEM=="tty", SUBSYSTEMS=="usb", DRIVERS=="usb", \\ KERNELS=="*-*.4.4", \\ SYMLINK="USBserial7"
But if one does not do too many changes, things are just fine, and it is possible to put numbers on the USB hub ports and be sure that the USB-to-serial adaptor connected to port 4 is going to be /dev/USBserial4 in the host system. I tried plugging in and out the adaptors in random order, unplugged and plugged the hub itself, booted the host, and the relationship between the port and the device node was stable.
If you have an USB 2.0 hub connected to an USB 2.0 port of the host, you might see error messages like "pl2303_open - failed submitting interrupt urb, error -28" in your log, while the serial ports plagued by this message are not going to work. This is due to an issue in the USB 2.0 code in the kernel (see the thread starting with my message on linux-kernel), Greg KH says that running USB 1.1 devices behind a USB 2.0 hub is a very difficult thing to do. A possible remedy is using an USB 1.1 hub, which is quite hard to obtain these days. But one can also unload the EHCI driver, disabling USB 2.0 completely, or do
echo P >/sys/class/usb_host/usb_hostB/companionto force port P of the EHCI controller on Bus B to run at low/full speed (so, Alan Stern says in the linux-kernel thread). After a few tries, I found out that my test notebook needs P=1 and B=1 for the correct port. If you have found the correct port, you'll see your devices disconnect and reconnect below the USB 1.1 root hub in lsusb and the urb error messages should be gone.
In the test environment, I only tested one adapter at a time, but all these tests were successful as well. I guess that many more of these adaptors will eventually need a powered USB hub, but with seven adaptors on a single hub, the power from the USB host is sufficient to live without extra power on the hub. And, I guess, there is some hard limit for the number of USB adapters being used, and some soft limit when managing the ports and adapters is going to get harder. But, I think that a site in need of more than, say, sixteen serial ports on the console server can afford a "real" console server from one of the major vendors, which can be accessed via ethernet, UMTS, ISDN and whatever you can think of.
This concludes the hardware part of my serial console server for the poor. Tomorrow, I'll blog about the software side needed on the host to allow the serial ports to be accessed from the network while still being reasonably secure. The next article will involve ssh, /etc/passwd shells, /etc/shells, a small perl script, ser2net and telnet. And, shell escape is the cue word for grey hairs sprouting on my head.
UPDATE 2015: give udevadm info command line for my own laziness.
Comments
Display comments as Linear | Threaded
Michael Greb on :
Don't know you've heard of Open Gear, we've been using them for at least a couple of years at work. Much cheaper than some of the bigger names. The software side of things is Linux based. They have a single port console server, don't know why you would want just one port, for $295 and an 8 port for $495. They go up to 48 ports at $1495. I thought more of the core software was open source but am just seeing http://sdtcon.sourceforge.net/ at the moment.
Philipp Kern on :
How do you handle power cycles? I still look for a power strip usable from Linux...
Marc 'Zugschlus' Haber on :
Not yet at all.
The APC Masterswitch was useable from Linux five years ago, though.
Jay Vaughan on :
Nice article .. just wanted to let you know I've done the same thing, albeit instead of using USB-Serial adapters, I'm using USB-MIDI adapters .. which are, of course, the same hardware, just coded to run at a different baudrate (31250 for MIDI) and so on ..
My $500 USB-MIDI "pro rack" solution has just been replaced by $70 worth of parts which work flawlessly!