Zugschlusbeobachtungen (Entries tagged as english)

perl and caches

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Mon, 20 Jun 2011 14:02:31 +0200

From the perl DBI manual page:

If you’d like the cache to managed intelligently, you can tie the hashref returned by “CachedKids” to an appropriate caching module, such as Tie::Cache::LRU

And what happens when I don’t do this? Will my cache be unintelligently managed then, with the consequence of my machine exploding when the cache is filled with more than a handful entries?

How much added complexity in packages to cater for apt's shortcomings?

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Sat, 23 Jan 2010 12:08:00 +0100

It is well known that apt has an issue when it comes to resolving circular dependencies. Therefore, Debian bug reporters have set out to eradicate circular dependencies from the archive. This does, however, add significant bloat to the actual packages, and I am questioning why this is really necessary.

For example, aide. aide has an aide-common package, depending on aide | aide-binary. aide-binary is a virtual package by aide, aide-dynamic, and aide-xen. All three providers of aide-binary depend on aide-common again, which is a circular dependency.

If I now follow the request posed in #545852 and remove aide-common’s dependency on aide | aide-binary, I’ll have to cater for the case of the aide-binary bein absent in each and every helper script included in aide-common, which would drive the script’s complexity up to a new level. I really hate that idea.

Is it really necessary to work around apt’s shortcomings by eliminating the case that apt doesn’t handle well from the archive, and driving up other packages’ complexity (and the possibility of bugs in this added complexity)?

Block devices in KVM guests

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Fri, 22 Jan 2010 14:08:00 +0100

In the last few days, I found the time to spend some with KVM and libvirt. Unfortunately, there is a subject that I haven’t yet found a satisfying solution: Naming of block devices in guest instances.

This is surely a common issue, but solutions are rare. Neither an article on Usenet (in German) nor the German version of this blog article has found solutions for the main question. I should have written this in English in the first place and am thus translating from German to english, hoping that there will be some answers and suggestions.

KVM is quite inflexible when it coms to configure block devices. It is possible to define on the host, which files or whole devices from the host should be visible in the guest. The documentation suggests that devices should be brought into the guest with the virtio model, which needs suppport in the guest kernel. Importing a device as emulated ATA or SCSI device brings a performance penalty.

The devices brought into the guest via virtio appear in the guest’s dev as /dev/vd<x> and do also have their corresponding entries in /dev/disk/by-uuid and /dev/disk/by-path. The vd<x> node is simply numbered in consecutive order as hd<x> and sd<x>. /dev/disk/by-uuid is the correct UUID of the file system found on the device, at least if it’s a block device partitioned inside the guest and formatted with ext3 (I didn’t try anything else yet). The terminology of the /dev/disk/by-path node is not yet understood, and I am somewhat reluctant to assume the PCI paths of emulated hardware as stable.

Looks like I am forced to configure inside the guest when I have changed the host’s mass storage system (for example, after migration to a different file system or after new block devices have been added) to accommodate for the new order of the /dev/vd<x> or to have the UUID correctly configured. This is like calling for configuration errors.

This is a reincarnation of the same issue that has been killed by LVM on Linuxes running on “bare metal”: The block device itself has a mnemonic name which is constant even in migration and copy actions. This works without file-system specific stuff like UUID or label, and wouldn’t even be possible with a UUID (which won’t be unique in this case). The mnemonic names of LVs are also available when the data is directly written to the raw device such as a CNFS buffer of a news server.

I would love to have something like a “paravirtualized device mapper interface” which would allow to say in the host’s configuration which of the host’s LVs should be visible in the guest with which name in /dev/mapper. That way, the guest’s configuration could remain stable during data wrangling operations on the host.

One solution is to have a single LV for each guest and import this LV as /dev/vda into the guest. /dev/vda would then be partitioned like a real disk and have its own LVM installation. This would however need kpartx if one wants to access the data from the host, and loses flexibility when resizing file systems.

These issues appear in every installation of KVM virtualization, and I would expect that there are gazillions of other possible solutions. I am interested in knowing how other people have tackled this issue and whether there are more possiblity that I haven’t thought of before. Maybe there is a solution that doesn’t leave me with the feeling of having implemented something ugly. Does the interface between the host’s KVM and the guest’s device mapper that I have been dreaming of maybe exist. Or is there any other possibility of configuring the device node’s name in the guest Linux on the host side?

TCP and mobile IP

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Fri, 04 Sep 2009 16:00:11 +0200

Steinar H. Gunderson, sesse, has written an interesting article about TCP performance. I didn’t find your blog’s comment function, so I am commenting with a trackback. (note: which didn’t work either, “The auto-discovered trackback URI does not match our target URI”)

I frequently use mobile internet, using various of the German GSM/UMTS network operators, out of a moving train. As you have written, this frequently causes packet loss which is not only not caused by congestion, but sends the congestion avoidance algorithms on a false path.

For example, when the train passes through the 3575 m long Distelrasentunnel between Frankfurt and Fulda, my network link is broken for like two minutes. Passing through other parts of Germany sometimes gives me a ping response of hundreds of thousands of microseconds by virtue of the rather huge send buffer the UMTS equipment has.

In these circumstances, ssh sessions frequently take tens of minutes to notice that the network is back before the session is useable again. Frequently, it doesn’t come up again before an hour has passed. And I have not found a way to work myself around this. Can you explain what’s happening here, and do you have any ideas to solve the issue?

The grub drama

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Sat, 30 May 2009 15:22:00 +0200

This is a rant. A rant which goes to the grub maintainers, and one that could go nearly identically to many people in the KDE environment or many other open source projects.

I really like grub. I really like grub 0.97 despite that it’s been unmaintained for years and not booting on two of my important machines. I should like grub 2 because its configuration looks more straightforward and for its better features - direct booting of .iso images, from LVM and RAID. But actually, I have learned to hate grub 2 since it is not finished and badly documented, and that its existence is already being used as an excuse for grub 0’s development having stopped years ago (and it being renamed to “grub-legacy” to clearly show that it’s the unloved child) - and things looks like this is not going to change any time soon.

grub 0.97 is the de facto standard in booting Linux, its most prevalent advantages probably being its capability of reading filesystems and its command shell, which allows booting kernels and other software that one wasn’t planning to boot when the system was shut down. There is a small fraction of people who still use lilo despite its disadvantages, for example on achitectures that are not supported by grub, if one wants to have the boot FS on LVM, if one just prefers lilo for non-technical reasons or on systems that grub doesn’t work on.

The last three systems, by the way, that I still run lilo on are Pyramid pizzaboxes with supermicro-made mainboards from 2002, which just refuse to boot if grub is used and the vendor just says that grub 0.97 is still beta and that they do not support grub on their systems: “Use Lilo.”

Grub’s last upstream release, 0.97, dates back to 2005, and upstream has stopped developing this grub release years ago. grub 2, the new way to go, had its first release in 2003, and the last “stable” release that people are supposed to use dates back to februar 2008. grub 2 is clearly a step in the right direction, but there are still bugs, and the most prevalent show stopper against grub 2 is the almost complete absence of any documentation. I have, for example, not yet found good documentation about the new config file format, which is a completely different deal from what we are used to with grub 0. The commands’ manual pages do little more than list the (obvious!) command line options without giving much more information about what the actually do, let alone usage examples. The documentation (which used to be really excellent for grub 0), has been “moved” to a wiki, which has some useful content but cannot keep up with the excellent documentation that grub 0 used to have.

Additionally, grub2 does not seem to have a grub shell which can be called from within the host system, which makes installing grub 2 in “exotic” situations much harder than it used to be with grub 0.

I am also quite disappointed that despite the grub upstream web pages claiming that grub 2 is under active development, the last release is more than fifteen months in the past while there are still so many shortcomings. The Debian maintainers of grub 2 are regularly packaging grub 2 subversion snapshots, which is a good idea in face of upstream’s release policy which is quite different from what’s being told in the wiki. I am, however, reluctant to rely on a development snapshot of a software to boot productive systems.

Guys, you cannot declare a widely used, well-documented software legacy when the replacement is still in a nearly unuseable shape and the docs are hidden in the helpful files named .c and .h. This is simply unacceptable. It’s a drama that the docs are still in such a pitiful shape fifteen months after your last release.

But, grub is not the only free software project that so blatantly scores developer fun over the needs of the end user. I know that it is a boring task to fix bugs and to work around design shortcomings in the existing version of software, and that it is much more fun to hack away on new designs and new code, but dropping old versions before the new version is up to par with the old one is annoying the users and driving people away to other software packages. This applies to grub, a quite low-level program, but also to larger and more complex projects like KDE who has recently pulled the same stunt by dropping development and support for KDE 3.5 while KDE 4 is still missing many many features and KDE 3.5’s stablity, thus driving people (back) to GNOME or alternative desktop environments. This “we give a damn about people who actually use our software, our compilers run without users as well” attitude is a show of disrespect for the user base and makes me very sad as this does indeed show that free software is in no way superior to its commercial counterparts.

Sadly, I do not have a solution for this issue since I know that free software developers are usually volunteers who do not have to report to anybody, and that they do not need to care for their user base. But I hope that people begin to reconsider their attitude and think about what kind of horrible damage is being done to the user base and to free software’s reputation.

Pushing a packet back and forth between Linux subsystems

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Sun, 10 May 2009 10:01:04 +0200

Linux policy routing is still incredibly painful if one wants to have more sophisticated routing than just “take source and destination IP address for the routing decision”. The mechanisms that have been in use seven years ago still work though, and I didn’t find any possibility to do it any easier. In this article, I’ll try to explain the “old” mechanisms and hope that somebody from lazyweb will comment and say “it can be done so much easier”.

This is a translation of the Usenet article <gu48cs$rul$1@news1.tnib.de> in de.comp.os.unix.networking.misc in the hope that the english-speaking blogosphere can give additional insights.

Given a Linux-based router with one internal network (int0), one perimeter network (per0) and two Internet connections (ext0, ext1) with one IP address each. We need to do source NAT to deliver Internet to the internal and perimeter networks. The internet connection on ext0 will be used for http and https, while all other traffic needs to go out on ext1. The perimeter network is reachable from the outside via destination NAT.

1: ext0:  mtu 1500 qdisc noqueue
    inet 10.81.221.145/29 brd 10.1.221.151 scope global ext0
2: ext1:  mtu 1500 qdisc noqueue
    inet 10.82.83.225/29 brd 10.82.83.231 scope global ext1
3: per0:  mtu 1500 qdisc noqueue
    inet 172.16.1.254/24 brd 172.16.1.255 scope global per0
4: int0:  mtu 1500 qdisc noqueue
    inet 192.168.8.254/24 brd 192.168.8.255 scope global int0

We have the following routing rules:

0:      from all lookup 255
10:     from all lookup main
32:     from all fwmark 0x12e lookup to_ext0
32:     from all fwmark 0x12f lookup to_ext1
32000:  from all lookup defaultroute
32000:  from all lookup defaultroute
32766:  from all lookup main
32767:  from all lookup default

Table main contains all rules for the directly connected networks, but no default route. Both to_ tables have one default route each, pointing to the respective ISP, and the table defaultroute has once more a default gateway pointing to ext1’s gateway.

The PREROUTING chain of the mangle table has these rules:

iptables --protocol tcp --dport 80 --in-int int+ --set-mark 0x12e
iptables --protocol tcp --dport 443 --in-int int+ --set-mark 0x12e
iptables --in-int int+ --set-mark 0x12f

and the POSTROUTING chain of the nat table has these:

iptables --match mark --mark 0x12e --out-int ext0 --jump SNAT --to-source 10.81.221.145
iptables --match mark --mark 0x12f --out-int ext1 --jump SNAT --to-source 10.82.83.225

If now somebody from the internal network accesses a web server on the Internet. The outgoig packet will first be marked in the PREROUTING chain, then the routing rules will use the firewall mark to consult the correct routing table and pass the packet back to the packet filter, where the POSTROUTING chain will use the outgoing interface and the firewall mark to SNAT the packet to the correct source IP so that the answer will actually reach us.

Has a less ugly way available way to do this become available in the last five years? Or is it still necessary to have the different parts of the networking subsystems play ping-pong with the packet? I particularly dislike that someone not familiar with policy routing will not see any default route in the routing table printed by “route” or “ip route”, which will probably be very confusing.

Is there any possibility to interact with the routing code without firewall marks, maybe by directly setting a gateway from a firewall rule?

Serial Console Server for the Poor III

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Mon, 30 Jun 2008 15:00:26 +0200

This is the third installment of my article about the Serial Console Server for the Poor. First installment here, Second installment here.

The first part of the article having covered the hardware and the udev part creating the device nodes, and the second part explaining how to solve the software part using ser2net, this part explains why ser2net was ditched in favor of cereal and how the console server operates with cereal now.

Cereal (which I was pointed to on IRC as a reaction to article II) is a nifty combination of runit’s runsv and screen. One of the less-known features of screen is that screen can also connect to a serial port instead of a “real” console. cereal makes use of this feature, putting up with the fact that screen can only set a baud rate. I have not seen a serial application with other parameters than 8n1 (8 data bits, no parity, 1 stop bit) in a decade, so I cannot comment whether this is a real disadvantage. I guess that it might be possible to set the serial port to the desired set of parameters before starting up screen, but I didn’t actually try it since I do not need it.

Cereal uses a set of configuration parameters and processes called a “session” to associate a serial port with a baud rate and a set of user/group settings to control who is able to start, stop, attach or detach a session. It doesn’t have a “real” configuration file. Instead, it has an admin program, cereal_admin, which takes the parameters associated with a session and writes them to an (intentionally? undocumented?) set of config files in /var/lib/cereal. Only a single user is able to attach to a session read/write, while an entire group can attach to a session read-only and can thus access the logs. A session is automatically started when the sysem boots and can be stopped and started using cereal_admin as well. Once a session is running, a specially configured screen process runs attached to the serial port, and users simply attach and detach to and from it as one would to a normally running screen. That way, one can scroll back and examine what was shown on the serial port while nobody was attached. This is quite handy when the device connected to the port prints out warnings, errors and status information to its serial port.

The screen configuration that is used is rather sophisticated. So, a permanent log file is written, which is also used for read-only access to the serial session. Also, a very informative status line is configured.

Unfortunately, one of screen’s biggest disadvantages applies here as well: User X cannot attach to a session started by User Y unless Y has access privileges for X’s pty (or vice versa, I never can remember), which can be a security risk. Additionally, it is not possible to su or sudo to X and then to attach to the session - one needs to have the original pty opened by the correct user, which usually means that one needs to do a “real” log in with the correct account (via /bin/login or sshing in to the box).

If one has the privileges necessary, one can use the cereal executeable to attach to the screen (which gives read-write access) or to tail -F the log file (giving timestamped read-only access). That way, one never knows that one is actually working with an appropriately configured screen that is kept running by runit.

The restrictions coming with cereal suggest a setup similiar to what I did with ser2net previously: Each session has its own user account, which forces an attach to the appropriately named cereal shell on login:

my-device-name:x:1801:1801:my-device-name,,,:/home/my-device-name:/usr/local/sbin/cerealshell

$ cat /usr/local/sbin/cerealshell
#!/bin/bash

set -eu

cereal attach $(id --user --name)
$

You can see that I decided to name the sessions after the actual device name, allowing the unterlying ttyUSB device name to change without change in the user interface. The actual users and sessions were created with a small shell script:

#!/bin/bash

createconsole() {
  NAME=“$1”
  PORT=“$2”
  BAUD=“$3”
  MUID=“${PORT/USBserial/180}”
  echo sudo addgroup --force-badname --gid $MUID $NAME
  echo sudo adduser --force-badname --gecos “$NAME,,,” --ingroup $NAME --uid $MUID --disabled-password
$NAME
  echo sudo adduser $NAME dialout
  echo sudo mkdir -p /home/$NAME/.ssh
  echo sudo chmod 755 /home/$NAME/.ssh
  echo sudo touch /home/$NAME/.ssh/authorized_keys
  echo sudo chmod 644 /home/$NAME/.ssh/authorized_keys
  echo sudo chown $NAME:$NAME /home/$NAME/.ssh /home/$NAME/.ssh/authorized_keys
  echo sudo cereal-admin create $1 /dev/$2 $3 $1 $1
  echo sleep 2
    echo sudo cereal-admin start $1
  echo echo -----
}

createconsole my-device-1 USBserial1 9600
createconsole my-device-2 USBserial3 115200

Note that the script also uses a hardcoded UID range which might need adaption to your local needs (and breaks for more than ten serial ports), and that it also takes care of creating an empty authorized_keys file with correct permissions.

That way one can simply ssh to my-devlce-1@consoleserver and find oneself immediately connected to the serial console, seeing the last things that happened on the serial port, and immediately being able to work with the device. That’s fine, comfortable and useable. I actually like that better than what I built with ser2net previously.

kbd seems to be the way to go

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Sat, 21 Jun 2008 15:48:35 +0200

This is just a small reminder (for me and others) that Debian is currently migrating from console-tools to kbd (back again, yes, those who have been around for a few years remember).

This information is obviously a closely-guarded secret. Console-tools is still Priority: important, and kbd is still Priority: extra. However, kbd seems to be much better maintained (current uploads happening, while console-log has seen its last maintainer upload two years ago), and unfortunately, neither package description suggests which package is the way to go. And Debian-installer still installs console-tools by default.

However, a few bugs were filed a year ago by the console-tools maintainer to drop console-tools from depends as console-tools is going away. So I guess that he knows what he’s doing...

Before I get around to adding console-tools back to console-log’s depends (as I almost did accidentally), I’ll better blog this to remind people of console-log going away. Maybe we’ll get the Priorities changed just in time for lenny.

Serial Console Server for the Poor II

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Thu, 05 Jun 2008 22:51:19 +0200

This is the second installment of my article about the Serial Console Server for the Poor. First installment here.

The last part of the article having covered the hardware and the udev part creating the device nodes, this part addresses the part of the software that connects the user to the device node.

The idea of the software side is to allow users access only to certain serial ports without giving out a shell account on the server itself. Communication between user and the console server should be encrypted (which certainly means ssh), and users should be able to authenticate both with a key and with a password. Serial parameters (baud rate, parity, stop bits etc) should be configured on the console server to keep this possible error source away from the users.

For years, I have been thinking about using UUCP’s cu to access the serial port. This time, it’s not going to be used because it offers a shell escape which cannot be turned off. That would give a user shell to the users indirectly, which is not what I want.

Minicom seems still to be the most-used application to access a serial port. I have never understood the sense of using a terminal emulator inside a terminal emulator, especially if one has to break the terminal emulator’s habit of trying to initialize a non-existent modem.

So I settled on the application that I usually use to access a serial port: ser2net. This is a Linux implementation of Cisco’s “reverse telnet” feature which “connects” a TCP port to a serial port with a given set of parameters. For example,

4016:telnet:600:/dev/USBserial6:38400 8DATABITS NONE 1STOPBIT banner

connects TCP port 4016 with /dev/USBserial6 with an 38400 8n1 parameter set. If you telnet to localhost 4016, you’ll find yourself talking to the device connected to the appropriate serial port.

This moves the security risk to the telnet client, which - unfortunately - offers a shell escape function from the telnet command line reachable with the escape character. The only way to remove this is to use the -E command line option, which turns off the escape function completely. This unfortunately excludes the user from cleanly quitting the telnet session, so the only way to get out of a telnet -E is to have the remote side close the serial port (which ain’t happening on a serial console) or to kill the telnet process, for example by closing the carrying ssh session. That’s a turn-off, but one that one can live with.

Now, the only challenge left is to have the appropriate telnet client started automatically for the user. I decided on connecting this to the account being connected to, so that ssh USBserial4@hostname will automatically do the right thing. Since password authentication needs to work (optionally), this precludes the use of the command modifier in an authorized_keys file.

Another possibility to force a command being executed on login is to set it as the users’ login shell in /etc/passwd. One drawback: The command must not have any parameters, so one needs to pass needed parameters to the program some other way. Additionally, the program must be listed in /etc/shells or it will be silently ignored.

My /etc/passwd entries look like

USBserial2:x:1002:1002:USBserial2,,,:/home/USBserial2:/usr/local/sbin/serialconnect-shell

with /usr/local/sbin/serialconnect-shell looking like

#!/bin/bash

set -eu

case “$(id --user --name)” in
  USBserial1) PORT=“2011”;; 
  USBserial2) PORT=“2012”;;
  USBserial3) PORT=“6013”;;
  USBserial4) PORT=“6014”;;
  USBserial5) PORT=“2015”;;
  USBserial6) PORT=“2016”;;
  USBserial7) PORT=“6017”;;
  *) echo “called from unknown account, terminating.”; exit 1;;
esac

telnet -E localhost $PORT

Just for reference, here are the appropriate entries in /etc/ser2net.conf:

2011:telnet:600:/dev/USBserial1:9600 8DATABITS NONE 1STOPBIT banner
2012:telnet:600:/dev/USBserial2:9600 8DATABITS NONE 1STOPBIT banner
6013:telnet:600:/dev/USBserial3:115200 8DATABITS NONE 1STOPBIT banner
6014:telnet:600:/dev/USBserial4:115200 8DATABITS NONE 1STOPBIT banner
2015:telnet:600:/dev/USBserial5:9600 8DATABITS NONE 1STOPBIT banner
2016:telnet:600:/dev/USBserial6:9600 8DATABITS NONE 1STOPBIT banner
6017:telnet:600:/dev/USBserial7:115200 8DATABITS NONE 1STOPBIT banner

If one wants to authenticate for a given serial port with an ssh key, all you need to do is put the appropriate key into the authorized_keys file of the appropriate account, and you’re all set.

I do sincerely hope that I didn’t put any blatant security goofs into this setup, but if I did, you’ll tell me in the comments, right?

Just one more challenge for my valued readers: The port numbers in my ser2net.conf have a system. Can you explain it to me?

Serial Console Server for the Poor I

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Tue, 03 Jun 2008 23:41:47 +0200

The serial port is still the way to access network components out of band. It is slow, but reliable, and remarkably well standardized. It does not have technical whiz-bangs that can fail when one needs things to just work. That makes it the natural way to access critical infrastructure and still being sure that this access vector still works when most other things are down.

Every communication link has two sides, so there is a market for devices with a network link and a bigger number of serial ports to connect the actual devices to. Commercial vendors have a broad choice of serial console servers. Most of them, especially the small products with five to ten ports, are quite expensive, so I have been investigating how do build a serial console server with el cheapo hardware.

USB comes to mind here, of course.

Picture of the fully equipped USB hub

The hardware side is actually quite easy: A seven-port USB hub, and seven USB-to-serial adaptors (using the widely-deployed Prolific PL2303) are easily purchased for well below a hundred Euros, and naively connected. In my test setup, connected to the hp compaq nc8000 that my blog readers should know only too well by now, the hub does not even need to have its power supply connected. The notebook can power the hub and the seven adaptors just fine.

When the UMTS card is plugged in, the ttyUSBx device nodes appear in the order of plugging in the devices. When the system is booted, the order is not very predictable, so one needs - again - udev to give predictable device node names.

udevinfo is an important tool to obtain the data set of an USB device which is available to match a device and to act appropriately when it is connected. udevinfo sorts the devices into a tree and shows the relationship of the devices, which device is “behind” which other device from the system’s point of view. udevinfo’s output starts at the leaf of the device tree and shows the information for each node up the tree until its root is reached.

One pitfall is that one can only match against the leaf and _one_ other node more up the tree. So you cannot do matches like “the pl2303-based device connected to port 3 on hub 2-3 will be designated USBserial3”. If you try this, your rule will silently be ignored. This is a major turn-off in udev’s implementation. I didn’t try it on sid, but etch definetely shows this behavior. So one can only match like “the tty on the USB subsystem connected to port 3 on hub 2-3 will be designated USBserial3”, but that’s probably as good as it’ll get.

The cause for my hub having only seven ports clearly shows itself when one looks at the udevinfo outputs of the fully equipped USB hub: The adapters are connected to ports 1, 2, 3, 4.1, 4.2, 4.3 and 4.4. The seven-port hub is actually two four-port hubs in a single case, with the second one connected to the first one’s port 4. This reflects itself, of course, in the udev rules necessary to address the adapters by the hub port they’re plugged in:

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“usb”, \
   KERNELS==“*-*.1”, \
   SYMLINK=“USBserial1”

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“usb”, \
   KERNELS==“*-*.2”, \
   SYMLINK=“USBserial2”

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“usb”, \
   KERNELS==“*-*.3”, \
   SYMLINK=“USBserial3”

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“usb”, \
   KERNELS==“*-*.4.1”, \
   SYMLINK=“USBserial4”

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“usb”, \
   KERNELS==“*-*.4.2”, \
   SYMLINK=“USBserial5”

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“usb”, \
   KERNELS==“*-*.4.3”, \
   SYMLINK=“USBserial6”

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“usb”, \
   KERNELS==“*-*.4.4”, \
   SYMLINK=“USBserial7”

These rules will only work for a similary structured hub that is directly connected to the notebook. If there are more hubs in between the “final” hub and the host, the address matched against in the KERNELS matches are going to have a different structure. In the test setup, the first two parts of the address needed to be wildcarded since these addresses keep changing when the host is booted or the hub plugged out and in. So I guess that these udev rules are probably going to fail miserably when more hubs and/or USB-to-serial adaptors are connected.

But if one does not do too many changes, things are just fine, and it is possible to put numbers on the USB hub ports and be sure that the USB-to-serial adaptor connected to port 4 is going to be /dev/USBserial4 in the host system. I tried plugging in and out the adaptors in random order, unplugged and plugged the hub itself, booted the host, and the relationship between the port and the device node was stable.

If you have an USB 2.0 hub connected to an USB 2.0 port of the host, you might see error messages like “pl2303_open - failed submitting interrupt urb, error -28” in your log, while the serial ports plagued by this message are not going to work. This is due to an issue in the USB 2.0 code in the kernel (see the thread starting with my message on linux-kernel), Greg KH says that running USB 1.1 devices behind a USB 2.0 hub is a very difficult thing to do. A possible remedy is using an USB 1.1 hub, which is quite hard to obtain these days. But one can also unload the EHCI driver, disabling USB 2.0 completely, or do

echo P >/sys/class/usb_host/usb_hostB/companion

to force port P of the EHCI controller on Bus B to run at low/full speed (so, Alan Stern says in the linux-kernel thread). After a few tries, I found out that my test notebook needs P=1 and B=1 for the correct port. If you have found the correct port, you’ll see your devices disconnect and reconnect below the USB 1.1 root hub in lsusb and the urb error messages should be gone.

In the test environment, I only tested one adapter at a time, but all these tests were successful as well. I guess that many more of these adaptors will eventually need a powered USB hub, but with seven adaptors on a single hub, the power from the USB host is sufficient to live without extra power on the hub. And, I guess, there is some hard limit for the number of USB adapters being used, and some soft limit when managing the ports and adapters is going to get harder. But, I think that a site in need of more than, say, sixteen serial ports on the console server can afford a “real” console server from one of the major vendors, which can be accessed via ethernet, UMTS, ISDN and whatever you can think of.

This concludes the hardware part of my serial console server for the poor. Tomorrow, I’ll blog about the software side needed on the host to allow the serial ports to be accessed from the network while still being reasonably secure. The next article will involve ssh, /etc/passwd shells, /etc/shells, a small perl script, ser2net and telnet. And, shell escape is the cue word for grey hairs sprouting on my head.

Works with a more recent card as well

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Tue, 03 Jun 2008 17:16:17 +0200

Today, I had the opportunity to try my UMTS initialization mechanism that I built this weekend with more recent hardware, a newer Option Globetrotter 3G Express Card with Vodafone branding (reporting itself to be a “Globetrotter HSDPA Modem” with Vendor ID 0xaf0 and Product ID 0x6701). To get the card connected to my test Notebook, a hp compaq nc8000, I had a “Expresscard in a PC card slot” adapter and a passive “Expresscard at a normal USB port” adapter. The USB adapter had cost about ten Euros, and I don’t imagine the PC card adapter to be much more expensive.

The later Option card is recognized as three USB-connected serial ports by the Option kernel driver (CONFIG_USB_SERIAL_OPTION) as the older one is, and it works just fine. The only difference is that the udev rule needs a different value for the ATTRS{modalias} setting. My umts-pin script complains about an “illegal seek” after entering the PIN, but the card registers itself with the network nevertheless. Both gammu to send SMS and pppd for IP connectivity worked out of the box.

Especially the “Expresscard-to-USB” adapter setup is very sexy for setups where neither an Expresscard nor a PC Card slot is available, and USB cables can be nice and long. So one could even mount the UMTS interface near the window and pull a longer USB cable to the actual system. I decided on putting the UMTS interface into a notebook, though, and mount the notebook near the window to be able to monitor the monitoring systems and send out SMS even when the whole datacenter is without power (by virtue of the notebook having a built-in UPS).

Automatic initialization of a Option 3G Datacard

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Sun, 01 Jun 2008 11:17:00 +0200

For mobile UMTS/GSM, I have been using an Option 3G Data Card for two and a half years now. I blogged about getting the card to work (in German, sorry) on Linux in July 2005. I never found the time - until now - to automate the card initialization so that I had been using a horrible chat script for card initialization when the PPP connection was built.

I recently took the time to automate this, so that the PIN is transmitted to the card automatically when the card is plugged in. This article documents what I did.

On a side note: Unfortunately, the vendors’ attitude towards Linux hasn’t changed since 2005. Their Hotlines still deny that their products can be used with Linux at all, and they surely do not publish any documentation that can be of help. Otoh, Vodafone has published a software that supposedly aids usage of their products under Linux. I haven’t tried it yet since it is not packaged yet for Debian. Additionally, Vodafone support media and sales do not seem to know about this effort, they still deny that their products work with Linux. Windows users happily install proprietary software products that do little more than sending a handful of AT commands to the emulated USB modem and hand over the connection to Windows’ PPP Stack. A very unsatisfying situation.

Just for the record: Dear Vodafone DE, a week ago you missed the sale of a new USB UMTS interface because you don’t even document it on Linux. This motivated me to look into the drawer that holds the old, non-HSDPA PC cards that have been decommissioned at the customers’ site and use an old, used device. Your fault.

But now back on topic: For a data center monitoring system, I want to send out alerts as text message (for German speakers: SMS - we have a rather strange vocabulary for mobile communications), and am reluctant to do so via IP: IP is one of the things that can fail, so it is preferred to directly inject the text messages into the network of the operator that provides service to the recipients of the text messages. Fortunately, most of the recipients are in the same network.

After evaluating current hardware solutions that could be plugged directly into the “real” server doing the monitoring and running into the wall of non-support provided by the network operators and hardware vendors, I decided on taking things a little further and to build an Ethernet-Connected text message device. An old hp compaq nc8000 (no, not mine, mine still works fine and is in daily use) and an old, first generation Option Datacard 3G will be mounted near the datacenter’s windows and equipped with Debian and Nagios. That way, the text message device can itself monitor the monitoring system and even send out a warning message when the data center goes out of power: It has a built-in UPS.

But now back on topic: For the text messages to go out reliably, it is necessary that the UMTS card is initialized automatically when the system comes up. A great opportunity to finally address this issue for Linux, on customers’ expense **grins**.

UMTS card initialization is done in two steps: First, when the card is plugged in, the virtual OHCI host port appears. Initialization of the virtual OHCI happens automatically. Most documentation on the web says that you now need to load an appropriately parametrized usbserial module, and I did it this way for years, but that’s not necessary any more. CONFIG_USB_SERIAL_OPTION is a dedicated kernel module for the Option 3G data card, which gets automatically loaded if it is available.

If you don’t have the Option module for some reason, you still need the following udev rule to automatically load the generic USB serial module. I figured that out yesterday before I learned about the Option module in a comment made to the original version of this article.

SUBSYSTEM==“usb”, \
   SYSFS{idProduct}==“5000”, SYSFS{idVendor}==“0af0”, \
   ACTION==“add”, \
   RUN+=“/sbin/modprobe usbserial vendor=0x$attr{idVendor} product=0x$attr{idProduct}”

SUBSYSTEM==“usb”, \
   SYSFS{idProduct}==“5000”, SYSFS{idVendor}==“0af0”, \
   ACTION==“remove”, \
   RUN+=“/sbin/modprobe -r usbserial”

Unfortunately, the usbserial and option modules stay around after the card is pulled. This doesn’t hurt though.

Next, we see three virtual serial interfaces, which we can detect via udev, and assign “speaking” device names and transmit the PIN. This goes into /etc/udev/rules.d/50-option3g.rules:

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“option”, \
   ATTRS{bInterfaceNumber}==“00”, \
   ATTRS{modalias}==“usb:v0AF0p5000d0000dc00dsc00dp00icFFiscFFipFF”, \
   SYMLINK=“UMTS-DATA”

SUBSYSTEM==“tty”, SUBSYSTEMS==“usb”, DRIVERS==“option”, \
   ATTRS{bInterfaceNumber}==“02”, \
   ATTRS{modalias}==“usb:v0AF0p5000d0000dc00dsc00dp00icFFiscFFipFF”, \
   RUN+=“/usr/local/sbin/umts-pin --device %k --symlink UMTS-CONTROL”

I am not sure what the modalias attribute identifies, but it looks like it identifies the hardware model: An identical Option Datacard 3G, plugged into the other PC Card Slot gets properly initialized as well.

These two udev stanzas do two different things: First, they make sure that the new devices are symlinked to /dev/UMTS-DATA and /dev/UMTS-CONTROL, suggesting the intended use of the interface. The second virtual interface of the 3G Data Card does not seem to be useable at all, and you cannot build an actual data connection over the third. So, we baptize the first interface /dev/UMTS-DATA and ignore the second. The third gets called /dev/UMTS-CONTROL.

Now to the real magic: /usr/local/sbin/umts-pin is a perl script that sends the PIN to the Card. %k gets expanded to the device name of the device that has just been detected, so umts-pin gets called with “--device ttyUSB2”. Please note that this udev stanza does not have a SYMLINK clause, but instead umts-pin creates the symlink after sending the PIN to keep other processes from trying to grab the device before it was properly initialized.

It makes use of Cosimo Streppone’s Device::Modem module (Device::Gsm is not yet packaged for Debian and might be of better use here) to talk to the UMTS device. Unfortunately, I have not yet fully understood the logging and answer processing features of Device::Modem, so the implementation might look a little clumsy. I hope that it can be clearly understood anyway.

The actual PIN is read from a configuration file in an apache-like format, which is by default looked for in /etc/umts/pin.conf:

<pin>
        <default>
                pin xyzw
        </default>
</pin>

The reason I settled for this rather complex configuration file format is that the ultimate goal is to automatically detect which of my SIMs is currently inserted in the UMTS card and to automatically send the correct PIN. I haven’t been successful in doing so since I didn’t find an AT command yet to get the UMTS card to divulge the SIM serial number or the IMSI before the PIN is transmitted to the SIM. So currently, the only things that are supported are the “default” stanza and an identically formatted “override” stanza which takes precedence over the default stanza if present.

If the first attempt to send the PIN fails, the script tries again ten seconds later. On my test system, this happens about once in twenty times. That must be some weird timing issue. So, if you have the wrong PIN configured, you’ll need the PUK after plugging in the card for the second time, since the first try will eat two of your three attempts. Beware.

If you feel uncomfortable with all this scripting and the “quality” of my code, you can also use gammu’s entersecuritycode function. I discovered this after my PIN script was already written, and gammu currently forces you to expose the PIN on the command line (see #484102), and of course my script’s PIN handling is vastly superior **grins**.

Universal boot stick for Debian, grml and the Debian installer

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Wed, 19 Mar 2008 23:27:00 +0100

For various reasons, I have the kernel and the initrd that my notebook needs to boot Linux on an USB stick. I recently added the Debian Installer and grml to the stick to allow additional uses of the stick.

Usually, USB sticks are formated with FAT32 and when they’re used to boot Linux, Syslinux or some derivative is used.

I do it differently on my boot stick, most prominently because I’d like to have Unixoid privileges on the stick, and because I’d like to continue using grub. So, my stick is formatted as ext3, and feels like an additional hard disk.

Booting from an USB stick is sometimes tricky. I have made the best experiences with formatting the USB stick as an USB ZIP, which basically means having 64 heads, 32 sectors and a single partition in the fourth primary partition slot. This can be done using mkdiskimage and is documented on Debian in /usr/share/doc/syslinux/usbkey.doc. Partitioning and generation of the file system is as straightforward as is installing grub using the standard, documented methods for hard disks.

I tried using grub 2, but have found it not sufficiently well-documented and stable to be used in this kind of critical environment. I am therefore still using grub-legacy despite the fact that the grub maintainers have ceased fixing grub-legacy bugs some months ago, refering to the pending release of grub 2.

I use unison to keep /boot on the stick in sync with /boot of the Linux system, which is on a crypted filesystem on the hard disk and thus unuseable until the system has booted. The hard disk /boot thus serves only as installation target and sync source so that the stick does not need to be plugged in during upgrades. update-grub is configured so that running it creates a menu.lst which can be used from the stick. This concludes booting the production system from the stick.

For the other systems, I have decided not to put them in the “default” boot menu from menu.lst, but instead include foo.lst files to the root of the stick, and when I need them, shell out from the grub menu to a command line by pressing “c” and then pulling in the new grub config file with the “configfile /foo.lst” command.

grml is my favorite rescue system. I use it all the time once a system starts acting up. One of its biggest advantages is its extremely flexible bootup process. That made it predestined to be the rescue system on my USB boot stick. grml is currently making a transition to a new build environment, which has had the side effect that the boot process has changed in some details. grml-small, the 60 MB variant that I had been using before, has been discontinued and replaced by grml-medium, a 170 MB variant which does definetely not fit any more on a business card CD, but 128 MB USB sticks have become relatively seldom.

For both grml flavours, all one needs to do is mount the .iso file loopback, copy the file’s contents to a subdirectory of the boot stick, and to drop this menu file to the stick:


title           grml-small
root            (hd0,3)
kernel          /grml-small/boot/isolinux/linux26 toram grml_dir=/grml-small/GRML/ grml_name=GRML ramdisk_size=100000
init=/etc/init lang=us BOOT_IMAGE=grml
initrd          /grml-small/boot/isolinux/minirt26.gz

title           grml-medium
root            (hd0,3)
kernel          /grml-medium/boot/grmlmedium/linux26 live-media-path=/grml-medium/live/ toram=grml-medium.squashfs
lang=us boot=live noeject noprompt keyboard=de
initrd          /grml-medium/boot/grmlmedium/initrd.gz

Since grml-medium works reasonably well, I do not expect to need grml-small very often any more.

The Debian Installer guys have published an initrd which can browse nearly arbitrary media for an debian-installer .iso file, mount it and install from that image. I have made use of that feature to allow Debian to be installed from my boot stick. /d-i holds a current debian-testing-i386-netinst.iso, and vmlinux and initrd.gz from http://ftp.nl.debian.org/debian/dists/testing/main/installer-i386/current/images/hd-media/. With that, this d-i.lst:


title debian installer
root (hd0,3)
kernel /d-i/vmlinuz
initrd /d-i/initrd.gz

does the trick and boots up the Installer, ready to start.

I guess that it would be easily possible to include these grub config stanzas to the default menu.lst, including grml and the Installer into the main grub menu, but I deliberately decided not to have these options directly in the menu for the time being.

It is always amazing how flexible today’s systems have become when it comes to boot Linux. For both rescue and installation, this is needed, and the respective authors have done a remarkable job.

Mobile Internet is affordable in Germany

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Sat, 04 Aug 2007 22:02:28 +0200

Last thursday and friday, I spent around eleven hours in the InterCity Express (ICE) of Deutsche Bahn. I was online, using Simyo GPRS, during this entire time. Thanks to the cellular network repeaters in ICE’s coach 3 and 23, this has worked reasonably well and has cost me EUR 5,27 - in a tariff with no basic charge and no commitment.

I really like that, because this usage of the mobile cellular network would have cost about fifty times more just a year ago. Thanks, Simyo, E Plus and the other E Plus resellers who have made the first step to reducing the cost of mobile Internet in Germany.

A few technical notes:

The repeaters in the ICE coaches do only repeat GSM and thus GPRS, but not UMTS. It is advised to lock the UMTS card to GPRS when using Internet from inside the ICE, since the card will otherwise move back and forth between UMTS and GPRS millions of times, which will always result in more than just a few seconds outage. It is much better to live with GPRS’ abysmally large latency.
I took the opportunity to empty my still-active, old, GPRS only, Simyo SIM which still holds like fifteen Euros. I suspect that I’ll need two more trips of this magnitude to successfully get rid of the amount on that prepaid card.
I need an external antenna for my UMTS card - it worked really really bad in my mom’s apartment right in the middle of Hamburg.
There seem to be a number of independent issues in Debian sid (the reason why I am blogging this in English): The connection occasionally hangs (a couple of times an hour) and does not come back by itself. Often, poff/pon is enough, but sometimes it is necessary to pull the UMTS card (an Option 3G PC-Card Datacard) and to re-insert it since there was no answer any more on ttyUSB0. Very annoying.
The Option 3G Datacard behaves differently depending on the SIM that is inserted. With my older, GPRS-only Simyo SIM, the response to AT+CPIN? is “SIM PUK2” after sending the PIN to the card, while it is “SIM PIN2” in the same situation with the newer UMTS-enabled Simyo SIM.
chat(8) sucks badly, because it doesn’t seem to elegantly handle this case - no regexps, no if/then, every command sent needs to generate exactly one answer from the card, everything else is an error after timeout.
When I do this more often, one of the now available flat rates (or, for starters, a 5 GB commitment tariff) for like 20 Euros becomes attractive.

About Acronyms and non-Acronyms?

mh+blog-zugschlus-de@zugschlus.de (Marc 'Zugschlus' Haber) — Fri, 23 Mar 2007 19:22:18 +0100

I keep wondering why people keep writing HUB, WEB and SPAM, where the correct technical terms are Hub, Web and Spam. Neither of the three expressions is an acronym.

Well, SPAM is, but Spiced Pork and Ham is a Trademark of Hormel Industries, and they ask people not to use their trademark to talk about Unsolicited Commercial/Bulk E-Mail on the Internet. They do, however, allow the expression Spam to be used for UCE/UBE.

Any idea why people keep treating Hub and Web as an acronym? It disturbes my reading tremendously.