Zugschlusbeobachtungen

Es war vor einigen Jahren, auf einem Treffen der SAGE Karlsruhe, wo im Rahmen eines Lightning Talks ssh proxycommand vorgestellt wurde. Ich hatte mich bisher immer mit einzelnen ssh-Aufrufen von Host zu Host weitergehangelt, und ssh proxycommand war für mich damals der erste Weg, direkt mit einem "natürlich" erscheindenen ssh-Aufruf trotz Sprunghost-Zwang auf dem Zielsystemen zu landen.

Das Verfahren wurde in neueren OpenSSH-Versionen noch zweimal vereinfacht, und in diesem Artikel möchte ich die Unterschiede herausstellen.

Starting with OpenSSH 7.2, a new "restrict" option for authorized_keys lines has become available. It sets all available restrictions that the current OpenSSH version can do (like no-agent-forwarding, no-x11-forwarding etc). One can individually turn on those features again by corresponding new options.

This saves one from sorrows when a new capability of OpenSSH is introduced through an update which is enabled by default, since one has to remember that restricted authorized_keys lines are in unse and then to manually add the restrictions.

On the downside, Debian jessie and CentOS 7 don't have a recent enough OpenSSH. So we'll have to continue worrying about new features being inadvertendly enabled for a while.

P.S.: Yes, I haven't blogged about Linux and Debian in English in a while.

Migrating a Debian installation between architectures has always been difficult. The recommended way to "crossgrade" an i386 Debian to amd64 Debian is to reinstall the system, move over data and configuration. For the more brave, in-place crossgrades usually involved chroots, rescue CDs, a lot of ar p | tar xf - data.tar.gz and luck.

I have never been brave when it comes to system administration, have done a lot of architecture migrations with reinstallation, and have always taken the opportunity to clear out the contamination that accumulates itself when a system is running for a long time. I would even recommend doing this to most people even now. However, I have a few very ugly systems in place that are still on i386 because I didn't dare going the reinstallation path.

Doing in-place crossgrades has become a lot easier since wheezy's release, since once now can have both i386 and amd64 libraries installed in parallel, which allows to replace foo:i386 with foo:amd64 without influencing the other parts of the system. The process is still full of pitfalls:

• In wheezy, many library packages are multiarch capable. This means that you can have those library packages installed for more than one architecture. This is a technical must for this way of crossgrade, so never use that for an older-than-wheezy system. It won't work, it needs at least Debian wheezy. Unfortunately, not all libraries in wheezy are multiarch capable. This makes the process harder and a lot less predictable, since a crosscrade including such packages is going to spew incomprehensible and misleading apt error messages. In my experience, for example the libaprutil-1-dbd-* packages and libonig2 are of this kind.
• apt removes a package before it reinstalls its new counterpart. This results in apt calling dpkg to remove dpkg, and then calling dpkg again to install dpkg. Guess which operation fails and the state of the system after this failure. Same applies to coreutils, which leaves the system without rm, which in turn dpkg of either architecture doesn't like. Using apt-get --download-only install to resolve dependencies and downloading the debs, followed by a traditional dpkg --install solves this issue since multiarch dpkg will replace a package with another one without deinstalling the first one first.
• At least for the process, you need a kernel that can run both 32bit and 64bit binaries for the i386 architecture. AFAIR, setting CONFIG_64BIT, CONFIG_X86_64 and CONFIG_IA32_EMULATION in the kernel configuration takes care of this.
• During the process, apt will temporary go into a badly broken state where it will refuse most operations. Be aware that you might need to manually download packages from the Internet. Be sure to have wget, curl, or a browser (maybe a text based one like elinks) available. dget is not going to help you here since it will only downloda packages for the native arch.
• During the process, apt wants to remove the better part of your system. It is important to not let it do this, as it wants to deinstall essential packages as well.
• Watch what your system does. During some steps, it might remove packages you might need. Keep track of the packages that were removed during the process and re-install them manually after finishing the crossgrade. Be sure not to purge packages that you might still need.
• It looks like the process is not always exactly reproducible. During the first tries, I found myself without an initrd at all, with an initrd that lacked the ext[234].ko kernel modules, without working e2fstools and in a number of other undesireable states of the system.

I have only tried this yet with a freshly installed minimal wheezy server system. Trying the process with "real life" systems has shown to be full of more surprises. I will document other pitfalls I have fallen into here at a later time. My minimal wheezy system was running in a KVM VM with its virtual disk as a LVM LV in the host system. I took a snapshot before beginning and used lvconvert --merge numerous time to return my LV to the original state. Be aware that lvconvert --merge removes the snapshot after merging it, so you'll need to re-create the snapshot before trying again.

The process is absolutely not for the faint of the heart, and intimate knowlegde of Debian mechanisms is required at many points in the process. Please seriously consider a reinstall+migrate approach instead of using this process, and be sure to practice it on a working copy of your system before touching the live system. And always have a backup.

During the process, I discussed things with Paul Tagliamonte, who has done this before, but on a live system and with a slightly more invasive approach. He has blogged about this. Thank you very much, your hints were very helpful.

In the last few weeks, I spent quite some time wondering about how to arrange the hard disk layout of my productive systems in the future. This article outlines my thoughts and would like to ask the lazyweb for comments.

I try to keep my Debian servers as identically as possible, making it possible to talk non-linux persons remotely through the system without having to worry about this particular box' configuration.