Zugschlusbeobachtungen

Here is a commented typescript of what I have done. Be warned: This was a lab setting with a rather minimal system. If you're going to try this with a production system, have a backup, or, if you're on snapshottable infrastructure, take snapshots and be prepared to derive from this process (each system is different) and, if in doubt, roll back if anything goes wrong and start over.

First let's see what we have and save a list of installed packages.

mh@swivel:~$ ssh wheezyarch.zugschlus.de
$ uname -a
Linux wheezyarch 3.9.4-zgsrv2008064 #2 SMP PREEMPT Wed Jun 5 12:57:51 UTC 2013 x86_64 GNU/Linux


$ dpkg --print-architecture
i386


$ dpkg --print-foreign-architectures


$ dpkg --list | grep i386 | wc -l
175


$ dpkg --list | grep amd64 | wc -l
0


$ sudo dpkg --get-selections | grep install | awk '{print $1}' | sed 's/:.*//;s/$/:amd64 install/;' > get-selections.pre


$

So we have an i386 system running an x86_64 kernel.

Next, we add amd64 as an additional architecture which allows us to install amd64 packages. We download the amd64 packages files. Note that dpkg still considers this an i386 system with amd64 as a foreign arch.

$ sudo dpkg --add-architecture amd64


$ sudo apt-get update
Get:1 http://security.debian.org wheezy/updates Release.gpg [836 B]
(…)
Fetched 6.155 kB in 4s (1.400 kB/s)
Reading package lists... Done


$ dpkg --print-architecture
i386


$ dpkg --print-foreign-architectures
amd64


$ dpkg --list | grep i386 | wc -l
175


$ dpkg --list | grep amd64 | wc -l
0


$

In the next step, we identify the multi-arch capable libraries and install their amd64 versions parallel to the i386 versions.

$ sudo apt-get install $(grep-status --field=Status "install ok installed" | grep-dctrl --field=Multi-Arch same --show-field=Package --no-field-names | sed 's/\\(.*\\)/\\1:amd64/')


$

If some of those library packages depend on non-multiarch libs such as libonig2, this is not going to work and you need to add those non-multiarchi libs to the apt-get command line manually (e.g. "apt-get install $(…) libonig2:amd64). This is one of the steps that may remove packages that you want to stay part of the system. For example, adding libonig2:amd64 will remove jed and other slang-based packages.

Next, download everything that is necessary to migrate dpkg and apt from i386 and amd64. Since we already have the amd64 libs, this shold not be a bg download. In this step, we cannot directly apt things as the remove-install cycle used by apt will leave us without working dpkg. One could work around this by un-aring the dpkg.deb and directly forcing the amd64 files upon the system, but I think this approach is much cleaner. apt-get download will not resolve dependices while apt-get --download install does. After the download, dpkg --install the packages. dpkg is not particularly smart about installation order, so Pre-Depends may fail or not. If they fail, dpkg will give a list of deb files that were affected, which is an easy help to finish their installation. This did not happen when I collected the typescript for this article, but it happened multiple times during my preparations. The apt-get clean call at the beginning makes sure that one can dpkg --install /var/cache/apt/archives/*.deb later without causing extra installation cycles.

$ sudo apt-get clean


$ sudo apt-get --download-only install dpkg:amd64 apt:amd64
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  aptitude:amd64 synaptic:amd64 wajig:amd64 dpkg-dev:amd64 apt-doc:amd64 python-apt:amd64
The following packages will be REMOVED:
  apt cron-apt dpkg
The following NEW packages will be installed:
  apt:amd64 dpkg:amd64
0 upgraded, 2 newly installed, 3 to remove and 0 not upgraded.
Need to get 3,852 kB of archives.
After this operation, 11.3 kB disk space will be freed.
Do you want to continue [Y/n]? y
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main dpkg amd64 1.16.10 [2,599 kB]
Get:2 http://debian.debian.zugschlus.de/debian/ wheezy/main apt amd64 0.9.7.9 [1,254 kB]
Fetched 3,852 kB in 5s (711 kB/s)
Download complete and in download only mode


$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 20073 files and directories currently installed.)
Preparing to replace apt 0.9.7.9 (using .../archives/apt_0.9.7.9_amd64.deb) ...
Unpacking replacement apt ...
Preparing to replace dpkg 1.16.10 (using .../dpkg_1.16.10_amd64.deb) ...
Unpacking replacement dpkg ...
Setting up apt (0.9.7.9) ...
gpg: key B98321F9: "Squeeze Stable Release Key " not changed
gpg: key 473041FA: "Debian Archive Automatic Signing Key (6.0/squeeze) " not changed
gpg: key 65FFB764: "Wheezy Stable Release Key " not changed
gpg: key 46925553: "Debian Archive Automatic Signing Key (7.0/wheezy) " not changed
gpg: Total number processed: 4
gpg:              unchanged: 4
Setting up dpkg (1.16.10) ...


$ sudo apt-get clean

From dpkg's point of view, the system is now already amd64, with i386 being a foreign arch. However, the majory of packages is still i386.

$ dpkg --print-architecture
amd64


$ dpkg --print-foreign-architectures
i386


$ dpkg --list | grep i386 | wc -l
173


$ dpkg --list | grep amd64 | wc -l
11


$ 

And now, the fun begins.

The apt resolver is currently in a very bad broken state since the system is missing most essential packages in its native architecture. This is not easily solved by apt-get -f install. In the easy case, apt-get -f install would simply try to zap most of the existing system, which is solved easily. In the non-easy case, apt-get -f install will consider itself unable to correct dependencies and give a misleading error message about held packages.

$ sudo apt-get --download-only -f install
Reading package lists... 1%
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... failed.
The following packages have unmet dependencies:
 acpi-support-base : Depends: acpid (>= 1.0.4) but it is not installed
 acpid:i386 : Depends: module-init-tools:i386 (> 3.1-rel-2) but it is not installable
 adduser : Depends: perl-base (>= 5.6.0) but it is not installed
 aide-common : Depends: aide (>= 0.15.1-6) but it is not installed or
                        aide-xen (>= 0.15.1-6) but it is not installed or
                        aide-dynamic (>= 0.15.1-6) but it is not installed
               Depends: bsd-mailx but it is not installed or
                        mailx
               Depends: initscripts (>= 2.88dsf-13.3) but it is not installed
 aide-config-zg2 : Depends: aide but it is not installed
 apache2.2-common:i386 : Depends: mime-support:i386 but it is not installable
 aptitude:i386 : Depends: aptitude-common:i386 (= 0.6.8.2-1) but it is not installable
 ca-certificates : Depends: openssl (>= 1.0.0) but it is not installed
 console-common : Depends: kbd or
                           console-tools (>= 1:0.2.3dbs-54) but it is not installed
 console-log : Depends: less but it is not installed
               Depends: daemon but it is not installed
               Depends: kbd
 debconf : PreDepends: perl-base (>= 5.6.1-4) but it is not installed
 debian-goodies : Depends: dctrl-tools but it is not installed or
                           grep-dctrl
                  Depends: perl but it is not installed
                  Depends: curl but it is not installed
                  Depends: whiptail but it is not installed or
                           dialog but it is not installed
 debsecan : Depends: python-apt but it is not installed
 exim4-base:i386 : Depends: exim4-config:i386 (>= 4.30) but it is not installable or
                            exim4-config-2:i386 but it is not installable
                   Depends: netbase:i386 but it is not installable
 ftp:i386 : Depends: netbase:i386 but it is not installable
 git:i386 : Depends: perl-modules:i386 but it is not installable
            Depends: liberror-perl:i386 but it is not installable
            Depends: git-man:i386 (> 1:1.7.10.4) but it is not installable
            Depends: git-man:i386 (< 1:1.7.10.4-.) but it is not installable
 grub-pc:i386 : Depends: ucf:i386 but it is not installable
 inetutils-ping:i386 : Depends: netbase:i386 but it is not installable
 initramfs-tools : Depends: klibc-utils (>= 2.0-1~) but it is not installed
                   Depends: cpio but it is not installed
 initscripts:i386 : Depends: sysv-rc:i386 but it is not installable or
                             file-rc:i386 but it is not installable
 kernel-image-zgsrv200806432 : Depends: linux-image-3.9.7-zgsrv2008064 (= 3.9.7.20130626.0) but it is not installable
 liberror-perl : Depends: perl (>= 5.6.0-16) but it is not installed
 libswitch-perl : Depends: perl but it is not installed
 lsof:i386 : Depends: libperl4-corelibs-perl:i386 but it is not installable or
                      perl:i386 (< 5.12.3-7) but 5.14.2-21 is installed
 ntp:i386 : Depends: netbase:i386 but it is not installable
 openssl-blacklist : Depends: openssl (>= 0.9.8g-9) but it is not installed
 perl:i386 : Depends: perl-modules:i386 (>= 5.14.2-21) but it is not installable
 perl-modules : Depends: perl (>= 5.14.2-1) but it is not installed
 python : Depends: python2.7 (>= 2.7.3-1~) but it is not installed
 python-apt:i386 : Depends: python:i386 (>= 2.6.6-7~) but it is not installable
                   Depends: python:i386 (< 2. but it is not installable
                   Depends: python-apt-common:i386 but it is not installable
 python-minimal : Depends: python2.7-minimal (>= 2.7.3-1~) but it is not installed
 python-pylibacl:i386 : Depends: python:i386 (>= 2.6) but it is not installable
                        Depends: python:i386 (< 2. but it is not installable
                        Depends: libjs-sphinxdoc:i386 (>= 1.0) but it is not installable
 python-pyxattr:i386 : Depends: python:i386 (>= 2.6) but it is not installable
                       Depends: python:i386 (< 2. but it is not installable
                       Depends: libjs-sphinxdoc:i386 (>= 1.0) but it is not installable
 python2.7:i386 : Depends: mime-support:i386 but it is not installable
 rdiff-backup:i386 : Depends: python:i386 (>= 2.6.6-7~) but it is not installable
                     Depends: python:i386 (< 2. but it is not installable
 ssl-cert : Depends: openssl (>= 0.9.8g-9) but it is not installed
 stow : Depends: perl but it is not installed
 sysv-rc : Depends: sysvinit-utils (>= 2.86.ds1-62) but it is not installed
           Depends: insserv (> 1.12.0-10) but it is not installed
 sysvinit:i386 : PreDepends: sysv-rc:i386 but it is not installable or
                             file-rc:i386 but it is not installable
 vim:i386 : Depends: vim-runtime:i386 (= 2:7.3.547-7) but it is not installable
 vsftpd:i386 : Depends: netbase:i386 but it is not installable
 zg2-rebootscript : Depends: mailx
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
E: Unable to correct dependencies


$

But we don't have any held packages on the system. In the case of this example, this behavior is caused by apache 2.2 depending indirectly on libaprutil1-dbd-pgsql which is a non-multiarch library in wheezy. Unfortunately, we cannot use apt here to resolve the dependencies since apt's state is broken. We need to manually resolve the dependencies, download the respective deb files from the archive and dpkg --install them manually. After a number of manual iterations, this particular case is closed by dpkg --install of apache2.2-bin, apache2.2-common, apache2-utils, libapr1, libaprutil1, libaprutil1-dbd-sqlite3, libaprutil1-ldap, perl and perl-base in their respective wheezy amd64 version.

We now are back to the state where apt-get -f install would again be ready to do things. If we let it do things, it would to wrong things and leave us without working dpkg. Hence, we let apt-get --download-only -f install download the packages that it wants to replace with its amd64 counterparts and their dependencies, and, again, use dpkg --install \*.deb to install them. It might be possible that apt-get --download-only -f -install does not want to download anything, in which case, we skip this step.

$ sudo apt-get clean


$ sudo apt-get --download-only -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  anacron:i386 cron:i386 daemon:i386 dctrl-tools:i386 gettext-base:i386 insserv:i386 libapr1:i386 libasprintf0c2:i386 libboost-iostreams1.49.0:i386
  libcap2:i386 libcurl3-gnutls:i386 libcwidget3:i386 libept1.4.12:i386 libexpat1:i386 libfreetype6:i386 libfuse2:i386 liblockfile1:i386 librsync1:i386
  libsigc++-2.0-0c2a:i386 libsqlite3-0:i386 libxapian22:i386 logrotate:i386 python2.7-minimal:i386 vim-common:i386
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  acpi-support-base acpid:i386 aide-common aide-config-zg2 apache2:i386 apache2-mpm-worker:i386 apache2-utils apache2.2-bin apache2.2-common
  aptitude:i386 bsd-mailx:i386 ca-certificates console-common console-data console-log curl:i386 debconf-utils debian-goodies debsecan dmsetup:i386
  e2fsprogs:i386 exim4-base:i386 exim4-config exim4-daemon-light:i386 ftp:i386 git:i386 grml-rescueboot grub-common:i386 grub-pc:i386 grub-pc-bin:i386
  grub2-common:i386 ifupdown:i386 inetutils-ping:i386 initramfs-tools initscripts:i386 ippl:i386 kernel-image-zgsrv200806432 libapr1 libaprutil1
  libaprutil1-dbd-pgsql:i386 libaprutil1-dbd-sqlite3 libaprutil1-ldap libcurl3:i386 libcurl3 libdevmapper-event1.02.1:i386 libdevmapper-event1.02.1
  libdevmapper1.02.1:i386 libdevmapper1.02.1 liberror-perl libparted0debian1:i386 libparted0debian1 libpq5:i386 libsasl2-modules:i386 libsasl2-modules
  libssl1.0.0:i386 libssl1.0.0 libswitch-perl libterm-readkey-perl:i386 locales lsb-release lsof:i386 lvm2:i386 molly-guard ntp:i386 openssh-client:i386
  openssh-server:i386 openssl:i386 openssl-blacklist openssl-blacklist-extra openvpn-blacklist parted:i386 perl perl-modules procps:i386 python
  python-apt:i386 python-apt-common python-minimal python-pylibacl:i386 python-pyxattr:i386 python2.7:i386 rdiff-backup:i386 rsyslog:i386 ssl-cert stow
  sysv-rc sysvinit:i386 ucf udev:i386 util-linux:i386 vim:i386 vsftpd:i386 zg2-rebootscript
0 upgraded, 0 newly installed, 93 to remove and 0 not upgraded.
After this operation, 163 MB disk space will be freed.
Do you want to continue [Y/n]? y
Download complete and in download only mode


$ sudo dpkg --install /var/cache/apt/archives/*.deb
dpkg: error processing /var/cache/apt/archives/*.deb (--install):
 cannot access archive: No such file or directory
Errors were encountered while processing:
 /var/cache/apt/archives/*.deb


$ sudo apt-get clean


$ dpkg --print-architecture
amd64


$ dpkg --print-foreign-architectures
i386


$ dpkg --list | grep i386 | wc -l
191


$ dpkg --list | grep amd64 | wc -l
89


$ 

The concluding apt-get clean prevents packages from being reinstalled again in the next dpkg --install \*.deb run.

Now, the apt resolver is kind of fixed, but apt will still want to remove most of the system. I haven't found a way to get out of this mess short of allowing apt to de-install those packages and to manually install their amd64 counterparts later. First, we see what apt wants to do, and paste the "The following packages will be removed" part of apt-get's output into a sed expression which fixes the architecture strings for us so that we can use the resulting file as input for the following installation procedure. After that, we let apt-get do what itself considers a bad thing and requires us to type a consent sentence. The key is that this deinstallation of essential packages breaks the system, but not apt and dpkg, which allows us to un-break the system afterwards.

$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  gettext-base:i386 libapt-inst1.5 libasprintf0c2:i386 libcurl3-gnutls:i386
  libfreetype6:i386 libfuse2:i386 libreadline5:i386
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  sysvinit-utils:i386
Suggested packages:
  bootlogd:i386 sash:i386
The following packages will be REMOVED:
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386
The following NEW packages will be installed:
  sysvinit-utils:i386
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  e2fsprogs:i386 util-linux:i386 (due to e2fsprogs:i386) sysvinit:i386
  sysvinit-utils tzdata (due to util-linux:i386)
0 upgraded, 1 newly installed, 62 to remove and 0 not upgraded.
Need to get 97,1 kB of archives.
After this operation, 131 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!' ? (abort with Ctrl-C)


$ sed 's/:i386//g' > removedpkg (paste "The following packages will be REMOVED:" stanza here and type Ctrl-D afterwards)
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386


$ cat removedpkg 
  adduser aide-common aptitude bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup e2fsprogs
  exim4-base exim4-config exim4-daemon-light git grub-common
  grub-pc grub-pc-bin grub2-common ifupdown
  initramfs-tools initscripts ippl jed libcurl3
  libdevmapper-event1.02.1 libdevmapper1.02.1 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof lvm2 molly-guard
  ntp openssh-client openssh-server openssl openssl-blacklist
  openssl-blacklist-extra parted perl perl-modules procps
  python-apt rsyslog stow sysv-rc sysvinit sysvinit-utils
  tzdata ucf udev util-linux


$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  gettext-base:i386 libapt-inst1.5 libasprintf0c2:i386 libcurl3-gnutls:i386
  libfreetype6:i386 libfuse2:i386 libreadline5:i386
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  sysvinit-utils:i386
Suggested packages:
  bootlogd:i386 sash:i386
The following packages will be REMOVED:
  adduser aide-common aptitude:i386 bsd-mailx console-common console-data
  console-log cron curl debian-goodies debsecan dmsetup:i386 e2fsprogs:i386
  exim4-base exim4-config exim4-daemon-light git:i386 grub-common:i386
  grub-pc:i386 grub-pc-bin:i386 grub2-common:i386 ifupdown:i386
  initramfs-tools initscripts ippl:i386 jed:i386 libcurl3
  libdevmapper-event1.02.1:i386 libdevmapper1.02.1:i386 liberror-perl
  libfile-find-rule-perl libnumber-compare-perl libparted0debian1:i386
  libswitch-perl libterm-readkey-perl libterm-readline-perl-perl
  libtext-glob-perl libtimedate-perl locales lsof:i386 lvm2:i386 molly-guard
  ntp:i386 openssh-client:i386 openssh-server:i386 openssl openssl-blacklist
  openssl-blacklist-extra parted:i386 perl perl-modules procps:i386
  python-apt:i386 rsyslog:i386 stow sysv-rc sysvinit:i386 sysvinit-utils
  tzdata ucf udev:i386 util-linux:i386
The following NEW packages will be installed:
  sysvinit-utils:i386
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  e2fsprogs:i386 util-linux:i386 (due to e2fsprogs:i386) sysvinit:i386
  sysvinit-utils tzdata (due to util-linux:i386)
0 upgraded, 1 newly installed, 62 to remove and 0 not upgraded.
Need to get 97,1 kB of archives.
After this operation, 131 MB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?] Yes, do as I say! (actually type this!)
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main sysvinit-utils i386 2.88dsf-41 [97,1 kB]
Fetched 97,1 kB in 0s (687 kB/s)    
(Reading database ... 17050 files and directories currently installed.)
Removing aide-common ...
(…)
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing e2fsprogs ...
Removing sysv-rc ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing sysvinit ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing sysvinit-utils ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing ucf ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing libdevmapper1.02.1:i386 ...
Removing libswitch-perl ...
Removing perl ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing dmsetup ...
Removing perl-modules ...
dpkg: warning: overriding problem because --force enabled:
 This is an essential package - it should not be removed.
Removing util-linux ...
Removing tzdata ...
Processing triggers for mime-support ...
Selecting previously unselected package sysvinit-utils.
(Reading database ... 9802 files and directories currently installed.)
Unpacking sysvinit-utils (from .../sysvinit-utils_2.88dsf-41_i386.deb) ...
Setting up sysvinit-utils (2.88dsf-41) ...


$

Apt still wants some packages to be present. I don't know why it doesn't install them during the first apt-get -f instal, run, but doing this manually is necessary to make apt happy.

$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  anacron:i386 cron:i386 daemon:i386 dctrl-tools:i386 gettext-base:i386 insserv:i386 kbd:i386 libasprintf0c2:i386 libboost-iostreams1.49.0:i386
(…)
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  heirloom-mailx libssl1.0.0
Suggested packages:
  exim4 mail-transport-agent
The following NEW packages will be installed:
  heirloom-mailx libssl1.0.0
0 upgraded, 2 newly installed, 0 to remove and 1 not upgraded.
1 not fully installed or removed.
Need to get 1493 kB of archives.
After this operation, 3539 kB of additional disk space will be used.
Do you want to continue [Y/n]? 
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main libssl1.0.0 amd64 1.0.1e-2 [1219 kB]
Get:2 http://debian.debian.zugschlus.de/debian/ wheezy/main heirloom-mailx amd64 12.5-2 [274 kB]
Fetched 1493 kB in 2s (700 kB/s)    
Selecting previously unselected package libssl1.0.0:amd64.
(Reading database ... 13042 files and directories currently installed.)
Unpacking libssl1.0.0:amd64 (from .../libssl1.0.0_1.0.1e-2_amd64.deb) ...
Selecting previously unselected package heirloom-mailx.
Unpacking heirloom-mailx (from .../heirloom-mailx_12.5-2_amd64.deb) ...
Setting up libssl1.0.0:amd64 (1.0.1e-2) ...
Setting up heirloom-mailx (12.5-2) ...
update-alternatives: using /usr/bin/heirloom-mailx to provide /usr/bin/mailx (mailx) in auto mode


$ sudo apt-get install $(cat removedpkg)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libssl1.0.0 is already the newest version.
The following packages were automatically installed and are no longer required:
  anacron:i386 libasprintf0c2:i386 libbsd0:i386 libcap2:i386 libcurl3-gnutls:i386 libedit2:i386 libexpat1:i386 libfreetype6:i386 libfribidi0:i386
  libfuse2:i386 libgssapi-krb5-2:i386 libidn11:i386 libk5crypto3:i386 libkeyutils1:i386 libkrb5-3:i386 libkrb5support0:i386 liblockfile1:i386
  libncursesw5:i386 libprocps0:i386 librsync1:i386 librtmp0:i386 libsigc++-2.0-0c2a:i386 libsqlite3-0:i386 libssh2-1:i386 libudev0:i386 psmisc:i386
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  aide cpio cron daemon dctrl-tools gettext-base insserv kbd klibc-utils less libboost-iostreams1.49.0 libcwidget3 libept1.4.12 libklibc libnewt0.52
  libopts25 libxapian22 logrotate python2.7-minimal sysvinit-utils vim-common whiptail
Suggested packages:
  acpi-support www-browser apache2-doc apache2-suexec apache2-suexec-custom tasksel debtags unicode-data libarchive1 anacron checksecurity
  popularity-contest xdg-utils zenity gpart e2fsck-static eximon4 exim4-doc-html exim4-doc-info spf-tools-perl swaks git-daemon-run git-daemon-sysvinit
  git-doc git-el git-arch git-cvs git-svn git-email git-gui gitk gitweb multiboot-doc grub-emu xorriso desktop-base isc-dhcp-client dhcp-client ppp
  rdnssd net-tools bootchart2 libcwidget-dev libparted0-dev libparted0-i18n libsasl2-modules-otp libsasl2-modules-ldap libsasl2-modules-sql
  libsasl2-modules-gssapi-mit libsasl2-modules-gssapi-heimdal xapian-tools lsb ntp-doc ssh-askpass libpam-ssh keychain monkeysphere rssh ufw parted-doc
  perl-doc libterm-readline-gnu-perl libterm-readline-perl-perl make libpod-plainer-perl python-doc python-tk python-apt-dbg python-gtk2 python-vte
  python-apt-doc python-pylibacl-dbg python-pyxattr-dbg python2.7-doc binutils binfmt-support rsyslog-mysql rsyslog-pgsql rsyslog-doc rsyslog-gnutls
  rsyslog-gssapi rsyslog-relp doc-base sysv-rc-conf bum bootlogd sash util-linux-locales dosfstools ctags vim-doc vim-scripts
Recommended packages:
  consolekit aptitude-doc-en aptitude-doc apt-xapian-index libparse-debianchangelog-perl exim4 postfix mail-transport-agent psmisc patch ssh-client
  os-prober busybox busybox-initramfs busybox-static xauth ncurses-term iso-codes usbutils
The following packages will be REMOVED:
  aide:i386 cpio:i386 cron:i386 daemon:i386 dctrl-tools:i386 gettext-base:i386 insserv:i386 kbd:i386 klibc-utils:i386 less:i386
  libboost-iostreams1.49.0:i386 libcwidget3:i386 libept1.4.12:i386 libklibc:i386 libnewt0.52:i386 libopts25:i386 libxapian22:i386 logrotate:i386
  python2.7-minimal:i386 sysvinit-utils:i386 vim-common:i386 whiptail:i386
The following NEW packages will be installed:
  acpi-support-base acpid aide aide-common aide-config-zg2 apache2 apache2-mpm-worker apache2-utils apache2.2-bin apache2.2-common aptitude bsd-mailx
  ca-certificates console-common console-data console-log cpio cron curl daemon dctrl-tools debconf-utils debian-goodies debsecan dmsetup e2fsprogs
  exim4-base exim4-config exim4-daemon-light ftp gettext-base git grml-rescueboot grub-common grub-pc grub-pc-bin grub2-common ifupdown inetutils-ping
  initramfs-tools initscripts insserv ippl kbd klibc-utils less libapr1 libaprutil1 libaprutil1-dbd-pgsql libaprutil1-dbd-sqlite3 libaprutil1-ldap
  libboost-iostreams1.49.0 libcurl3 libcwidget3 libdevmapper-event1.02.1 libdevmapper1.02.1 libept1.4.12 liberror-perl libklibc libnewt0.52 libopts25
  libparted0debian1 libpq5 libsasl2-modules libswitch-perl libterm-readkey-perl libxapian22 locales logrotate lsb-release lsof lvm2 molly-guard ntp
  openssh-client openssh-server openssl openssl-blacklist openssl-blacklist-extra openvpn-blacklist parted perl perl-modules procps python python-apt
  python-apt-common python-minimal python-pylibacl python-pyxattr python2.7 python2.7-minimal rdiff-backup rsyslog ssl-cert stow sysv-rc sysvinit
  sysvinit-utils ucf udev util-linux vim vim-common vsftpd whiptail
The following packages will be upgraded:
  zg2-rebootscript
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
  sysvinit-utils:i386
1 upgraded, 106 newly installed, 22 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 64.2 MB of archives.
After this operation, 148 MB of additional disk space will be used.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
 ?] Yes, do as I say! (actually type this!)
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main e2fslibs amd64 1.42.5-1.1 [197 kB]
(…)
Fetched 55.3 MB in 20s (2648 kB/s)                                            
perl: warning: Setting locale failed.
(…)


$

Under some circumstances, you might have some maintainer scripts fail because of missing essential stuff. For example, some packages might fail their installation because of invoke-rc.d not being present on the system. If that happens, you need to manually dpkg --install /var/cache/apt/archives/*.deb. If this fails, it will give you a list of packages that failed to install. The easiest way is to simply repeat the dpkg --install /var/cache/apt/archives/*.deb. Or you can just dpkg --install the failing packages from /var/cache/apt/archives. Strangely, dpkg does not give the full package file name for all failing packages. The dpkg --install should work the second time since all packages have been unpacked before, and the dependencies of the failing package might even already been configured.

This process may re-ask some debconf questions that you have already answered during initial system setup. I find it comforting to actually see that my answers were preserved, but you can always export DEBIAN_FRONTEND=noninteractive if you want installation to be silent. If you know how this step can be done more elegantly, please comment.

We now, finally, have the resolver in a consistent state.

$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.


$ dpkg --print-architecture
amd64


$ dpkg --print-foreign-architectures
i386


$ dpkg --list | grep i386 | wc -l
121


$ dpkg --list | grep amd64 | wc -l
119


$ 

Now we can force-crossgrade all i386 packages that are still left on the system. Again, we use the apt-get --download install, dpkg --install stunt as coreutils gets crossgraded at this step and dpkg is less than happy if it finds itself without /bin/rm. Remember, with --download, nothing is actually removed during this step even if apt-get claims to remove things. Apt will also complain that many of these packages are already installed. Those are libs that were pulled in previously. The first dpkg --install run in this step will complain about gazillions of dependency problems, which will all be resolved in concluding apt-get -f install after the second dpkg --install run, this time a "manual" one. The big list of packages that are "already the newest version" are multi-arch libraries which are misdetected by our simple command line because their i386 counterpart is also still present.

$ sudo apt-get clean


$ sudo apt-get --download-only install $(dpkg --list | awk '{if( $1 == "ii" && $4 == "i386" ) { print $2 }}' | sed 's/:i386//')
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libacl1 is already the newest version.
libapt-inst1.5 is already the newest version.
libapt-pkg4.12 is already the newest version.
libattr1 is already the newest version.
(…)
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use 'apt-get autoremove' to remove them.
Suggested packages:
  powermgmt-base bash-doc bzip2-doc diffutils-doc wdiff mlocate locate
  gnupg-doc xloadimage imagemagick eog libpcsclite1 iproute-doc resolvconf
  avahi-autoipd nfs-common ncompress indent
Recommended packages:
  bsdmainutils gnupg-curl libatm1
The following packages will be REMOVED:
  anacron:i386 apt-utils:i386 base-files:i386 base-passwd:i386 bash:i386
(…)
The following NEW packages will be installed:
  anacron apt-utils base-files base-passwd bash bsdutils busybox bzip2
(…)
0 upgraded, 56 newly installed, 49 to remove and 0 not upgraded.
Need to get 24,1 MB of archives.
After this operation, 2.030 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://debian.debian.zugschlus.de/debian/ wheezy/main kmod amd64 9-3 [60,5 kB]
(…)
Fetched 24,1 MB in 11s (2.120 kB/s)                                           
Download complete and in download only mode


$ sudo dpkg --install /var/cache/apt/archives/*.deb
(Reading database ... 17098 files and directories currently installed.)
Preparing to replace adduser 3.113+nmu3 (using .../adduser_3.113+nmu3_all.deb) ...
Unpacking replacement adduser ...
(…)
Errors were encountered while processing:
 /var/cache/apt/archives/bash_4.2+dfsg-0.1_amd64.deb
 /var/cache/apt/archives/coreutils_8.13-3.5_amd64.deb
 initscripts
 lvm2
 procps
 rsyslog
 sysvinit
 sysv-rc
 udev
 util-linux
 aide-common
 dmsetup
 e2fsprogs
 ifupdown
 initramfs-tools
 libdevmapper1.02.1:amd64
 libdevmapper-event1.02.1:amd64
 libparted0debian1:amd64
 molly-guard
 openssh-server
 parted
 grub-common
 grub-pc
 grub-pc-bin
 grub2-common


$ sudo dpkg --install /var/cache/apt/archives/bash_4.2+dfsg-0.1_amd64.deb /var/cache/apt/archives/coreutils_8.13-3.5_amd64.deb
(Reading database ... 17129 files and directories currently installed.)
Preparing to replace bash 4.2+dfsg-0.1 (using .../bash_4.2+dfsg-0.1_amd64.deb) ...
Unpacking replacement bash ...
Preparing to replace coreutils 8.13-3.5 (using .../coreutils_8.13-3.5_amd64.deb) ...
Unpacking replacement coreutils ...
Setting up bash (4.2+dfsg-0.1) ...
update-alternatives: using /usr/share/man/man7/bash-builtins.7.gz to provide /usr/share/man/man7/builtins.7.gz (builtins.7.gz) in auto mode
Setting up coreutils (8.13-3.5) ...


$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libapt-inst1.5:i386 libapt-pkg4.12:i386 libasprintf0c2:i386 libatm1:i386 libbsd0:i386 libbz2-1.0:i386 libcap2:i386 libcurl3-gnutls:i386 libedit2:i386
  libexpat1:i386 libfreetype6:i386 libfribidi0:i386 libfuse2:i386 libgcrypt11:i386 libgnutls26:i386 libgpg-error0:i386 libgssapi-krb5-2:i386
  libidn11:i386 libk5crypto3:i386 libkeyutils1:i386 libkmod2:i386 libkrb5-3:i386 libkrb5support0:i386 libldap-2.4-2:i386 liblockfile1:i386
  libmagic1:i386 libncursesw5:i386 libp11-kit0:i386 libpci3:i386 libpcre3:i386 libpng12-0:i386 libpopt0:i386 libprocps0:i386 libreadline6:i386
  librsync1:i386 librtmp0:i386 libsasl2-2:i386 libsemanage1:i386 libsigc++-2.0-0c2a:i386 libslang2:i386 libsqlite3-0:i386 libssh2-1:i386 libstdc++6:i386
  libtasn1-3:i386 libudev0:i386 libustr-1.0-1:i386 libwrap0:i386
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.


$ sudo apt-get -f install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libasprintf0c2:i386 libcurl3-gnutls:i386 libfreetype6:i386 libfuse2:i386
  libreadline5:i386
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.


$ dpkg --print-architecture
amd64


$ dpkg --print-foreign-architectures
i386


$ dpkg --list | grep i386 | wc -l
72


$ dpkg --list | grep amd64 | wc -l
175


$ 

We have completed the actual crossgrade. All non-multiarch packages are now amd64. The resolver is now fine as well.

At this current state, all i386 packages that are still installed should be unneeded libs. We can safely nuke them. Careful minds may inspect the list of packages to be removed for important or non-lib items. After completion of the command, a second call of the same command verifies that we were successful.

$ sudo apt-get remove $(dpkg --list | awk '{if( $1 == "ii" && $4 == "i386" ) { print $2 }}' )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  e2fslibs:i386 gcc-4.7-base:i386 libacl1:i386 libapt-inst1.5:i386
  libapt-pkg4.12:i386 libasprintf0c2:i386 libattr1:i386 libblkid1:i386
  libbsd0:i386 libbz2-1.0:i386 libc6:i386 libcap2:i386 libcomerr2:i386
  libcurl3:i386 libcurl3-gnutls:i386 libdb5.1:i386 libedit2:i386
  libexpat1:i386 libfreetype6:i386 libfuse2:i386 libgcc1:i386
  libgcrypt11:i386 libgdbm3:i386 libgnutls26:i386 libgpg-error0:i386
  libgpm2:i386 libgssapi-krb5-2:i386 libidn11:i386 libk5crypto3:i386
  libkeyutils1:i386 libkmod2:i386 libkrb5-3:i386 libkrb5support0:i386
  libldap-2.4-2:i386 liblockfile1:i386 liblzma5:i386 libmagic1:i386
  libncurses5:i386 libncursesw5:i386 libp11-kit0:i386 libpam-modules:i386
  libpam0g:i386 libpci3:i386 libpcre3:i386 libpng12-0:i386 libpopt0:i386
  libprocps0:i386 libreadline5:i386 libreadline6:i386 librtmp0:i386
  libsasl2-2:i386 libselinux1:i386 libsemanage1:i386 libsepol1:i386
  libsigc++-2.0-0c2a:i386 libslang2:i386 libsqlite3-0:i386 libss2:i386
  libssh2-1:i386 libssl1.0.0:i386 libstdc++6:i386 libtasn1-3:i386
  libtinfo5:i386 libudev0:i386 libusb-0.1-4:i386 libustr-1.0-1:i386
  libuuid1:i386 libwrap0:i386 linux-image-3.9.4-zgsrv2008064:i386 zlib1g:i386
0 upgraded, 0 newly installed, 70 to remove and 0 not upgraded.
After this operation, 67,5 MB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 17111 files and directories currently installed.)
Removing e2fslibs:i386 ...
(…)


$ sudo apt-get remove $(dpkg --list | awk '{if( $1 == "ii" && $4 == "i386" ) { print $2 }}' )
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.


$ dpkg --print-architecture
amd64


$ dpkg --print-foreign-architectures
i386


$ dpkg --list | grep i386 | wc -l
70


$ dpkg --list | grep amd64 | wc -l
175


$ 

To finally make the packages really vanish from the system, we purge all packages that dpkg --list still reports in state "rc", which means "removed, configuration still present". Thankfully, dpkg seems to handle the case where a dpkg-conffile belongs to a transitioned package gracefully.

$ sudo dpkg --purge $(dpkg --list | awk '{if ($1 == "rc") { print $2 }}')
(Reading database ... 15868 files and directories currently installed.)
Removing e2fslibs:i386 ...
Purging configuration files for e2fslibs:i386 ...
(…)


$ dpkg --print-architecture
amd64


$ dpkg --print-foreign-architectures
i386


$ dpkg --list | grep i386 | wc -l
0


$ dpkg --list | grep amd64 | wc -l
175


$ 

We now verify that all packages we have are either arch all or arch amd64.

$ dpkg --list | grep -v '\\(amd64\\\|all\\)'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                            Version                   Architecture Description
+++-===============================-=========================-============-========================================================================


$

As I found myself suddenly without e2fsprogs during one of my experiments, I check for e2fsprogs being present. If this package is missing, you'll only find out after rebooting.

$ dpkg --list e2fsprogs
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  e2fsprogs      1.42.5-1.1   amd64        ext2/ext3/ext4 file system utilit


$

My locally built amd64 kernel with 32 bit support was an arch i386 package as well, so it got zapped in the migration process. Replace it with a "real" amd64 kernel. This step will vary if you use Debian kernels. Do not forget the firmwares.

$ sudo apt-get -t wheezy-zg-experimental install kernel-image-zgserver linux-firmware-image
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  cpio initramfs-tools kernel-image-zgsrv20080 klibc-utils libklibc
  linux-image-3.9.4-zgsrv20080
Suggested packages:
  libarchive1
The following NEW packages will be installed:
  cpio initramfs-tools kernel-image-zgserver kernel-image-zgsrv20080
  klibc-utils libklibc linux-firmware-image linux-image-3.9.4-zgsrv20080
0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
Need to get 11,2 MB of archives.
After this operation, 30,5 MB of additional disk space will be used.
Do you want to continue [Y/n]? 
Get:1 http://zg20110.debian.zugschlus.de/ wheezy-zg-experimental/main linux-image-3.9.4-zgsrv20080 amd64 3.9.4.20130605.0 [10,4 MB]
(…)


$

I now make sure that the initrd I have built contains all file system modules and is of the right architecture:

$ mkdir -p /tmp/initrd


$ cd /tmp/initrd


$ < /boot/initrd.img-3.9.4-zgsrv20080 gunzip | cpio -i
32053 blocks


$ find . -name '*ext*.ko'
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext3/ext3.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext2/ext2.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko


$ file ./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko
./lib/modules/3.9.4-zgsrv20080/kernel/fs/ext4/ext4.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=0x259a642448805c9597932ee25c62c8e6a00a32ad, not stripped


$ 

As the last step, remove i386 as an architecture and reboot into the newly migrated system. We verify that the system considers itself amd64 with no foreign architectures.

$ sudo dpkg --remove-architecture i386


$ dpkg --print-architecture
amd64


$ dpkg --print-foreign-architectures


$ dpkg --list | grep i386 | wc -l
0


$ dpkg --list | grep amd64 | wc -l
181


$ sudo shutdown -r now
Broadcast message from root@wheezyarch (pts/0) (Wed Jun 12 16:09:24 2013):
The system is going down for reboot NOW!


$ Connection to wheezyarch.zugschlus.de closed by remote host.
Connection to wheezyarch.zugschlus.de closed.



$ ssh wheezyarch.zugschlus.de
Linux wheezyarch 3.9.4-zgsrv20080 #2 SMP PREEMPT Wed Jun 5 13:12:04 UTC 2013 x86_64


$ dpkg --print-architecture
amd64


$

We now check whether we have lost any packages in this process:

$ sudo dpkg --get-selections | grep install | awk '{print $1}' | sed 's/:.*//;s/$/:amd64 install/;' > get-selections.post


$ wc -l get-selections.*
  234 get-selections.post
  226 get-selections.pre


$ diff --unified=0 get-selections.pre get-selections.post 
--- get-selections.pre  2013-06-12 19:47:40.278176000 +0200
+++ get-selections.post 2013-06-12 20:28:51.389874000 +0200
@@ -22,0 +23 @@
+cpio:amd64 install
@@ -58,0 +60 @@
+initramfs-tools:amd64 install
@@ -68,0 +71,3 @@
+kernel-image-zgserver:amd64 install
+kernel-image-zgsrv20080:amd64 install
+klibc-utils:amd64 install
@@ -108,0 +114 @@
+libklibc:amd64 install
@@ -163,0 +170,2 @@
+linux-firmware-image:amd64 install
+linux-image-3.9.4-zgsrv20080:amd64 install


$

So we have not actually lost things, but we have gained the pure amd64 kernel and the packages that are needed to build a proper initrd. These were missing from the initial install of the i386 test VM. If this comparision has not indentified packages that were deinstalled, but you made note of their deinstallaiton, this is now the time to reinstall them.

The downside of the process is that we have lost most of the "automatically installed" markers in aptitude which make sure to remove a library once nothing depends on it any more. If you make a point in having those markers correctly set, as I do, this is tedious work to set it on every package in the aptitude curses interface, making aptitude suggest removing the entire system, and then again remove it from the packages you actually intend to keep. This should not result in anything being removed, otherwise you have goofed previously.

All Done! I am now off perfecting my upgrade process so that my ancient i386 lenny and squeeze machines can first be moved to wheezy and then to amd64.