security.debian.org overloaded

The recently released security update of Xfree86 for sarge has made something happen what I have been fearing for years: Gazillions of systems downloading the update have slashdotted security.debian.org, which is now sluggishly responding for the second day in a row.

This doesn’t make my cron-apts happy, which in turn bury me under error message e-mails. Ungood, since one might miss an important update in the avalanche of “cannot pull packages.gz from security.debian.org” mails.

This experience has shown that having security.debian.org a single point of failure is not as good of an idea as we thought. I am afraid that the security team will have to reconsider and to finally establish mirrors for security.debian.org to spread the load.

Update: After reading much of the discussion about the topic, I find it strange that nobody besides me blogged about the issue, but we actually have an official announcement about the sdo outage. Very good.

Establishing a mirror network for sdo is not quite easy since unlike for the main archive, we cannot use “randomly offered” mirroring services, but we need the sdo mirrors under our control. Main reason for this is that we need sdo mirrors to be fast, because people would begin to complain that an update has not yet hit the mirrors after the advisory has gone out. Already today, it frequently happens that the cron-apt processes have detected an update quite some time before the release of the actual advisory, and you don’t want the process to take even longer. Even push mirroring is way too slow.

I like the idea of not establishing mirrors, but caching proxies distributed around the world, so that the issue of a mirror pulse simply vanishes: The proxy would fetch the update on the first incoming request, and deliver from its local cache for some time before looking on the actual sdo server again whether the file is still current. Neat idea.