On May 4th 2013, the Debian team released a new version (7.0) code named ‘Wheezy.’ This is a significant update and the Evernote Operations team is very excited about qualifying it for use in the near future. We are currently running version 6.0 code named ‘Squeeze.’
The release announcement is located here: http://www.debian.org/News/2013/20130504
We did experience a slight issue after this release that was self induced. A little background before we get into the issue. We maintain a mirror of Debian for internal use only. This prevents traffic going to the Internet during our machine builds and generally makes the builds happen very fast. We use a project called ‘apt-mirror’ to make this relatively easy to maintain and automate. See: http://apt-mirror.github.io/ The model we have setup for this allows us to pick and choose the sections of the Debian repository we would like to mirror in our own environment. The apt-mirror script runs every night out of cron. Debian refers to the releases using naming such as: stable, oldstable, unstable, and testing. As major releases get pushed by the Debian team, the naming for the old and new release changes. In this case Wheezy went from ‘unstable’ to ‘stable’ and Squeeze went from ‘stable’ to ‘oldstable.’
We discovered Monday morning, after the Wheezy release, that we could no longer build the (now) older version of Debian 6 ( Squeeze). This was a result of the suite changing the name from ‘stable’ to ‘oldstable’ in the ‘Release’ file and how our repository is set up. As we are not mirroring the entire repository, some of it needs to be maintained by hand. More specifically, inside the apt repository the Debian project maintains symbolic links to the various suites. The symbolic link got changed but we did not mirror this change internally. To see this upstream browse ftp://ftp.us.debian.org/debian/dists/ and notice the symbolic links to the suites. Unfortunately this was not entirely obvious to us at first and we had to spend some time debugging the choose-mirror program. This is used during build time to pick the correct repository and version to install. After spending a few minutes looking at the source code, we realize what was happening. The fix was simple. We deleted the current ‘stable -> squeeze’ link and changed it to ‘oldstable -> squeeze’ inside our dists sub-directory. This allowed builds to function as before.
Directory listing of our server to make this clear:
debian/dists# ls -l total 12 drwxrwxr-x 5 apt-mirror apt-mirror 4096 Jan 30 2012 lenny drwxrwxr-x 5 apt-mirror apt-mirror 4096 May 6 14:26 squeeze drwxrwxr-x 5 apt-mirror apt-mirror 4096 Apr 2 08:27 squeeze-updates lrwxrwxrwx 1 apt-mirror apt-mirror 7 Jan 26 2012 stable -> squeeze
debian/dists# ls -l total 12 drwxrwxr-x 5 apt-mirror apt-mirror 4096 Jan 30 2012 lenny lrwxrwxrwx 1 root root 7 May 6 14:34 oldstable -> squeeze drwxrwxr-x 5 apt-mirror apt-mirror 4096 May 6 15:13 squeeze drwxrwxr-x 5 apt-mirror apt-mirror 4096 Apr 2 08:27 squeeze-updates
Header of release file showing suite name change:
debian/dists# cat oldstable/Release Origin: Debian Label: Debian Suite: oldstable Version: 6.0.7 Codename: squeeze Date: Sat, 04 May 2013 12:31:36 UTC [..]
Again we are excited about this new version however it will require qualification work before we will begin rolling this out into our staging and then production environments so we needed to get the Squeeze installs working again quickly.