Evernote Tech Blog

The Care and Feeding of Elephants

New version of Debian is released

debianOn May 4th 2013, the Debian team released a new version (7.0) code named ‘Wheezy.’ This is a significant update and the Evernote Operations team is very excited about qualifying it for use in the near future. We are currently running version 6.0 code named ‘Squeeze.’

The release announcement is located here: http://www.debian.org/News/2013/20130504

We did experience a slight issue after this release that was self induced. A little background before we get into the issue. We maintain a mirror of Debian for internal use only. This prevents traffic going to the Internet during our machine builds and generally makes the builds happen very fast. We use a project called ‘apt-mirror’ to make this relatively easy to maintain and automate. See: http://apt-mirror.github.io/ The model we have setup for this allows us to pick and choose the sections of the Debian repository we would like to mirror in our own environment. The apt-mirror script runs every night out of cron. Debian refers to the releases using naming such as: stable, oldstable, unstable, and testing. As major releases get pushed by the Debian team, the naming for the old and new release changes. In this case Wheezy went from ‘unstable’ to ‘stable’ and Squeeze went from ‘stable’ to ‘oldstable.’

We discovered Monday morning, after the Wheezy release, that we could no longer build the (now) older version of Debian 6 ( Squeeze). This was a result of the suite changing the name from ‘stable’ to ‘oldstable’ in the ‘Release’ file and how our repository is set up. As we are not mirroring the entire repository, some of it needs to be maintained by hand. More specifically, inside the apt repository the Debian project maintains symbolic links to the various suites. The symbolic link got changed but we did not mirror this change internally. To see this upstream browse ftp://ftp.us.debian.org/debian/dists/ and notice the symbolic links to the suites. Unfortunately this was not entirely obvious to us at first and we had to spend some time debugging the choose-mirror program. This is used during build time to pick the correct repository and version to install. After spending a few minutes looking at the source code, we realize what was happening. The fix was simple. We deleted the current ‘stable -> squeeze’ link and changed it to ‘oldstable -> squeeze’ inside our dists sub-directory. This allowed builds to function as before.

Directory listing of our server to make this clear:

before:

debian/dists# ls -l
total 12
drwxrwxr-x 5 apt-mirror apt-mirror 4096 Jan 30  2012 lenny
drwxrwxr-x 5 apt-mirror apt-mirror 4096 May  6 14:26 squeeze
drwxrwxr-x 5 apt-mirror apt-mirror 4096 Apr  2 08:27 squeeze-updates
lrwxrwxrwx 1 apt-mirror apt-mirror    7 Jan 26  2012 stable -> squeeze

after:

debian/dists# ls -l
total 12
drwxrwxr-x 5 apt-mirror apt-mirror 4096 Jan 30  2012 lenny
lrwxrwxrwx 1 root       root          7 May  6 14:34 oldstable -> squeeze
drwxrwxr-x 5 apt-mirror apt-mirror 4096 May  6 15:13 squeeze
drwxrwxr-x 5 apt-mirror apt-mirror 4096 Apr  2 08:27 squeeze-updates

Header of release file showing suite name change:

debian/dists# cat oldstable/Release
Origin: Debian
Label: Debian
Suite: oldstable
Version: 6.0.7
Codename: squeeze
Date: Sat, 04 May 2013 12:31:36 UTC
[..]

Again we are excited about this new version however it will require qualification work before we will begin rolling this out into our staging and then production environments so we needed to get the Squeeze installs working again quickly.

  1. Very good writing and interesting.

  2. This is an interesting look at your backend systems. I am curious how you qualify a new Debian release, and how you manage your roadmap versus the Debian roadmap.

  3. Ah and here I was hoping that the actual Evernote client has been released for Debian…

    Nevertheless, thanks for sharing Ops internals, always interested in hearing about these.


Leave a Comment

* Required fields