Evernote Tech Blog

The Care and Feeding of Elephants

Automated builds at Evernote

With more than 100 developers spread out among several countries, a robust continuous integration and build automation practice is crucial to Evernote’s development process. Through automated builds and tests, we can head off problems as soon as they are introduced into our code, resulting in a more efficient process and a reduced chance of impacting our users. Our overall strategy is based on the underlying principles from Martin Fowler’s 2006 article on Continuous Integration.

The Engineering Services team at Evernote operates a Jenkins master server and about 30 slave machines for building and testing. The slaves run a variety of operating systems (Windows, OS-X, and Linux), SDKs, build tools (Xcode, Android SDK, Visual Studio, Maven, etc), and test suites. Standard practice is for each build job to run on at least two slaves, so that if one is offline for any reason the other machine will seamlessly handle the build.

Jenkins

Our typical build-test-release workflow includes a verification build every time code is committed, a daily build for internal testers, and public builds (both beta and final release) that, in the case of our client applications, trigger an automatic deployment from within the client app.

In our Jenkins environment we build not only our Java-based web service, but also various applications for iOS, Android, Windows, OS-X, Windows Phone, Pebble and Blackberry. Every team is a little different, but the Evernote for Android client app provides a useful example of how we leverage automation.

  1. Every time an Android developer commits code to the main branch, a build is triggered. In the case of the Android client app this automated build is an internal test version that’s automatically deployed, so it’s important to make sure these builds pass basic usability testing.
  2. A successful build job triggers a build verification test (BVT), which consists of automated functional tests running on Android devices.
  3. A successful BVT triggers a deployment of the client app by pushing out a xml file to a location that is monitored by the clients.
  4. When the clients detect the new file, they immediately download and install the new version.

Our QA team receives emails when release candidate versions are ready for testing and sign-off, and these executables are readily available for download via Jenkins. The Android team publishes public beta and release versions through the Google Play store. All released versions of our apps are built and signed in our Jenkins environment.

Build System

For the Java-based Evernote web service, the workflow is relatively similar to the above. Developers commit code to the web service component projects, and those commits trigger verification builds that incorporate a series of tests. Successful builds trigger an automatic deployment to our staging environment, where QA engineers perform tests against the new code. We do a public release of our web service almost every Wednesday, at which time the build that’s been approved by QA is deployed by our operations team.

For our Maven-based jobs, we manage dependencies across projects using Artifactory, an open source tool that tightly controls how and where build artifacts are used in our environment.

Jenkins offers about 900 open source plugins of which we use nearly 75. Some of the most important plugins for our environment include:

  • Git, Maven, Gerrit Trigger
  • Email Extension
  • S3 Publisher
  • Parameterized Trigger
  • Token Macro
  • Jira

A follow-up blog post will cover some interesting parts of our web service production deployment process.

  1. I’d love to hear about the following:

    * do you have automation around provisioning Jenkins Slaves and Jenkins Job Config (Chef/Puppet?)
    * how long does a full run take from code checkin to deployment?

  2. Hi Phil, thanks for your comment. We use Puppet for config management on our Linux hosts (including the Jenkins master and some of the slaves). We do not currently use any configuration management tools for our Windows and OS-X slaves, but we hope to in the near future.

    A “full run” on the Evernote Android client project takes about 45 minutes. That includes build, full testing, and auto-deploy for internal Evernote Android clients.

  3. Hi

    What means exactly “Android Clients”. Are you using a couple of real android devices which download automatically the builds and run the tests aotuomated? If yes, which tools are running theses devices in order to run the test.

    Thanks

  4. Mike, thanks for the question. I checked with our Android test automation guy, and he said we are using Spoon to install / run automated Android test on several phones/tablets connected via USB. http://square.github.io/spoon/

    We also have several Evernote employees who run the latest builds on their Android devices. They get these builds through an auto-update process that often delivers 2 or more new builds per day.

  5. I would love to understand, how do you guys handle only UI deployments since it has a chance of breaking due to browser caching.
    Do you guys use Grunt or a similar service anywhere in between?


Leave a Comment

* Required fields