With more than 100 developers spread out among several countries, a robust continuous integration and build automation practice is crucial to Evernote’s development process. Through automated builds and tests, we can head off problems as soon as they are introduced into our code, resulting in a more efficient process and a reduced chance of impacting our users. Our overall strategy is based on the underlying principles from Martin Fowler’s 2006 article on Continuous Integration.
The Engineering Services team at Evernote operates a Jenkins master server and about 30 slave machines for building and testing. The slaves run a variety of operating systems (Windows, OS-X, and Linux), SDKs, build tools (Xcode, Android SDK, Visual Studio, Maven, etc), and test suites. Standard practice is for each build job to run on at least two slaves, so that if one is offline for any reason the other machine will seamlessly handle the build.
Our typical build-test-release workflow includes a verification build every time code is committed, a daily build for internal testers, and public builds (both beta and final release) that, in the case of our client applications, trigger an automatic deployment from within the client app.
In our Jenkins environment we build not only our Java-based web service, but also various applications for iOS, Android, Windows, OS-X, Windows Phone, Pebble and Blackberry. Every team is a little different, but the Evernote for Android client app provides a useful example of how we leverage automation.
- Every time an Android developer commits code to the main branch, a build is triggered. In the case of the Android client app this automated build is an internal test version that’s automatically deployed, so it’s important to make sure these builds pass basic usability testing.
- A successful build job triggers a build verification test (BVT), which consists of automated functional tests running on Android devices.
- A successful BVT triggers a deployment of the client app by pushing out a xml file to a location that is monitored by the clients.
- When the clients detect the new file, they immediately download and install the new version.
Our QA team receives emails when release candidate versions are ready for testing and sign-off, and these executables are readily available for download via Jenkins. The Android team publishes public beta and release versions through the Google Play store. All released versions of our apps are built and signed in our Jenkins environment.
For the Java-based Evernote web service, the workflow is relatively similar to the above. Developers commit code to the web service component projects, and those commits trigger verification builds that incorporate a series of tests. Successful builds trigger an automatic deployment to our staging environment, where QA engineers perform tests against the new code. We do a public release of our web service almost every Wednesday, at which time the build that’s been approved by QA is deployed by our operations team.
For our Maven-based jobs, we manage dependencies across projects using Artifactory, an open source tool that tightly controls how and where build artifacts are used in our environment.
Jenkins offers about 900 open source plugins of which we use nearly 75. Some of the most important plugins for our environment include:
- Git, Maven, Gerrit Trigger
- Email Extension
- S3 Publisher
- Parameterized Trigger
- Token Macro
A follow-up blog post will cover some interesting parts of our web service production deployment process.