As we stated in previous articles, security and privacy are very important to us. In addition to the product security enhancements we made to the service, we recently updated the SSL certificate for www.evernote.com to use a 2048-bit key. Many other companies are going through the same process because by the end of 2013, certificates with 1024-bit keys issued by major certificate authorities will expire.  Our original 1024-bit certificate was set to expire in August of this year, but we prepared early to make sure we had enough time to address any contingencies.
A 2048-bit certificate is more secure because it is computationally more expensive to crack on modern hardware without the private key, but it also results in performance implications on systems responsible for processing them. The initial SSL handshake is the most taxing part of the process because it involves public-key cryptography and asymmetric encryption. After the handshake is complete, the client and server should have a secure channel to privately exchange data using symmetric cryptography, which is not as computationally expensive as the SSL handshake. We use server load balancers from A10 Networks which have built-in SSL chips that help offload the SSL handshake and encryption from our servers. We worked closely with our vendor to make sure our load balancers had the horsepower to handle the volume. We also executed a cautious transition plan that helped make the update smooth and successful.
Our transition started in our non-production environments, since the certificates there were scheduled to expire months before our production site. Our QA and staging environments have been using a 2048-bit certificate on separate load balancers since January of this year, so we were confident that all of our Evernote clients were fully compatible. The certificate on our developer sandbox was also updated in September 2012, so our third-party developers on Evernote Trunk have been able to verify compatibility as well. We ran some benchmark tests as well, but our staging load balancer hardware did not have the same performance capabilities as the ones in production.
Early in June, we acquired a new 2048-bit certificate from Thawte to use temporarily on a separate virtual server running on our production load balancers. This allowed us to keep the www.evernote.com virtual server on the original 1024-bit certificate from Verisign. Internally we called this test virtual server www-2408.evernote.com. With some internal DNS tweaks, we were able to use our normal QA and validation procedures to verify the new certificate in production.
After we were sure the test virtual server was functioning properly, we ran tests on the primary www.evernote.com virtual server. The ability to transparently change between the old and new certificate on our load balancers gave us a lot of control and flexibility for each test; we did not have to restart any services or interrupt existing sessions. During each test, we used our monitoring tools to track important performance data. We use collectd and graphite to measure system and network performance internally. We use a combination of Pingdom and Thousand Eyes to give us statistics from a remote point of view. These were the key metrics we analyzed:
- Load balancer CPU and memory utilization
- SSL transaction rates, or the number of new SSL handshakes per second
- SSL concurrency, or the number of SSL connections active at the same time
- SSL request latency, or the time it takes to negotiate a new SSL handshake
We started with a short, 4 hour test on Saturday, June 8th, which is usually when our lowest traffic levels occur. The test ran successfully without any noticeable impact to the load balancer CPU and memory utilization. We expected slightly higher SSL handshake latencies, in part because a 2048-bit certificate with the intermediate CA certificates resulted in more bytes being transferred between the client and our load balancers. Most importantly, we did not see any abnormal deviations in SSL latency, connection rates, and concurrency; this indicated that clients were successfully establishing SSL sessions on the new certificate. Wide swings outside of our expected baselines in any of those metrics would have meant the performance was unstable.
We repeated the test a couple of times during the week: 8 hours on June 13th, and a 7-day period from June 18th to the 25th. Here is a graph showing our SSL request rates between Monday, June 17th and the 24th. The blue area represents the measured rate and the dashed green shows our expected baseline. I omitted the Y-axis intentionally to hide our volume, but it at least shows how our peaks come and go.
After we were satisfied with the performance on the 2048-bit certificate, we finally cut over www.evernote.com to an updated, 3-year certificate from Verisign on June 27th.
You can view our server certificate from any web browser by visiting our web client. The process varies between browsers, but here is our new certificate as seen from Chrome on Mac OS; just click the padlock, Connection, and Certificate Information.
As a reminder, all connections from an Evernote client or a web browser to www.evernote.com should always use SSL; this includes all free, Premium, and Business users. In any case, we are glad to be making progress in improving the security and privacy of the service for our users.
 CA/Browser Forum (2013, May 31). “Baseline Requirements for the Issuance and Management of Publicly-Trusted Certificates, v1.1.5” (Appendix A). Retrieved from https://www.cabforum.org/documents.html