Evernote experienced a series of hardware failures on one of our servers between July 1st – July 4th, which potentially affected 6,323 users worldwide. As a result of the failure, some of the notes created and edited by these users between July 1st and July 4th were not properly recorded on the Evernote servers. We immediately contacted all affected users via email and our support team walked them through the recovery process. We automatically upgraded all potentially affected users to Evernote Premium (or added a year of Premium to anyone who had already upgraded) because we wanted to make sure that they had access to priority tech support if they needed help recovering their notes and as a partial apology for the inconvenience.
Less than one fifth of one percent of our users were in the potentially affected group and we were able to identify 100% of them from the server logs. Because of this, we decided not to post a wider announcement to give our support staff the time to work with the actual people affected instead of fielding a flood of requests from the more than 99% of users who were not in the affected group but had no way of determining that themselves.
If you did not receive an email from us in early July, then you were not affected.
Because of the redundancy inherent to Evernote (copies of notes saved locally, email and browser history), the majority of the affected users were able to recover all their notes.
We want to assure you that this was a one-time issue. We have significantly improved our reporting and redundancy infrastructure to ensure that it does not happen again. We sincerely apologize for the inconvenience to our affected users. Even though most of them didn’t wind up losing any data they still had to read through a lengthy and potentially worrying email. For the ones that did lose data, we hope that knowing exactly which notes were effected over a four day period is enough information to recover or recreate the most important ones.
We received replies from several hundred of the affected users, and we are extremely grateful for their understanding and continued support. We are posting this now because of erroneous information that we’ve seen popping up on the web.
For the technically minded, here’s what happened
Every user’s data is stored on a “shard”. A shard is made up of a server together with a redundant fail-over server. If there is any problem with a server, the system automatically fails over to the second server in the shard. We currently have 37 shards. Shard 22 was the one that had problems last month. The data in each server is stored on a RAID 1 (fully redundant) array. All data is also backed up on-site and off-site. A full copy of your notes are also stored on the Windows and Mac clients (and the iPhone and iPad clients for Premium users who enable that option). This means that every note in Evernote is stored in at least six redundant locations: the disk on the primary server, the RAID mirror, the fail-over server on the shard and it’s RAID mirror, the on-site backup and the offsite backup. Most users also have another one or two full copies on their local clients. This makes data loss in Evernote extremely rare. The problem with shard 22 was a very idiosyncratic intermittent combination of hardware problems with both the primary server and the fail-over mechanism. Basically, the shard kept failing over back and forth between two servers over the time period causing some of the data created during that time to get overwritten. Everything created before the failure was easy to recover from backup. The chance of this particular sequence of failures happening again is extremely low, but we’ve modified the fail-over mechanism, just in case, to make sure that it is impossible to override data even in the worst-case scenario.
