Evernote Tech Blog

The Care and Feeding of Elephants

So API Together: Evernote and Thrift

When we started to plan the Evernote service in 2007, we knew that we would need to support both “thin” clients (like web browsers) and “thick” synchronizing clients on the day that we launched. This forced us us to think about remote protocols and client APIs before we built any web GUI, rather than waiting a few months to staple an API onto an existing web service.

Our application forced certain requirements onto this API, including:

  1. Cross-platform. When we launched in February ’08, we had production code using server-side Java, client-side Win32 (and WinMobile) C++, and client-side Obj-C Cocoa.
  2. Binary efficient. Clients synchronize structured Notes that may contain hundreds of embedded images totaling tens of megabytes. We wanted an API that would transmit a 15MB note using ~15MB of bandwidth.
  3. Forward/backward compatibility. Once someone has installed version 1.1 of our client on their laptop, we don’t want to force them to upgrade the local software every time we extend our data model.
  4. Native bindings. We didn’t want to write a bunch of parsing/marshaling code on every client. This is time consuming and error-prone, and tends to make #3 unlikely in practice.
  5. Standard-y and/or open-source. All things being equal, we didn’t want to lock our service API into a single commercial ORB product for the usual reasons.
  6. Not gigantic. I’d prefer not to deploy a 1MB runtime with 200 classes on every mobile client.

We spent a couple of months researching and testing various alternatives. XML-RPC or SOAP were strong in some areas (1, 5).  ZeroC’s ICE was strong in others (2, 4). We also argued a bit about rolling our own little ad-hoc protocol in a fit of NIH.

friend-of-Evernote recommended we take a look at Facebook’s recently-open Thrift framework. Facebook was using this internally for back-end servers to exchange messages with other internal servers, frequently across language boundaries (e.g. PHP to C++). Other folks seemed to use Thrift for a similar task: back-end internal server communication.

We were looking for something a bit different: a framework that could be used for server-to-server communications in our backend, but also for massive client-server synchronization over the Internet.  At the time, Thrift was the best fit we could find for our requirements:

  1. Cross-platform. We define our data model and service operations using Thrift’s Interface Definition Language, and then run their compiler to spit out client (and server) structures and service stubs for a dozen different languages.
  2. Binary efficient. If we specify a ‘binary’ data field holding 1MB of data, that will marshal as 1MB on the wire.
  3. Forward/backward compatibility. This is where Thrift really shines. If you are careful and understand how Thrift works (big caveat, obviously), then you can add structures, fields, service methods, and function parameters without breaking existing clients. The Windows or Mac clients we released 3 years ago can still sync with Evernote today.
  4. Native bindings. See #1. At the time, Thrift didn’t include anything for Objective-C Cocoa, so Andrew McGeachie (our one-man Mac team) added this to the Thrift compiler.
  5. Standard-y/Open-source. Facebook gave Thrift to Apache, which was awfully nice of them.
  6. Not gigantic. The Thrift runtime libraries and generated code were really small and straightforward. I could easily read them and understand exactly what they were doing. (Since then, it’s grown a bit flabbier as various contributors’ use cases have been added and optimized, but I’d say that it’s still pretty compact compared to alternatives.)

The end result was the Evernote Service API, which allows all of our own clients (and hundreds of third-party applications) to talk to a common API using generated native code. With over three million active users, frequently using Evernote on multiple platforms, I think that the majority of computers/devices running Thrift code are Evernote clients.

Et tu?

You’re going to implement an API for your web service … should you use Thrift, too?

If your application had the exact same requirements as Evernote, then Thrift may be a good choice. If you don’t have a complex data model with big binary attachments (requirement #2), then the answer isn’t so clear.

Web services with simpler data models tend to use simpler REST protocols with ad hoc XML or JSON data marshaling. This approach means that simple operations are really simple to test and implement. If I only need to do two things with Twitter’s API, I can test them manually from the command-line with curl/wget and then hand-roll the code into my app with printf/println/regexps/etc.  This means that there’s a very low bar for third party developers to get started with this type of API.

Our Thrift API imposes a higher burden on developers, who need to get all of the transport layer details done and the library dependencies into their app before they can do any testing at all. We’ve tried to help this a bit with the sample code and packaged libraries in our API distribution, but it’s still going to be more work than a simple REST scheme.

On the other hand, the low bar for these types of ad hoc data marshaling APIs tends to be offset by subsequent compatibility hassles (#3, above). Our Twitter gateway uses the third-party Twitter4J library to talk to Twitter’s REST-based API. Our gateway has broken twice in the last year or so due to Twitter’s server-side changes and Twitter4J’s incorrect assumptions about the interpretation of the ad hoc XML data structures (e.g. the maximum size of a twitter direct message ID).

A more formal IDL and native code generation may help provide better long-term client stability, so the initial Thrift overhead for developers might be justified for some services that are concerned about client stability and longevity.

15 Comments

  1. Ah, yes. I remember porting Thrift to WinCE/C++. That wasn’t too bad. Good times.

  2. Ah, I always wondered why you settled with Thrift. Thanks for the clear reasonng on your choice. I didn’t know about the forward/backward compatibility.

    Do you know if Facebook still uses Thrift heavily?

    • I haven’t heard what Facebook is doing with Thrift these days.

  3. Is protobuf(google) a better choice for now?

    • I think that it would be a good alternative. When Google first released it, the language support was limited to only a few languages, and the backward/forward compatibility wasn’t as strong. It looks like the language support is much stronger now (http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns), and I think the compatibility issues may be resolved, so it could be a good alternative. But I haven’t spent any time on it, so I don’t have a good recent comparison. Older comparisons:
      http://stackoverflow.com/questions/69316/biggest-differences-of-thrift-vs-protocol-buffers
      http://stuartsierra.com/2008/07/10/thrift-vs-protocol-buffers

  4. Do you know if there will ever be a Javascript implementation of the Thrift API? I’m referring to the discussion at http://forum.evernote.com/phpbb/viewtopic.php?f=43&t=21893

    Is it technically impossible or is there just no demand?

    • The full API can’t be expressed in Javascript, which has no concept of binary data. (I.e. there’s no “byte[]” in Javascript). So we’ve done some adaptations for internal usage, but don’t currently have something safe and clean for partners.

  5. Hi,
    I noticed the last comment on this is from 2011. We’re also using Thrift to help clean up & combine our internal/external APIs. So I’d like to ask if Thrift’s still working out well for Evernote or if you’re exploring other options these days ?

    We’re also kicking around the idea of using Thrift client-side with our mobile apps to allow for reuse of Thrift generated code, but I’m finding not everyone understands or agrees with the benifits of this idea.

    • Thrift is still a good fit for us. It handles heterogeneous data with big binary blobs well, backward compatibility is reliable, and the cross-platform support is solid.

      A simpler app that just needed to exchange small messages or single atomic files would probably be fine with a simple JSON REST protocol. But encoding a 25MB note with multiple binary images into JSON would be a mess, and the workarounds are clunky. So we’re still happy with Thrift.

      But we stopped following the head release of Thrift a couple of years ago. The emphasis of the current maintainers was more for heavy server-server usage, with a heavier runtime dependency on clients.

  6. >> But we stopped following the head release of Thrift a couple of years ago. The emphasis of the current maintainers was more for heavy server-server usage, with a heavier runtime dependency on clients.

    Can you explain a bit ? What do you mean by “headless thrift”. Thanks in advance.

    • The generated code started to include more dependencies to libraries (e.g. logging frameworks) that didn’t add any value when the endpoint was a PC or smartphone. Those dependencies also made our API more painful for third-party developers to work with.

      It also started to incorporate a lot of extra meta-programming structures into each generated service and class which bloated out the number of classes and methods considerably. That doesn’t matter on a server with 64GB of RAM, but (for example) Android apps have a limit on classes and methods that make it undesirable to generate dozens of each for every service method.

      I’m sure those changes were useful for server-side applications, where a few more dependent libraries and extra code don’t really matter. But we found the original lean design of Thrift more useful for client/server, so stopped tracking the head.

  7. Thanks Dave!
    I am also regretful since the generate client-stub of so heavy with so many libs dependices.

  8. They seem to be using upgraded version of thrift which they named as fbthrift. The details about fbthrift could be found on https://code.facebook.com/posts/1468950976659943/under-the-hood-building-and-open-sourcing-fbthrift/.

  9. Hi Dave,
    Which thrift version (both compiler and runtime library) does Evernote use for the compiling and runtime lib? Looks like Evernote modified the thrift runtime library and inline it into the SDK, is the purpose only for reduce the footprint by remove the server side code?

    We are also considering use Thrift+HTTP as our public API/SDK framework, although our reason is different (our main reason is least SDK developing and maintain effort without hurting code quality). My only concern about Thrift is it does not generate readable/simple restful API (Thrift do support JSON but it does not quite readable and not non-thrift code friendly) and the developer may complain.

    An Thrift equivalent framework which can generate multi-language stub then serializing to clean and readable JSON is desirable for us. E.g. a multiple language JSON Schema code generator + JSON RPC are what needed, but looks like the first tool is missing, what a pity.

    • Originally, we only distributed our generated classes and not the Thrift runtime library.
      Now, we include the Thrift runtime library with our SDK to make it easier for our third-party developers to work with the API.

      Simple REST/JSON APIs are great if your data model is relatively simple. If you’re passing complex data structures with binary data, then the simplicity starts to go away.
      One advantage of a uniform compiled IDL is that you avoid parsing ambiguity in your developers. E.g. if you are spitting out JSON with numeric values, then a developer may parse those into either int32 or int64 without any problems. But if your server returns a value that doesn’t fit in 32 bits, all of the developers who chose an int32 will have problems (e.g. http://techcrunch.com/2009/06/12/all-hell-may-break-loose-on-twitter-in-2-hours/).
      With a compiled IDL, you can specify that a number is a 64-bit integer and then not worry about your developers using the wrong native data type with their hand-rolled JSON parsing code.


Leave a Comment

* Required fields