Comments on: Shard Boiled

By: Dave Engberg

Dave Engberg — Sun, 06 May 2012 05:37:20 +0000

We started with pairs of physical boxes, but found the replication management to be more of a hassle, and the CPU utilization was more wasteful since 50% of the boxes were sitting completely idle at all times. Virtualized server images are easier to manage, since the entire “shard” can be replicated over DRBD and brought up on a second box.

We did a fair amount of testing to find a combination of kernel and virtualization platform to minimize the IO overhead. We’ll try to put together a post with some of these details. Short version: kernel 3.2.10 on Xen provided significantly better local random r/w IO to the SSD RAID than earlier kernels and/or KVM. We also tested various file systems and decided to go with ext4.

By: Jamie

Jamie — Sat, 05 May 2012 21:13:03 +0000

Hi Dave,

Very insightful blogs into the world of Evernote and plug ins in general. I was just wondering if you have written a concise report on the functional description of Evernote?

Thanks,

Jamie.

By: Peter

Peter — Sat, 05 May 2012 05:31:25 +0000

Dave, thanks for the awesome post. I enjoy reading the post very much.

I am wondering how do you determine to run the service in a virtual machine given you can control the hardware and the I/O cost of a VM may be high? Is it because a VM is easy to manage?

Also, what file system is running on top of SSD? Do you use a log structured file system or an ordinary file system, e.g. ext4, xfs? Or mysql’s storage engine could make better use of SSD?

Thanks.
Regards,
Peter

By: Dave Engberg

Dave Engberg — Fri, 04 May 2012 13:22:56 +0000

Currently, we’re putting shards and their WebDAV servers in the same rack so that all of the IO goes through the top-of-rack switches in the same rack, and none of it goes through the core switches. This will prevent the file traffic from competing with service traffic on the core, especially if we ever need to rebuild a WebDAV server that fails completely. But we may change this in the future, or just migrate WebDAV servers out when they become “full.”

Resource files are written twice locally and once offsite, each on RAID6 arrays. The offsite copy is the “backup” copy since we never delete from the offsite array. The MySQL databases are replicated locally and then backed up offsite nightly via Xtrabackup and rsync copying.

By: David E Y Sarna

David E Y Sarna — Fri, 04 May 2012 10:31:28 +0000

I am sure you are familiar with Xeround http://xeround.com.
It seems like it might offer some benefits for Evernote, as it is a NOSQL that and is robust, self-healing and self-scaling and requires no code changes. Happy to make and intro to the CEO if you like.

By: Darren Millin

Darren Millin — Fri, 04 May 2012 07:33:04 +0000

Hi Dave,

Thanks for a great post. My backfround is in infrastructure/sys administration.
I have a couple of questions:

1. Why host both file servers in the same rack, and what happens when power is lost
during a synchronous write?

2. I’m not clear on how the data is backed up. Is the data just replicated off-site, or
is it replicated, and the replicated data is then backed up?

Regards,

Darren

By: Phil Libin

Phil Libin — Thu, 03 May 2012 14:47:38 +0000

That’s pretty great.

By: Dave Engberg

Dave Engberg — Thu, 03 May 2012 14:20:22 +0000

By: dave™

dave™ — Thu, 03 May 2012 14:18:46 +0000

here’s one http://imgur.com/Wq0SU

By: Dave Engberg

Dave Engberg — Thu, 03 May 2012 14:04:16 +0000

We’ve supported clients that can’t display certain things, but we want people to feel confident that anything that they put into “Evernote” will work well in “Evernote”, for reasonable definitions of the word “Evernote.”