Comments on: Evernote Indexing System

By: Owyn

Owyn — Tue, 04 Oct 2011 23:17:29 +0000

Alex. This has been a big help in clarifying my understanding of how Evernote works.

Thanks.

By: Alex Pashintsev

Alex Pashintsev — Tue, 04 Oct 2011 22:19:12 +0000

>> What is the minimum change required to a note to force a re-index of its resources?

Actually, it is not the note change that matters, but the *resource* itself. The ‘fingerprint’ is stored separately for each of the resources, so if you want an image to be indexed anew, it should be actually different — even if in a single bit. Changing the note that contains it will have no effect.

>> I do understand that processing time for a resource will add to total transaction processing time.

Yes – while it is just seconds for an average image, it will certainly be much longer for a huge multi-page PDF.

By: Owyn

Owyn — Tue, 04 Oct 2011 13:34:09 +0000

PPS: Guessing 3 sigma for premium users of about 5 minutes, excepting any abnormal site maintenance problems.

By: Owyn

Owyn — Tue, 04 Oct 2011 13:24:58 +0000

PS: I do understand that processing time for a resource will add to total transaction processing time. However, the worst case example that Dave Engberg mentioned in his ETC presentation was, IIRC, about an hour for a specific PDF.

By: Owyn

Owyn — Tue, 04 Oct 2011 13:18:18 +0000

My example of a null change got trashed by your html filter.

e.g. [space][backspace]

By: Owyn

Owyn — Tue, 04 Oct 2011 13:14:53 +0000

Thanks for the response. Very helpful.

Bit confused about the queue management as described. It suggests LIFO for Premium and FIFO for Free.

The service times as described suggest service time distribution of 1 sigma of 2 minutes, 2 sigma of 15 minutes, and 3 sigma of perhaps 2 hours. However, I have seen frequent comments on forum about unsearchable and therefore unindexed resources that have lasted days. Some of those are explainable by the user not understanding the desktop client 2 sync requirement to see the indexing. Some are not.

What is the minimum change required to a note to force a re-index of its resources? Something like touching the note body with a null change? e.g .

Wish there was a more direct attribute that could be searched/tested to determine a notes index status, however…

Thanks again for the very useful information.

By: Alex Pashintsev

Alex Pashintsev — Tue, 04 Oct 2011 00:21:04 +0000

The post was intended to be more of a technical design overview, thus did not focus so much on the usage side — but let’s touch it too.
To your questions:
– free and premium queues are actually the same queue, only that premium images are inserted in the front, instead of being placed in the end of it.
– so if the indexing system is not too busy, there may be not much of a difference in processing time – in both cases image will pass the full cycle in a matter of minutes. But if there is a spike of user activity, the reco queue may span an hour or more. In this case premium images will still be processed in couple of minutes – while regular submissions will have to wait.
– to check if a note was already indexed, sync your desktop EN client once again in a few minutes after you created the note. For PDF notes, check for the option ‘Save Searchable PDF’ when you right-click the PDF object in question. If this line is present in the menu, the note was already indexed. For image notes, choose to ‘Export’ the note to the ‘archive’ .enex format. This will produce an XML file that you can open with a text editor or your browser. Look in the end for the tag which wraps the image indexing results – if it is present, image was processed and is searchable now.
– once a note is created and synced up to the Service, its resources are processed according to the user settings at the moment. This includes language preferences, premium status, etc. If any of those are changed after the note was synced up, it will not affect the processing. The note’s images will still stay in the end of the reco queue if the user status changes to ‘premium’. They will still be recognized as English-only even if the user preferences change to ‘Japanese+English’ from ‘English’ while the resources are still in queue.
N.B. EN Service keeps the ‘fingerprints’ of all images and PDFs in the system to avoid storing multiple copies of the same file. So even after the language preferences were changed, same image included into a different note will just assume personality of its first copy — including its reco index. So to have the index change to a different language, it will be required to provide a slightly modified version of the image, or request all nothes reindexing in the user personal settings.

By: Owyn

Owyn — Sun, 02 Oct 2011 21:20:52 +0000

Ummm. Fixing typos in previous post.

Good article, however, it does not cover several of the most important issues from a user’s perspective.

- How are the Free and Premium user queues managed?
- What are typical service times for the queues?
- How can a user determine if a note’s resources have been indexed?
- What happens to pending unindexed resource requests when a user upgrades from free to premium?

By: Owyn

Owyn — Sun, 02 Oct 2011 21:00:56 +0000

Good article, unfortunately, it does not cover several of the most important issues from a user’s perspective.

- How are the Free and Premium user queues managed
- What are typical service times for the queues
- How can a user determine if a note resource has been indexed
- What happens to pending unindexed resource requests when a user upgrades from to premium.