The following is a behind the scenes walkthrough of Descriptive Search within Evernote by our Augmented Intelligence Engineer, Adam Walz. For our public announcement of Descriptive Search, click here.
Evernote has always had a great keyword search experience being able to surface notes that match not only in the body of the note but also within images and documents. However, when confronted with a blinking cursor in the search bar we often find ourselves struggling to remember what exactly we named that particular note. We realize that while keywords are an integral part of the search experience, we as humans have a natural tendency to relive our memories by the places we’ve been, the dates we created our notes or even the types of files the note contains. How often do you wish you could just search by typing San Francisco last week ? Now, with Descriptive Search for Mac you can!
Realizing that search needs to evolve from keywords, Descriptive Search was our attempt to create a natural extension of your thought process. Because this is an ambitious attempt, we decided to start with support for the English language on the Mac client. In forthcoming releases, we will expand to additional languages and platforms.
Tearing Open The Seams
Evernote has always had an advanced Search Grammar which also allows you to search the metadata associated with your notes and resources. However, using it requires a syntax that is difficult to remember, and far from intuitive.
For example, a search for all your notes that contain all ‘office documents’ would entail typing the query:
any: resource:application/msword resource:application/vnd.openxmlformats-officedocument.wordprocessingml.document resource:application/vnd.ms-excel resource:application/vnd.openxmlformats-officedocument.spreadsheetml.sheet resource:application/mspowerpoint resource:application/vnd.openxmlformats-officedocument.presentationml.presentation
Now that’s a query that I wouldn’t expect anyone to remember or want to type! Descriptive Search allows you to do this by simply typing ‘office documents’.
Our implementation goal for the first version of Descriptive Search was to support a large subset of what our Search Grammar supports by adding a natural language query interface so you can use everyday language to find notes the way you remember them.
At an implementation level, this comes down to a classic Parsing problem. Parsing is the process of analyzing the syntax of a piece of text to pull out the important parts and understand the meaning behind the text. When we have to deal with the nuances of multiple languages, dialects, and an ever expanding dictionary things get interesting.
Natural languages are characterized by their lack of a strict grammar. You could write the same search many different ways, and Evernote should be able to understand that any number of queries could have the same meaning. In the world of natural language processing this is called a semantic grammar. Our semantic grammar is specifically created to pull out the meaning in your query, and throw away anything that is not helpful in finding your notes.
This all comes together in the following sequence of steps:
1. Preprocessing Step
Like most natural language systems, our goal in the preprocessing step is to sanitize the users input query to remove irrelevant language nuances. This begins by detecting the words in the users input query. While this might seem like a trivial task, finding what constitutes a word for languages like Chinese and Japanese gets into difficult areas of natural language processing. This becomes one of the most critical steps to get right. Since we did not want multiple conjugations of a phrase to cause parsing to fail, we then remove common words from the input language (stop word detection and removal) and condense the words down to their root word (stemming).
Like everything else we do at evernote, we also wanted to make your job easier, so we expand your query using our Type-Ahead Search system. This means faster, more accurate searches for the user with less typing.
2. Honoring User Specified Contextual Hints
It is often the case where the meaning of a search can be ambiguous when seen only one word at a time. The word ‘cute’ in the previous example could be a tag, part of a notebook title, or simply a keyword in the text of your notes. However, if you instead specified ‘tagged cute’, we look at the neighboring words to determine if you specified a contextual hint for the word. In this case the hint would be ‘tag’. We then verify that this is a good suggestion by doing a quick cross reference against your tag list to make sure that you actually do have a tag named ‘cute’.
3. Grammar Parsing
The next step is what makes Descriptive Search so powerful. A word such as ‘image’ may not appear as a keyword anywhere in your notes, and even if it did, when you type ‘image’ you are most likely not looking for a keyword in the text of your notes. You want to see notes with have attached image files. This is where parsing comes in.
What we have at the heart of this system is a Semantic Grammar that is very carefully handcrafted to meet the specifications of the particular user language, such as English. This grammar takes into account the nuances of the language with several very advanced parsing rules, even finding synonyms of the words we want to detect. This grammar is then cross compiled to the native platform which also gives us the benefit of maintainability and portability across platforms.
At runtime, we pass the preprocessed query through this semantic grammar parser which looks for character patterns that closely match one of the rules in our grammar. All matched character patterns are then replaced with an unambiguously formatted search token.
As seen in the example below, the pattern ‘image’ detected by the grammar parser is replaced by its equivalent evernote search grammar resource token.
4. Content Matching
While the Semantic Grammar can cover patterns that are common across all our Evernote users like the existence of date ranges or file types, there is another very important category of queries which is derived from the users own personal content, e.g. your own notebooks or tags or places you’ve been. In this step we will close this gap by taking the unmatched words left in your query and checking them against your own user index.
Going back to our example query from the previous step (‘cat tag:cute resource:image/*’), the word cat is still unmatched. We cross reference the unmatched words in the query against the metadata in the search index to find that maybe you have a notebook named ‘cat pictures‘ and that you have a tag named ‘cats‘. In the case of multiple metadata matches like this, we use a probabilistic model to determine which of these suggestions closely matches your intent and query. Remember, if we happen to get this wrong, the Contextual Hint step can be used to provide more context about your meaning.
5. Suggestion Creation
We have now found a match for all of the important words in your search. However pulling out the meaning of your query behind the scenes is only part of what we feel makes a search “Descriptive”. The user experience matters a great deal to us and we know you wouldn’t want to see a result in the form of “notebook:cats tag:cute resource:image/*”
The Suggestion Creation step formats the search grammar result into an an equivalent descriptive phrase in your language making the suggestions conversational and easy to read.
If you haven’t already used this feature on the Mac client I encourage all of you to give it a try. If you need a little help getting started you can refer to our knowledge base article. At the same time, our team is hard at work bringing this experience to all the other platforms and increasing our support for different languages and a broader range of queries.
Keep an eye out for these new features with coming releases.
Adam Walz is an Augmented Intelligence Engineer at Evernote where he is focused on taking the Search experience to the next level. The Augmented Intelligence team is on a mission to improve the search experience for Evernote and make memories more discoverable.
[Opening] Join Adam and the ‘Augmented Intelligence’ Team – we are hiring! Software Engineer