API

Wearables Update: Evernote for Pebble Now with Cyrillic and Image Support (Really!)

Posted by Damian Mehers on 23 Apr 2014

Posted by Damian Mehers on 23 Apr 2014

pebble-story-232x232

The following is a behind the scenes walkthrough on our latest updates for Evernote for Pebble by our lead wearables engineer, Damian Mehers. For our announcement of Evernote for Pebble, click here.

Since the initial release of the Evernote app for the Pebble we’ve made several improvements.  I’m going to briefly talk about what’s changed, and then dive into technical details about how I implemented two of the more challenging changes: supporting custom fonts and generating bitmaps from JavaScript.

First we will review the straightforward improvements:

Preserving line-breaks in notes

If you used the initial release of Evernote for Pebble, you may have noticed that some of your notes looked a little garbled.  This was because the app was not preserving line-breaks in your notes:

Screen Shot 2014-04-16 at 18.23.53 pebble-screenshot_2014-04-16_18-18-39

With the latest release, we’ve address this and it should be much better now:

pebble-screenshot_2014-04-16_18-19-37

Note font size

Soon after we launched Evernote for Pebble, we received helpful feedback.  Thankfully, some people have much better eye-sight than I do.  We added a setting for those keen-eyed people that can read smaller fonts:

pebble-screenshot_2014-04-17_13-22-47 pebble-screenshot_2014-04-16_18-29-34 pebble-screenshot_2014-04-16_18-29-40 pebble-screenshot_2014-04-16_18-29-24 pebble-screenshot_2014-04-16_18-29-47 pebble-screenshot_2014-04-16_18-29-55

Reminder order

Some people use Evernote reminders to “pin” their most important notes to the top of a notebook, and some people use them as actual reminders.

For the “pinned important notes” use case, it makes sense to list the reminders in the order in which they are pinned on the desktop. For the “Notes as actual reminders” use case it makes more sense to list them in the order that each reminder is set to go off.

With the latest release, you choose the order, just like you can on Evernote for Mac.  The default is to order them based on the order in which they are pinned, but if you use reminders as actual reminders, you can see them by reminder date instead:

pebble-screenshot_2014-04-17_13-22-47 pebble-screenshot_2014-04-16_18-35-36 pebble-screenshot_2014-04-17_17-56-04

Custom fonts & Cyrillic Support

The Pebble doesn’t support fonts with accents on them, which is why the initial release of Evernote didn’t support such fonts either.   You’ll have noticed that when you receive a text message with an emoji in it, you just get little squares in the notification on your Pebble.  The same thing happens for notes with cyrillic characters.  Look what happens to the note’s title below:

Screen Shot 2014-04-16 at 18.39.58

pebble-screenshot_2014-04-16_18-40-26

Soon after the initial release we received a deluge of requests for support for PebbleBits which is an unofficial way in which the Pebble can be made to support additional fonts.  Essentially you generate, download and install custom firmware which includes additional fonts.

In order to add support for this in the Evernote JavaScript companion app that runs on the phone, so that it can send the correct strings over to Evernote app running on the Pebble, I needed to convert the UTF-16 strings containing note text over to the equivalent UTF-8 bytes to send over to the Pebble.

image_thumb27

This is the JavaScript code which pushes the UTF-8 bytes into a byte array buffer from a UTF-16 String:

// From http://stackoverflow.com/a/18729536/3390
function addUTF8StringToBuffer(buffer, text) {
  var utf8 = unescape(encodeURIComponent(text));
  for (var i = 0; i < utf8.length; i++) {
    buffer.push(utf8.charCodeAt(i));
  }

  buffer.push(0);
}

You’ll see that I am NUL terminating the string for consumption by the C code on the Pebble.

Once you’ve generated and installed custom firmware with the appropriate character set(s) you’ll see the note titles and content displayed properly:

pebble-screenshot_2014-04-16_18-51-37

Note bitmaps – Images on the Pebble!

Previously when you viewed a note in Evernote for Pebble, the app looked to see if there was any text in the note content, and if there was it displayed the text, otherwise it displayed this message:

pebble-screenshot_2014-04-17_12-30-08

Wouldn’t it be cool, I thought, if we could display an image on the Pebble if the note contains an image?  That way, when I’m shopping for yogurt, and I’m faced with an overwhelming array of dairy produce, I could glance at my watch to remind myself of what it was I was supposed to buy, assuming I’d previously captured a photo of the yoghurt in Evernote on my phone:

Screen Shot 2014-04-17 at 12.34.03

There were a couple of challenges to overcome.  The Pebble has a black-and-white display, and it requires its bitmaps to be in a specific format with a specific header.  On the other hand you can store all kinds of images in Evernote, including PNGs, JPGs, BMPs etc.  And they can have all kinds of color depths.

Using the Evernote Thumbnail API

Fortunately there is an Evernote Thumbnail API, which as a developer you can use to request a thumbnail for a specific note or resource.  What is more, you can request a specific thumbnail size, and the icing on the cake is that you can request that the thumbnail be in a specific format, such as a bitmap.

This was perfect.  In my JavaScript I request the thumbnail using code like this:

  var xmlhttp = new XMLHttpRequest();
  var url = pbnote.webUrl + '/thm/note/' + noteGuid + '.bmp?size='
            + pbnote.CONSTS.THUMBNAIL_SIZE;
  xmlhttp.open("GET", url, true);
  xmlhttp.setRequestHeader('Auth', pbnote.authTokenEvernote);
  xmlhttp.responseType = 'arraybuffer';
  xmlhttp.onload = function () {
    if (xmlhttp.readyState == 4 && xmlhttp.status == 200) {
      pbnote.sender.convertToPebbleBitmap(xmlhttp.response, onOk, onError);
    } else {
      pbnote.log.e('fetchBitmap xmlhttp.readyState=' + xmlhttp.readyState + 
                   ' and xmlhttp.status=' + xmlhttp.status);
      onError(xmlhttp.statusText);
    }
  };

The size constant is 140.  I’m requesting bitmaps by adding ‘.bmp’ to the URL.

This is all very well, and works flawlessly, but what I get back is a color bitmap as a byte array, and what I need to send to the Pebble is a black-and-white bitmap.

Generating the Pebble Bitmap Header (from JavaScript)

I needed to parse the bitmap, and for each pixel convert the Red/Green/Blue components to a black or white pixel, and add the corresponding header.  From JavaScript.

I went searching on the web to see if anyone else had already solved this problem.  No one had, but the good news was that someone had written a library in JavaScript that knew how to parse bitmaps and get the RGB values for each pixel.  The bad news was that they had written the library for Node.js, which uses a special Buffer type for binary data, rather than Typed Arrays that are used in web browsers and the Pebble.

No problem, I thought.  I’ll just write my own Buffer class which wraps a typed array and exposes all the methods that are needed by the library.  That worked, and I was able to parse the bitmap that I received back from the Evernote service, and get the RGB components for each pixel.

All that was left was to generate the appropriate header that the Pebble expects:

  // Buffer is a our own implementation of the bare minumim Node.js Buffer class' 
  // functionality since the bitmap class relies on it
  var buffer = new Buffer(bytes);
  var bitmap = new Bitmap(buffer);
  bitmap.init();

  var width = bitmap.getWidth();
  var height = bitmap.getHeight();

  var data = bitmap.getData(true); // true means it returns 'rgb' objects

  // Calculate the number of bytes per row, one bit per pixel, padded to 4 bytes
  var rowSizePaddedWords = Math.floor((width + 31) / 32);
  var widthBytes = rowSizePaddedWords * 4;

  var flags = 1 << 12; // The version number is at bit 12.  Version is 1
  var result = [];  // Array of bytes that we produce
  pushUInt16(result, widthBytes); // row_size_bytes
  pushUInt16(result, flags); // info_flags
  pushUInt16(result, 0); // bounds.origin.x
  pushUInt16(result, 0); // bounds.origin.y
  pushUInt16(result, width); // bounds.size.w
  pushUInt16(result, height); // bounds.size.h

Generating a Pebble black and white bitmap from a color bitmap (in JavaScript)

Now that I had the header generated (I stole some techniques from a Python tool in the Pebble SDK), I walked through the bitmap’s pixels, converting each pixel to black-and-white:

var currentInt = 0;
for (var i = 0; i < width; i++) {
  var bit = 0;
  var row = data[i];
  for (var b = 0; b < row.length; b++) {
    var rgb = row[b];
    // I'm using the lightness method per
    // http://www.johndcook.com/blog/2009/08/24/algorithms-convert-color-grayscale/
    var isBlack = (Math.max(rgb.r, rgb.g, rgb.b) + Math.min(rgb.r, rgb.g, rgb.b))/ 2 
                   < pbnote.CONSTS.BLACK_CUTOFF;

    // This is the luminosity method, which doesn't seem to give as good results
    //var isBlack = (0.21 * rgb.r + 0.71 * rgb.g + 0.07 * rgb.b) 
    //               < pbnote.CONSTS.BLACK_CUTOFF;
    if (!isBlack) {
      currentInt |= (1 << bit);
    }

    bit += 1;
    if (bit == 32) {
      bit = 0;
      pushUInt32(result, currentInt);
      currentInt = 0;
    }
  }
  if (bit > 0) {
    pushUInt32(result, currentInt);
  }
}

The code on the Pebble side does nothing more than load the bitmap using gbitmap_create_with_data and display it.

Amazingly enough, images rendered on Evernote for Pebble work:

pebble-screenshot_2014-04-17_12-37-10 pebble-screenshot_2014-04-17_12-37-06

I’m using the Evernote Thumbnail API to download a bitmap, converting it to a black-and-white bitmap, adding the appropriate Pebble bitmap header and sending it over to the Pebble for display.

pebble_image_conversion

It might be worth looking at compressing the bitmap prior to sending over to the Pebble, but I’d need to balance reduced battery consumption on the Pebble due to less Bluetooth messages being sent, against the increased battery consumption on the Pebble due to increased CPU usage to decompress the image.  It would also be smart to look at dithering the image to replicate grey-scale.  If anyone has a library to do that in JavaScript, let me know!

About Damian

DamianPortait10PercentDamian Mehers is a Senior Software Engineer at Evernote, currently focused on Evernote and wearable devices.  Damian created  Evernote for the Pebble and the Samsung Galaxy Gear.  He also worked on Evernote Food for Android, and created the initial release of Evernote for Windows Phone.

@Damian Mehers damian@evernote.com

View more stories in 'API'

One comment RSS

  • James Gray

    Here’s some code for dithering in Javascript:

    http://blog.ivank.net/floyd-steinberg-dithering-in-javascript.html

    Also, here’s a Pebble forum thread that may help:

    http://forums.getpebble.com/discussion/13896/watchapp-proof-of-concept-image-viewer