#mirrorapi

I'm giving a few talks on the Mirror API for Google Glass, and I needed to organize some thoughts on it. I figured that writing a blog post would a good way to do that.

Here is the slide deck for the shared content of my Mirror talks:

Since I don't want to miss anything, I'm going to follow the structure of the slides in this post. What I am not going to do, is be consistent with languages. I'll be bouncing between Java and JavaScript, with bits of Android flavored Java thrown in for good measure. It really just comes down to what I have code for right now in which language. The goal here is to discuss the concepts, and to get a good high-level view.

What is the Mirror API?

What is it?

The Mirror API is one API available for creating Glassware. The main other option is the GDK. Mirror is REST based, mostly done server-side; it pushes content to Glass, and may listen for Glass to respond. The GDK is client-side, and is similar to programming for Android. The general rule of thumb, is that if you can do what you need to do with Mirror, that's the right choice. Mirror is a bit more constrained, and is likely to deliver a good user experience, without a lot of effort. It also allows Glass to manage execution in a battery friendly way.

What does it do?

Mirror handles push messages from servers, and responds as necessary to them. The basic case is that your server pushes a message to Glass, then the message displayed on Glass as a Timeline Card.

How does it work?

If you're coming from the Android world then you might be familiar with GCM, and push messages. Mirror works using GCM, similar to how Android uses GCM.

You can send messages to Glass, and receive messages from Glass. Sending messages is much easier than receiving messages. On the sending side, all you need is a network connection, and the ability to do the OAuth dance. If you want to receive data from Glass, you'll also need some way for Glass to reach you, via a callback URL.

What’s the flow?

Glass registers with Google's servers, and then your server sends requests to Google's servers, and Google's servers communicate with Glass. You can send messages to Google's servers, and you can receive messages back from Glass via callback URLs that you provide. With that, you can get a back and forth going.

SO MANY QUESTIONS?!?!?!

I know, there's a lot here. We'll try to step through as much as possible in this post.

Setting up

Before you do anything else, you'll need to head over to the Google Developer Console, and register your app. Create a new app, turn on 'Mirror API', and turn off everything else. Then, go into the Credentials section, so that you can get your Client Id and Client Secret. This is outlined pretty clearly in the Glass Quick Start documentation.

OAuth

To communicate with Glass, you'll first need to do do the OAuth dance. The user visits your website, and clicks a link that will ask them to authenticate with one of their Google accounts. It then asks for permission to update their Mirror Timeline. After that's done, you'll receive a request at the callback URL, with a code. The code needs to be exchanged for a token. Once you have the token, you can begin making authenticated requests. The token will expire, so you'll need to be able to re-request it when it does.

Further reading in the Quick Start.

Timeline

The timeline is the collection of Cards that you have on Glass, starting from the latest, and going back into the past. It also includes pinned cards, and things happening 'in the future', provided by Google Now.

Here's a video explanation from the Glass Team:

Why a Timeline?

Well, you'd have to ask the Glass team, why exactly a Timeline, but I could venture a guess. One of Glass's issues is that it is more difficult to navigate with. You can't dive into apps like you can on your phone, at least not quickly and easily. There's no 'Recent Apps' section, no home screen and no launcher (well, except from the voice commands).

Launching apps with voice commands allows you to do things, as opposed to going to an app and poking around. For example, saying OK Glass, start a bike ride launches Strava into a mode where it immediately begins tracking me with GPS and giving me feedback on that tracking. It does not let me look at my history there.

Glass tends to operate on push rather than pull. Glassware surfaces information to you, as it becomes relevant. This ability allows Glass to become incredibly contextual, though we have only seen bits of this so far with Google Now. Imagine a contextual Glassware that keeps a todo list of all of your tasks, both work and home. It could surface a Card when you get to your office with the highest priority work tasks that you have for that day. On the weekend, after you're awake, it could tell you that you need to get your car into the shop to get your smog check.

A typical category of Glassware today is news, which all work well with a push approach. Now, this is not terribly useful on Glass, as it's difficult to consume, but there are some good ones that have figured out how to deliver the content in a way that is pleasant to consume on Glass (Unamo, and CNN), or to allow you to save it off to Pocket for picking up later (Winkfeed).

The timeline is semi-ephemeral. Your cards will stick around for 7 days, and will be lost after that. Before you get all clingy and nostalgic, consider that that information is 7 days old, and will not only be stale, but difficult to even scroll to at that point. I have rarely gone back more than a day in my history to dig for something that I saw, and that was usually for a photo. Cards are usually a view into a service, so the information is rarely lost to the ether. Instead of scrolling for a solid minute trying to find something, you can pull out your phone and go to the app on your phone. Like G+ Photos, Gmail or Hangouts. Apps like Strava don't even put your recent rides in your timeline, you have to either go to the app or their website for history.

What are Cards?

Cards are the individual items in the timeline. Messages that you send from your server to Glass users typically end up as timeline cards on the user's Glass. There are a number of basic actions, as well as custom actions, that users can take on cards, depending on what the card supports. How this works will become apparent once we look at the code.

How do I insert Cards into the Timeline?

Using the API, of course! Let's look at how to do this in a couple of languages.

Java:

TimelineItem timelineItem = new TimelineItem(); timelineItem.setText("Hello World"); Mirror service = new Mirror.Builder(new NetHttpTransport(), new JsonFactory(), null).setApplicationName(appName).build(); service.timeline().insert(timelineItem).setOauthToken(token).execute();

JavaScript:

client .mirror.timeline.insert( { "text": "Hello world", "menuItems": [ {"action": "DELETE"} ] } ) .withAuthClient(oauth2Client) .execute(function (err, data) { if (!!err) errorCallback(err); else successCallback(data); });

Further reading in the Developer Guides, and Reference.

Locations

Location with Glass can mean one of two things, either you're pushing a location to Glass, so that the user can use it to navigate somewhere. Or, you're requesting the user's location from Glass. The Field Trip Glassware subscribes to the user's location so that it can let the user know about interesting things in the area.

Sending Locations to Glass

Sending locations is very simple. Basically, you send a timeline card with a location parameter, and add the NAVIGATION menu item.

JavaScript:

client .mirror.timeline.insert( { "text": "Let's meet at the Hacker Dojo!", "location": { "kind": "mirror#location", "latitude": 37.4028344, "longitude": -122.0496017, "displayName": "Hacker Dojo", "address": "599 Fairchild Dr, Mountain View, CA" }, "menuItems": [ {"action":"NAVIGATE"}, {"action": "REPLY"}, {"action": "DELETE"} ] } ) .withAuthClient(oauth2Client) .execute(function (err, data) { if (!!err) errorCallback(err); else successCallback(data); });

I'm using this right now for something very simple. In a Mirror/Android app that I'm working on, I can share locations from Maps to Glass, and then use Glass for navigation. Nothing is more annoying than having Glass on and then using your phone for navigation because the street is impossible to pronounce correctly.

Getting Locations from Glass

This is a little more tricky. First, you need to add a permission to your scope, https://www.googleapis.com/auth/glass.location. Getting locations back from Glass is going to require you to have a real server that is publicly accessible. It should also support SSL requests if you're in production.

Here's some code from the Java Quick Start that deals with handling location updates:

LOG.info("Notification of updated location"); Location location = MirrorClient.getMirror(credential).locations().get(notification.getItemId()).execute(); LOG.info("New location is " + location.getLatitude() + ", " + location.getLongitude()); MirrorClient.insertTimelineItem( credential, new TimelineItem().setText("Java Quick Start says you are now at " + location.getLatitude() + " by " + location.getLongitude()) .setNotification(new NotificationConfig().setLevel("DEFAULT")) .setLocation(location).setMenuItems(Lists.newArrayList(new MenuItem().setAction("NAVIGATE"))));

This will really help enable contextual Glassware. You could imagine Starbucks Glassware that offers you a discounted coffee when you get near a Starbucks. Or a Dumb Starbucks Glassware that does the same thing, but claims to be an art installation.

Further reading in the Developer Guides, and Reference.

Menus

Menus are the actions that a user can take on a card. They make cards interactive. The most basic one is DELETE, which, obviously, allows the user to delete a card. Other useful actions are REPLY, READ_ALOUD, and NAVIGATE.

You can see menu items added to cards in previous examples. I will repeat the JavaScript navigation example here:

JavaScript:

client .mirror.timeline.insert( { "text": "Let's meet at the Hacker Dojo!", "location": { "kind": "mirror#location", "latitude": 37.4028344, "longitude": -122.0496017, "displayName": "Hacker Dojo", "address": "599 Fairchild Dr, Mountain View, CA" }, "menuItems": [ {"action":"NAVIGATE"}, {"action": "REPLY"}, {"action": "DELETE"} ] }) .withAuthClient(oauth2Client) .execute(function (err, data) { if (!!err) errorCallback(err); else successCallback(data); });

The Card generated from the example should allow you to get directions to the Hacker Dojo. You can also add custom menus, like Winkfeed's incredibly useful Save to Pocket menu item.

Further reading in the Developer Guides, and Reference.

Subscriptions

Subscriptions allow you to get data back from the user. This can happen automatically, if you're subscribed to the user's locations, or when the user takes some action on a card that is going to interact with your service. E.g., you can subscribe to INSERT, UPDATE, and DELETE notifications. You will also get notifications if you support the user replying to a card with REPLY.

In Java:

// Subscribe to timeline updates MirrorClient.insertSubscription(credential, WebUtil.buildUrl(req, "/notify"), userId, "timeline");

Subscriptions require a server, and cannot be done with localhost. However, tools like ngrok may be used to get a publicly routable address for your localhost. I have heard that ngrok does not play nice with Java, but you might be able to find another tool that does.

Among other things, you can learn from these sorts of subscriptions. You can learn which types of Cards get deleted the most, for example. Subscriptions can also listen for custom menus, voice commands and locations. With Evernote, you can say, take a note and it will send the text of your note to the Evernote Glassware.

Further reading in the Developer Guides, and Reference.

Contacts

Another important type of subscription is subscribing for SHARE notifications. The SHARE action also involves Contacts.

Contacts are endpoints for content. They can be Glassware or people, though if they're people, they are probably still really some sort of Glassware that maps to people. Sharing to Contacts allow your Glassware to receive content from other Glassware. You can insert a Contact into a user's time like so (in JavaScript):

client .mirror.contacts.insert( { "id": "emil10001", "displayName": "emil10001", "iconUrl": "https://secure.gravatar.com/avatar/bc6e3312f288a4d00ba25500a2c8f6d9.png", "priority": 7, "acceptCommands": [ {"type": "POST_AN_UPDATE"}, {"type": "TAKE_A_NOTE"} ] }) .withAuthClient(oauth2Client) .execute(function (err, data) { if (!!err) errorCallback(err); else successCallback(data); });

It's obvious what you want people Contacts for, but what about Glassware contacts? Well, imagine sharing an email with Evernote, or a news item with Pocket. Now, consider that as the provider of the email or news item, you don't need to provide the Evernote or Pocket integration yourself, you just need to allow sharing of that content. It's a very powerful idea, analogous to Android's powerful sharing functionality.

Further reading in the Developer Guides, and Reference.

Code Samples

I have three sample projects that I have been pulling code from for this post.

A slightly modified Java Quick Start provided by the Glass Team

My Android Mirror demo

Google Glass UI is based on the concept of "timeline" and "cards". See, for example, my earlier posts:

Basics of the Google Mirror API (Developer Preview)

Google Glass User Interface

Timeline-Based UI Paradigm of Google Glass

Timelines and/or Cards are not necessarily unique to Google Glass. The Timeline metaphor is used by many apps and frameworks. The card-based UI is also rather common. The concept of "card" in computer user interface dates back to as early as the time of Mac Hypercard, or even earlier.

I've been thinking about building some apps for Google Glass using GDK (or, Android SDK, at this point). Considering that the GDK (or, its preview version) will be released early next week, it's probably too ambitious to start creating a brand new app. Rather, It'll be easier to "port" existing Android apps to Google Glass. So, I looked at some of my old Android apps (some of which are only partially done). One of the very first apps I developed on Android was a Rock Paper Scissors game. I cannot remember exactly when, but it was probably 4~5 years ago. My first Android phone was Motorola Droid, and the app was developed using Droid (whatever version of Android it was running on, probably, Froyo?)

I recently accepted an invite to purchase an Explorer Edition of Google Glass. Well, actually, the company I work for accepted (man, I love working for TiKL, Inc.), and I've been the one working on everything Glass at our office. (In case you're wondering, a write-up on my thoughts about Glass so far is in the works.) So far, my efforts have been focused on building apks to run on Glass, which is incredibly easy if you're and Android hacker like myself. What isn't so easy for someone mainly focused on Android is trying to figure out the Mirror API.

The Glass team has put out a number of quick-start demos for working with the Mirror API, with a focus on getting over the OAuth2 hump. They demonstrate this in a number of languages, but they haven't put out anything for node.js, which is what I use whenever I need a quick demo server for something. What Google has provided, is a github library that provides some simple mechanisms to interact with all of their services. It's pretty awesome, and it allows you to pretty easily inspect the JSON data that is returned as the client object, and look at what's available.

I figured out how to do the Mirror stuff by looking at the Google+ example in the googleapis github, implementing that first, so that I could get the authentication working. Then, making a couple of educated guesses about what I might need to change to get the Mirror API endpoints, along with looking at the Mirror documentation for Java and Python. Once I had G+ authentication up, moving it over to getting authenticated with the Mirror API was really easy, and getting valid requests written was not a big leap from there.

I still do not have much done, my main goal was to simply get node.js talking to Glass, which I did. Speaking of, here's the github repo for my demo. Again, all this does right now is allows a user to authenticate, pushes the message 'hello world' to Glass, and requests the user's timeline for the registered app.

Instructions

Follow the instructions in the Mirror API documentation for registering a new app. It's basically the same flow for starting any new project using a Google API. Just follow the instructions, it's easy. Once you've registered, pull down my repository and run npm install. This will install the googleapis npm package, as well as express.js if you need it.

Next, copy sample_config.json to config.json, and replace the values with those from the Google Developer's console.

Run node app.js. Finally, visit the app in your browser, and authenticate with Google. You should get a push to your Glass with the text 'Hello world'.

Digging in a bit

Now that you've seen this working, let's look at some code. This is simply chunks from the app.js file, so if you want to just look at the whole thing at once, here it is.

First up, we've got some things to require, the googleapis, and the related oauth stuff.

We should look at the routes involved here first, before we look at anything specific to the API. Without the routes, you don't really know what's being called, and where the most important parameters come from, so here are the routes:

As you can see, there are only two routes, the root, and the oauth2 callback routes. We start by looking to see if we already have a token, if they don't, we send them to Google to go request one. Otherwise, we use the token we have. If they didn't have a token, they'll return to us on the route oauth2callback (unless you change that route on your config), and we take the code provided in the request data to parse out their token.

**WARNING**: this is not intended for a multi-user case, just for you! Don't use this unmodified!

We'll provide a couple of generic callbacks, for simplicity. Whenever you're unsure of what to do next, or aren't sure if something's working, dump whatever data that you've got.

Alright, first thing's first, we need to get ourselves a token. This method gets called when the user returns from giving us permission to connect to their account, and will use the provided oauth2Client to pull the token out of the code that is returned to us. We then store that token in the oauth2Client.

Once we have a token, we can do stuff with it. First thing to do is request a client that can make requests on the Mirror API, and in the callback, we can use the returned client to make those requests.

The following are two simple requests that you can make once you're authenticated and get your client object:

I probably don't really need to explain what's going on there, it's incredibly simple. But, in case you like reading things, the first one inserts the text 'hello world' into your timeline on Glass. If you're using this with the account that you've attached Glass to, then that text will show up in your eyeball. It's neat. The second request simply shows the timeline items that your app has generated.

Notes

Currently, this is incredibly simple, and as such is completely lacking in style, structure, and functionality. This is intended to demonstrate just two things, first how to authenticate with Google for the Mirror API with OAuth2, using node.js. Second, it begins to show how you can use node.js to make requests with the Mirror API. Think of it as the quick-start for node.js.

Go get the source