#ember data

sequence diagrams help make sense of Ember Data

Ember Data is a fantastic piece of software that makes it easier to both interact with the server-side while it also enforces a strict and feature-rich model-layer within your application. That is all well and good - once you understand how it works. In this blog post I will explain how Ember Data works by going through how the underlying Identity Map implementation works inside of Ember Data.

There are many ways to visualize Ember Data, but I like to think of Ember Data as an intermediate layer between your models and the backend. This intermediate layer consists of a store acting like a cache for the data you have received from the backend, as well as a data-transport layer that will marshal both the transport as well as the translation between what the backend provides (as JSON) and what your Ember application needs in terms of rich Ember Objects.

Sideloading data to avoid unnecessary requests

As you can see in the example above, we had to make 2 requests to the server, the first to fetch the blog post, and the second to fetch the comments that are related to the post. Allthough this is not that bad in this example, its easy to imaging situations where it will be necessary to reduce the amount of XHR requests made.

Why Client-Generated IDs?

In most client-side frameworks (e.g. Backbone, Angular, Ember) models created in the web browser get temporary IDs (client IDs) and get their permanent ID from the server after a successful save.

For client-heavy applications, generating IDs on the client can simplify interaction with the server. Consider the case where the user creates a post with several attachments. You’ll need to make sure that the post is saved and has its ID resolved before attempting to save the child models in order to preserve this relationship. Take a look at this Ember issue and you’ll start to appreciate the complexity of this use case.

It can be liberating to know the ID of a model before it is saved. While uploading an attachment, you could associate it with other models or send someone a link to where the attachment will be. Users could work offline and save records to the database later without worrying about how to resolve and update IDs.

Because we’re using UUIDs, you could hypothetically find any record in your application given only an ID. I’ve not seen this used for anything practical yet, but there could be interesting possibilities if we kept a global store of IDs and their associated objects on the client.

If you find yourself generating secret IDs for resources (think account activation URLs) you may save yourself some code by using unique IDs. Security-wise, it really can’t hurt.

A Brief History of (My Experience With) Client-Side IDs

I first encountered client-generated IDs when working with Spine. Looking back at older versions you can see that Spine included a UUID generator. Every model was was given a unique ID upon creation.

Spine Author, Alex Maccaw, was critical of Backbone’s temporary client IDs:

… a solution that Backbone uses is generating an internal cid (or client id). You can use this cid temporarily before the server responds with the real identifier… I’m not such a fan of that solution, so I’ve taken a different tack with Spine.

However, in one commit that says it all ("add CIDs (inspired by Backbone)") Alex brought Spine inline with Backbone’s server-generated ID strategy.

Despite this new direction, Spine remained flexible and is a great platform for client-generated IDs. Backbone isn’t as friendly for client-generated IDs, mainly because it uses the existence of a permanent ID to determine whether a record has been persisted. But, as Zed Shaw would say, Programming Motherf*er.

class App.Model extends Backbone.Model initialize: -> unless @id @set('id', utils.guid()) unless @id @_new = true @once 'sync', => @_new = false isNew: -> @_new

Client-Side IDs with Ember Data and PostgreSQL

Ember Data is rarin’ to go with client-generated IDs. Override the dedicated method in the RESTAdapter and you’re golden.

DS.RESTAdapter.reopen generateIdForRecord: -> 'xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx'.replace /[xy]/g, (c) -> r = Math.random()*16|0 v = if c == 'x' then r else (r&0x3|0x8) v.toString(16)

This simple ID generating algorithm has been working from me, but If you’re looking for really robust ID generation you may want to check out Eric Elliot’s CUID library.

The generateIdForRecord function and the inclusion of a specification for client-generated IDs on jsonapi.org gives me confidence that Ember will continue to be a good fit for this approach.

Even though we’re talking about client-generated IDs, there will be times when the server needs to generate IDs of its own. If you’re using Mongo, you’re in luck — it already uses unique IDs. Here’s how you can whip up some matching unique IDs in PostgreSQL and Rails 4 with a couple of migrations.

# Enable UUID generation in PostgreSQL class SetupDatabase < ActiveRecord::Migration def change enable_extension "uuid-ossp" end end

class CreateDocuments < ActiveRecord::Migration def change # Use {id: :uuid} to use UUIDs as the primary key. create_table :areas, id: :uuid do |t| t.string :name # t.belongs_to :category will use an integer. You'll need to setup your # associations like this. t.uuid :category_id end end end

Tradeoffs

I haven’t hit any major pain points with client-generated IDs yet but here are some quick considerations before you dive in:

Model.first and Model.last won’t behave the way you expect in Rails, since the IDs may not be sequential.

There are some rumblings on the Internet about the impact of unique IDs relation join performance, but I couldn’t find anything conclusive.

URLs are uglier.

Requiring users to click a submit or save button in a web application is great for developers but can be a burden on users in some situations. Sometimes it is nice to automatically save the user’s data as they fill out forms.

Automatically saving models in Ember can be a little tricky. A first pass might look like this:

App.Document = DS.Model.extend # some attributes here... saveWhenDirty: (-> @get('store').commit() if @get('isDirty') ).observes('isDirty')

This solution has some significant problems:

Everytime the user types a character an ajax request is sent to the server resulting in a flood of requests.

As the user types, Ember throws errors saying that you can’t modify attributes when a model is “inFlight”.

Debouncing Our Save

Let’s tackle the “too many saves” problem first. Ideally we want to save the user’s data at some logical time. In many applications saving happens when a form is submitted or when a field blurs. However handling all of the cases where we want to save can get a little finicky. On their own, the built-in Ember controls work very reliably for updating model attributes. When I started extending these views to handle events where the user would want to save (e.g. focusOut, change) I ran into some edge cases where the events would not fire. For instance, when using the back button after editing a field I found that the change event would not fire on a field with focus.

In any event we also want to handle the case where a user is editing a lot of text in a textarea and wants to save their work as they go. We want to save when the user types.

This is a good case for a debounce function. Each time a debounce function is called it resets a timer and waits to call another function. So in this case as the user types, the debounce function is called repeatedly but only fires the save function when the user pauses. Here is a simple debounce function based on underscore’s debounce.

App.debounce = (func, wait) -> timeout = null -> context = this args = arguments lastArg = args[args.length - 1] immediate = true if lastArg and lastArg.now later = -> timeout = null func.apply(context, args) clearTimeout(timeout) if immediate func.apply(context, args) else timeout = setTimeout(later, wait)

One bonus feature here is that there is a way to clear out the pending function call and execute immediately by passing {now: true} to our debounced function.

Let’s use that function to implement some saner saving.

App.DocumentController = Ember.ObjectController.extend autoSave: (-> @debouncedSave() ).observes('content.body') debouncedSave: App.debounce (-> @save()), 1000 save: -> @get('store').commit()

So now when the user stops typing for 1 second, the save function will be called.

Give Our Attributes a Buffer

Even though we’re saving less often, we’ll still get errors from Ember if the user stops typing for a second and then immediately starts typing again while the record is saving. To solve this problem we’ll have users edit a buffer that holds the attribute instead of editing it directly on the model. Since writing this code, I’ve seen this pattern called a Buffered Proxy.

#= require ./debounce BUFFER_DELAY = 1000 App.AutoSaving = Ember.Mixin.create # Buffered fields are saved after the set buffer delay. bufferedFields: [] # InstaSave fields save the model as soon as they change. instaSaveFields: [] # Setup buffers to write to instead of directly editing # the model attributes. _buffers: Ember.Map.create() # If we update a field that has been specified as one of the # bufferedFields or instaSaveFields do a safeSet to write # to a buffer if the model isSaving. setUnknownProperty: (key, value) -> @_safeSet(key, value) @_debouncedSave() if @get('bufferedFields').contains(key) @_debouncedSave(now: true) if @get('instaSaveFields').contains(key) # Pull properties from the buffer if they have been set there. unknownProperty: (key) -> if @_buffers.has(key) then @_buffers.get(key) else @_super(key) _safeSet: (key, value) -> if @_isInflight() @get('_buffers').set(key, value) else @get('content').set(key, value) _flushBuffersWhenSavingIsComplete: (-> return if @get('content.isSaving') @_flushBuffers() ).observes('content.isSaving') _flushBuffers: -> @get('_buffers').forEach (key, value) => @get('content').set(key, value) @set('_buffers', Ember.Map.create()) # Write the buffers to the actual content and save or # try saving again later. _safeSave: -> return unless @get('content.store') unless @_isInflight() @_flushBuffers() @get('content.store').commit() else @_debouncedSave() _isInflight: -> @get('content.isSaving') or @get('content.isLoading') _debouncedSave: App.debounce (-> @_safeSave()), BUFFER_DELAY # When the model is about to change out from under the controller we must # immediately save any pending changes and clear out the buffers. _saveNowAndClear: (-> return unless @get('content') @_debouncedSave(now: true) ).observesBefore('content')

This mixin can be applied in a controller to provide a buffer between the Ember fields and the model’s attributes so that users can continue to type while saving. Here is an example of this mixin in use.

App.DocumentController = Ember.ObjectController.extend App.AutoSaving, bufferedFields: ['title', 'body'] instaSaveFields: ['postedAt', 'category']

Good Enough?

In practice this mixin has been working for me, but there is definitely some room for improvement. For one, it would be nice to let developers signal in the templates rather than in the controller whether fields should be buffered or instantly saved. This way there wouldn’t be duplication of field names in the controller and the template. One idea is to give fields special bindings like {{input value=bodyBuffered}} to determine how they should be handled.

Also I still see the possibility of losing data if the model isSaving when the model changes out from under the controller. However, in practice I haven’t been able to produce this behavior.

Have you implemented something to handle autosaving? Have ideas for improvements? Please let me know!

Updates:

Thanks to @workmanw for a technical review of my code snippets.

I found a similar mixin by Tim Evans and loved how his version immediately updated the model attributes if it could.  I worked that feature into my version above.