Tales of Timehop

Writing a Pebble watchface is fun! Its a chance to write C code that gets heavily checked by pebble tools and the C compiler.

In case you’d like to read-and-follow-along, the source code for this blog post can be found here:

https://github.com/timehop/pebble-time-watch-face

In order to make a functioning watch application, the first order of business is to setup a subscription to pebble’s tick_timer_service.  As such, the watchface app was setup to subscribe to the MINUTE_UNIT as a part of the design of the look n’ feel of the application and to save on battery life. In the tick_timer_service_subscribe callback, localtime is copied into date, time buffers and those buffers are displayed in text layers. Here’s a screen shot of the watchface after the addition of the buffers and text layers:

Here’s the code that accomplishes this task:

//called in the the init() method

tick_timer_service_subscribe(MINUTE_UNIT, tick_handler);

//performed in the update_datetime() method

time_t ltime = time(NULL);

struct tm *tick_time = localtime(&ltime);

strftime(time_buffer, sizeof("00:00"), "%I:%M", tick_time);

strftime(date_buffer, sizeof("Sun, Oct 25"), "%a, %b %e", tick_time); // Display buffers on the TextLayers text_layer_set_text(s_date_layer, date_buffer); text_layer_set_text(s_time_layer, time_buffer);

Next up: drawing a white circle as a background behind the date, time. This is accomplished by setting an update procedure on a layer and using graphics_fill_circle to draw the circle. 

Here’s a screen shot of the watchface after the addition of circle background:

Here’s the code that draws a white circle:

layer_set_update_proc(s_date_time_layer, draw_circle_layer_update_proc);

static void draw_circle_layer_update_proc(Layer *this_layer, GContext *ctx) {

  GPoint center = GPoint(72, 88);

  graphics_context_set_fill_color(ctx, GColorWhite);

  graphics_fill_circle(ctx, center, 64);

}

Finally: added a bitmap of abe, of course! The key here is to set the bitmap layer compositing mode to GCompOpSet to enable the use of a transparent background as well as adding the bitmap layer as a child of s_date_time_layer that contains the drawn circle. Here’s the code that adds a transparent abe at the bottom of the screen:

s_image_bitmap_layer = bitmap_layer_create(GRect(48, 123, 53, 45));

bitmap_layer_set_compositing_mode(s_image_bitmap_layer, GCompOpSet);

s_image_bitmap = gbitmap_create_with_resource(RESOURCE_ID_ABE);

bitmap_layer_set_bitmap(s_image_bitmap_layer,s_image_bitmap);

layer_add_child(s_date_time_layer, bitmap_layer_get_layer(s_image_bitmap_layer));

Note:  abe.png has been added to resources/images directory.

2,675,812,470

That’s the number of rows in our largest database table. Or it was, one year ago today. Since then, it’s grown to over 60 billion rows. On average, that’s roughly 160mm inserts per day.

When I think about it, that seems like quite a lot. But because the database we use is Amazon’s DynamoDB, I rarely have to think about it at all. In fact, when Timehop’s growth spiked from 5,000 users/day to more than 55,000 users/day in a span of eight weeks, it was DynamoDB that saved our butts.

What is DynamoDB?

I don’t intend to explain the ins and outs of what DynamoDB is or what it’s intended for, there’s plenty of documentation online for that. But I will mention the most salient point: That DynamoDB is consistent, fast, andscales without limit. And when I say “scales without limit,” I literally mean there is no theoretical boundary to be made aware of. That isn’t to say that all of DynamoDB’s features behave identically at every scale, but in terms of consistency and speed it absolutely does. More on that below. First, some history.

The Ex I Worry About Seeing in My Timehop is Mongo

Timehop currently uses DynamoDB as it’s primary data store. Every historical tweet, photo, checkin, and status is persisted to DynamoDB as time-series data. It wasn’t always this way. Back in ye olden days, we used a combination of Postgres and Mongo. So why the switch?

Listen, Mongo is great. I mean, I’m sure it’s pretty good. I mean, there’s probably people out there who are pretty good with it. I mean, I don’t think my soul was prepared for using Mongo at scale.

When we were small, Mongo was a joy to use. We didn’t have to think too hard about the documents we were storing and adding an index wasn’t very painful. Once we hit the 2TB mark, however, we started to see severe and unpredictable spikes in query latency. Now, this might have been due to our hosting provider and had nothing to do with Mongo itself (We were using a managed service through Heroku at the time). Or, if we rolled our own we might have solved things with a sharded solution. But to be honest, none of us at Timehop are DBAs and a hosted database has always been a more attractive option (As long as it didn’t break the bank!). So we started to look at alternatives. Our friends at GroupMe had some nice things to say about using DynamoDB and it seemed pretty cost-effective too, so we gave it a go.

Early Mistakes with DynamoDB

As is often the case with new technologies, DynamoDB first seemed unwieldy. The biggest hurdle for us right out of the gate was the lack of a mature client library written in Go, our primary language (Things have changed since then). But once we nailed down the subset of API queries we actually needed to understand, things started to move along. That’s when we started making real mistakes.

Mistake #1: Disrespecting Throughput Errors

This one should have been obvious as the error handling docs are quite clear. DynamoDB has two main types of http errors: retryable and non-retryable. The most common type of retryable error is a throughput exception. The way dynamo’s pricing model works is that you pay for query throughput, where reads and writes are configured independently. And the way Amazon enforces throughput capacity is to throttle any queries that bust it. It’s up to the client to handle the errors and retry requests with an exponential backoff. Our early implementations assumed this would be a rare occurrence, when in reality, you will likely incur some amount of throttling at any level of usage. Make sure you handle this frequent type of error.

Mistake #2: Not Understanding How Partitions Affect Throughput

All DynamoDB tables are partitioned under the covers and each partition gets a sliver of your provisioned (purchased) capacity. The equation is simple:

Total Provisioned Throughput / Partitions = Throughput Per Partition

What this means is that if your query patterns are not well distributed across hash keys, you may only achieve a fraction of your provisioned throughput. Not understanding this point can lead to serious headaches.

It’s also important to say that the number of partitions that exist are not directly exposed to you (However, there is now a helpful guide to help you estimate the number of partitions you are accruing). A single table may have 1 partition or 1000. That means it’s difficult to predict exactly how hot a key can become. And the larger your table grows the more partitions that will be allocated to it. Which means you may not even notice hot key issues until you are at scale. The best thing you can do is fully understand how to design and use a well distributed hash key.

Mistake #3: Poorly Designed Hash Keys

DynamoDB tables can be configured with just a hash key, or with a composite hash and range key. Timehop’s data is in time-series, so for us a range key is a necessity. But because hash keys always map to a single virtual node in a partition, a large set of range keys per hash key can lead to hot-key problems. I dug into my old emails and found a few write capacity graphs that illustrate this point:

^^ March 2015 

^^ March 2014

The red lines here indicate provisioned throughput, ie: the most you could ever use. The blue lines represent actual throughput achieved.

The top graph represents our current schema, which has a well-designed hash key based on a combination of user id and calendar date.

The bottom graph shows our first stab last March, where our hash key was just the user id. Notice the difference? Yeah, it was bad. It took us about 3 attempts at loading and destroying entire tables before we got it right.

Advanced Lessons

Once we stopped making the above mistakes we found further ways to optimize our DynamoDB usage patterns and reduce costs. These points are actually covered in the developer guidelines, but we had to make our own mistakes to reach the same conclusions.

Lesson #1: Partition Your Hash Keys

Every hash key will consistently hit a single virtual node in DynamoDB. That means that every range key associated with a hash key also hits that same node. If a Timehop user has 5000 items of content (a rough average) and most of that content is written at around the same time, that can result in a very hot hash key. Anything you can do to split the data across multiple hash keys will improve this. The pros and cons of doing so are:

Pro: Hash key distribution increases linearly with each hash key partition; Throughput errors decrease.

Con: The number of network queries required to write the same amount of data increases linearly with each hash key partition; the likelihood of i/o errors and poor network latency increases.

For Timehop our hash key partition strategy was a natural function of our use case: We partition all user data by date. This gives us a significant increase in hash key distribution. As an example, a user with 5000 items of content across 7 years of social media activity sees about a 2.5k times increase in distribution:

Without partitioning (ie: hash key = “<user_id>”):

5000 items per hash key on average

With partitioning (ie: hash key = “<date>:<user_id>”):

5000 / 365 days / 7 years = about 2 items per hash key on average

Lesson #2: Temporally Distribute Your Write Activity

As I stated earlier, a table that uses a range key can expect at least somethrottling to occur. How many errors you experience is directly tied to how well distributed your hash keys are. However, if you really want to maximize your throughput you should also consider temporally distributing the writes you make against a single hash key.

Our writes happen asynchronously when a user signs up and connects their social media accounts. At any given moment the write requests that occur are heavily skewed towards the subset of hash keys that represent the most recent new user. In order to encourage well-distributed writes, we built a simple write buffer that sits in front of DynamoDB (essentially a managed Redis set). A separate process drains random elements from the buffer and issues batch write requests to DynamoDB. The more users that sign up together, the more data that ends up in the buffer. The key benefit of this architecture pattern is the more data that’s in the buffer, the more randomization there is when draining it and therefore more distributed the hash keys are in our batch write requests. So increasing scale actually improvesour write performance. Yay!

Scaling Without Limit

So now that we’ve got our usage patterns stabilized, what’s the real benefit? The most explicit win we saw from using DynamoDB was during our most significant period of user growth, which just happened to coincide with our final draft of a DynamoDB strategy.

During the 8 week period between March and May of 2014 our user growth started to spike upwards from roughly 5k signups/day to more than 55k signups/day. As you can imagine we were hyper-focused on our infrastructure during this time.

There were many many fires. Each dip in the graph probably corresponds to some outage or backend service falling over due to load. But what the dips definitelydo not represent, is an outage or performance problem with DynamoDB. In fact, for the last year, no matter how large our table grew or how many rows we’ve written to it (currently in the order of hundreds of millions per day) the query latency has always been single digit milliseconds, usually 4–6.

I can’t stress enough how nice it was not to have to think about DynamoDB during this time (aside from a few throughput increases here and there). This was a new experience for Timehop Engineering as in the past the database was always the choke point during periods of growth. The stability and performance of DynamoDB allowed us to focus our efforts on improving the rest of our infrastructure and gave us confidence that we can handle the next large spike in user signups.

AWS describes DynamoDB as a providing “single-digit millisecond latency at any scale” (emphasis mine). We’ve asked the DynamoDB team what the theoretical limits are and the answer is, barring the physical limitations of allocating enough servers, there are none. In evidence of this, our table is quite large now, about 100TB, and the performance is the same as on day one.

Thanks for reading! I hope you’ve gleaned something useful from this post. We will continue to optimize our DynamoDB usage patterns where we find opportunity and broadcast the lessons we learn along the way.

Welcome to the 9th post of Tales of Timehop. This post is operations focused and touches on how a strong philosophy can help scale a startup. This post was written by Timehop’s Head of Talent and Community Development, Tess Manning.

Here at Timehop, enabling the talent on our team to be autonomous and aligned is crucial for our success and ultimately allows us to go above and beyond as a group. As simple as it sounds, we strive to maintain this dynamic by working in tandem with our core values. As mentioned in another post by Jake, building frameworks for decision making is a critical part of scaling an organization. Core values act as our philosophical code of conduct. They help us navigate new situations, unpredictable problems, and tough decisions as our team grows.

At the end of last year, Benny and I embarked on a mission to suss out what we really view as existing core values at Timehop. Here’s where we landed.

TIMEHOP CORE VALUES

1. Delight Users (learn what people like and what makes their day)

Example: We know that our user community (and our team!) struggles with “exes and feels” in Timehop on the reg, so for Valentine’s Day our Product team provided users the choice to avoid #Valentimehop if they weren’t prepared for the lovey dovey overload.

2. Ship early, ship often (not trying is worse than failing)

Example: We had a hunch recently that our nostalgia obsessed community will love crazy vintage filters on their old school pictures. Our design and engineer teams immediately launched a v1 of Polaroid and Gameboy filters to a small cohort of users. Maybe we’ll keep the feature, maybe we won’t! Depends on if our users validate that hunch. Either way we’ll learn.

3. Trust trumps rules (we don’t have a lot of rules because we trust our individuals to make decisions in the interest of the team).

Example: Sounds simple, but we don’t have a vacation policy. We trust (and encourage) you to take what you need, and when you do, you are a responsible colleague that communicates and makes plans with the team.

4. Always strive for better (have a higher expectation of yourself than your teammates do)

Example: Individual heart2hearts and team level retros are scheduled times where we can all sit down and talk about what is making us happy, sad, and confused. These are great for exposing where we can make improvements on an individual, team, and company level. You can’t take action on what you don’t know!

5. Be respectful (seek to understand before seeking to be understood)

Example: During a distributed work experiment in February we realized that the tone of text in Slack or email can be easily misinterpreted and rustle emotions. During our retro meeting we discussed how we should assume positive intent in these situations, but also feel comfortable reminding each other of this value.

These values all came from things that already existed inside Timehop’s people, product, and process. We simply just did our best to put words around them!

HOW THEY HELP US SCALE

1. Feedback. Timehoppers can use Core Values to respectfully call out a colleague if someone is violating our core values and it upsets them. This builds trust and it allows us to speak more freely and fearlessly with one another.

2. Our weekly All Hands has a “Shout Outs” slide. If a Timehopper is living and breathing our core values we give the company an opportunity to high five them publicly.

3. We use questions about our core values in interviews and performance reviews. (Something like: “What’s the last mistake you made, and how’d you improve from it?”, or “Do you have any examples of you bettering yourself, the team, the product?” ). These help us assess emotional intelligence and self-awareness.

4. Tough Choices. Having a code to fall back on allows individuals to think through difficult decisions (Do I share this information or not? How should we prioritize this initiative?).

AT THE END OF THE DAY

Companies are just a group of people, and a group of people is just a community. Strong communities that survive adversity and advance together are tight knit, know what they stand for, and act in unison because of it. This is what Timehop stands for, for now, until we evolve to make them even better (see #4)!

Welcome to the 8th post of Tales of Timehop. This is an engineering focused post that discusses why Timehop chose Go. This post was written by Timehop's CTO, Benny Wong. 

Here at Timehop, we’ve been running Go in production for over a year and a half and have fallen in love with it. We’ve gone from having our median response time of 700ms with our Rails app to a 95th percentile response time of 70ms. We do this while serving over six million unique users every single day.

A couple of weeks ago, I sat down with my friends Luke & Kayvon to talk about how and why we use Go. They wanted some perspective on how we’ve used it and whether it makes sense for them to use it at Universe, their new startup.

I thought I’d share the highlights of our conversation in hopes that it would be useful to other engineering teams considering Go.

What prompted you to first consider Go?

We originally built Timehop as, like many startups, a Rails app. Then we started growing extremely quickly and what we built in Rails couldn’t keep up.

A lot of what we do lends itself to being parallelized. Whenever a user opens the app, we gather data from all the years they have content. The queries we issue to our database are independent of each other, which makes them very easily parallelized. We tried making it work with Ruby, but ultimately, Ruby doesn’t have true multithreading and the abstractions to work around that limitation felt brittle.

When we set out to explore new languages we had three things at the top of our wish list: easy concurrency, true parallelism, and performance.

Why did Go win compared to the other languages you considered?

We looked at a few other languages (Node.js primarily), but ultimately Go had sold us on three major things:

Performance — Go code is compiled down to machine code. There is no VM or interpreter that adds overhead.

Static typing — Turns out computers are way better than humans at a whole class of errors when you know what type a variable is. Who knew?

Sane, readable concurrency — Goroutines and channels make concurrent code easy to read and reason about. Those constructs also make safe concurrent code without needing explicit locks. Also, no callback spaghetti.

Those were the initial points that really sold us on it. As time went on, we added to the list:

Dead-simple deployment — it compiles down to a single binary with all of it’s dependencies. More on that later.

Amazing toolchain — Go also includes tons of amazing tools, not the least of which is the code formatter`go fmt`. It has eliminated code formatting debates, and with it, an untold amount of wasted developer-hours.

Extremely robust standard library — we’ve found that we haven’t needed a ton of third party libraries. The libraries provided by the language are extremely well thought out.

Go checked off all of the boxes for our requirements — and then some. The additional benefits made it a clear winner in our book.

Were there any surprises — positive or negative — once you started using Go?

We were all hesitant about losing productivity in the switch. Ruby is a very expressive programming language which allowed us to write a lot of code quickly. We were concerned we’d lose that switching to a type safe, compiled language.

We didn’t need to be concerned. Very quickly after the switch we found ourselves writing Go code just as fast and a lot safer. Go’s type safety prevented a lot of the fat fingered mistakes that are all too common in Ruby.

Compiling our code also turned out not to be an issue — our largest Go app compiles in ~2.5 seconds at worst.

How did the team ramp up? What’s your advice to help teams go through this process smoothly?

I wrote a post when I was learning the language and it’s been the starting point for the team.

TL;DR: Tour of Go, Effective Go, and reading the standard library.

What are Go’s weaknesses?

Dependency management.

Go has a convenient import scheme where you include packages by it’s location, ie.

import “github.com/timehop/golog/log”

You can pull down the code with a simple “go get” command in your terminal. While convenient, it can also be deployment headache because it pulls the HEAD from the repo. This can get in the way of shipping a feature because someone else’s code changed locations or had breaking API changes.

The most popular dependency management tool right now is Godep. At a high level, Godep pulls all of your dependencies and then vendors them into your project — essentially copying your code into your project so that you always have a working copy of your dependencies.

It is worth mentioning that this weakness is by design. The creators of Go have specifically avoided building a dependency system because they wouldn’t know what a general solution would look like to others.

What Go libraries are critical to deployment on the modern web?

When we first started writing Go, we googled around for “Rails for Go.” We quickly realized that was overkill for building out JSON API.

All of our web services simply use the standard net/http library and Gorillamux for routing. Others seem to do the same.

What are options for hosting? How does deployment work?

We started on Heroku because our Rails app were hosted there as well. It was simple to deploy using the Go buildpack.

We eventually migrated to EC2 and deployment was just as easy. Go apps compile down to a single binary, so it can be as simple as scp-ing the binary to a box and running it.

Our current process is:

Push code to Github

Run tests via Travis

On success, build the binaries, tar them, and upload to S3

The app servers simply have to pull down the tar, unpack it, and run it (we do this via Chef)

We haven’t needed to use this, but it also makes compiling binaries for different architectures as easy as:

$ GOOS=linux GOARCH=arm go build main.go

That means you could easily build your app for many types of operating systems and embedded devices.

Is the language suited for building APIs?

Yes. The Go encoding libraries make writing APIs stupidly simple. Take, for example, a User struct (which is similar to a class, but distinctly not):

type User struct {  FirstName string `json:"first_name"`  LastName string `json:"last_name"`  Password string `json:"-"` }

Those tags after those fields define how they’re going to be serialized out to JSON. No need for custom serialization functions.

To output an instance of a User, it is simply:

u := User{FirstName: "Abe", LastName: "Dino", Password: "p4ssw0rd"} jsonBytes, _ := json.Marshal(u)

How does the language deal with polymorphism and modularization?

Go isn’t an object-oriented language — it doesn’t have any type hierarchy. While it initially took some getting used to, it quickly became a positive. Because of the lack of inheritance, it naturally encourages composition.

It also has really nice interface system. Unlike other languages where the class has to explicitly declare that it is implementing an interface, in Go, a struct simply is. It’s all very philosophical. We’ve found it to be one of the most powerful features of the language. You can read more about why here.

How important is Google’s involvement in the project?

It’s huge — having Google release and continue to invest in the language is great for everyone. It’s clear they’re using it internally and that can only mean continued improvement to the language, tools, and ecosystem.

In addition to the backing of Google, the community has grown by an insane amount since we started. It’s been amazing to see the community come alive, which has been a boon the amount of open source code and content now available in go.

Hopefully, this was helpful for teams trying to decide whether Go is worth a look. There are lots of questions whenever you consider a new language, and hopefully we’ve answered some of those here. We love writing Go and have had a ton of success doing so.

Welcome to the 7th post from Tales of Timehop! This post is focused on iOS engineering and the learning curve behind understanding ReactiveCocoa and MVVM. This post was written by Evan Coleman.

When I walked into my first day of work at Timehop this past summer, I wasn’t sure what to expect. I had just graduated from college and had spent my spare time during the school year doing contract work for them. This mainly consisted of fixing bugs and adding small features.

On that first day, I sat down with Chris and he briefed me on what was going on in the iOS world at Timehop. He told me about TimehopThree, which was meant to be a complete rewrite of the iOS app that would take full advantage of ReactiveCocoa and MVVM. I had heard of ReactiveCocoa before, but knew nothing about it. MVVM, on the other hand, I had never heard anything about that. He gave me a quick overview of what MVVM was and how it made more sense for our app than MVC. I was convinced pretty quickly on that front.

MVC never really seemed to fit in in the world of Cocoa to me having seen many views and controllers mashed into giant UIViewController subclasses that quickly grew out of control. A lot of UIViewController code was typically dedicated to formatting model data for the views. In MVVM, this was abstracted out into a concept called view models. While this does mean more files in your project, it also means much leaner files; combined with ReactiveCocoa, it also means that the majority of your code will be in the -init or -viewDidLoad methods. A good example of this would be a view that displays an image with a loading indicator.

ReactiveCocoa is largely made up of a construct called signals. A signal is essentially a pipe that sends values. Looking at the above code you’ll see a lot of RAC() and RACObserve() -- macros defined by ReactiveCocoa. RAC() will continually set some variable to whatever a signal sends, whereas RACObserve() is essentially a wrapper around key-value observing (KVO) that returns a signal that sends whenever the value of some property changes. Back to the example, we’re binding the image view’s image property to the view model’s image. MVVM allows us to abstract out the image fetching logic into the view model, which in MVC would typically be contained to the view controller. The abstraction of this logic, along with the binding, allows the view to be really dumb in that it knows nothing about where the image came from, just that it needs to be displayed. Next, we’re binding the progress indicator’s progress property to the view model’s progress. We don’t want to show the progress indicator when there’s an image, so we can observe the view model’s shouldShowProgress property which maps whether the image is nil or not to a boolean. A similar method is used to display an error view when an error occurs. The view model uses the -distinctUntilChanged method to withhold sending values until the value actually changes, this ensures that we only create a new ErrorViewModel when the error actually changes. The -ignore: method allows us to prevent values from continuing down the signal chain, in this case, we don’t want to create an ErrorViewModel without an error.

The image view example is a good example of how ReactiveCocoa and MVVM excel on a small scale, but as I continued using it on larger and larger scales, the benefits started to become even more apparent. At first, none of it made any sense to me. I spent the next week pouring over the ReactiveCocoa docs, but no matter how much I read, signals, streams and commands were all just words to me. Looking at sample code, all I saw were unnecessarily giant method calls and an unsettling amount of square brackets. Reading about programming never really helped me get anywhere. Looking back, I’m still glad I took the time to read over the documentation. It gave me a good overview of what functional programming was, even if it didn’t mean much to me at the time. I knew the only way to get a handle on it was to actually write some code. So I set off to build a CitiBike app for the iPhone.

It was pretty difficult to get started. A lot of the initial code and architecture was lifted from TimehopThree. I came into work on multiple occasions and sat down with Chris as he helped me fix bugs I had run into the night before, or explained to me how I should architect some component of the app. I managed to get the app working. And it got me to the point where I felt comfortable writing code for TimehopThree.

While I now had a greater understanding of ReactiveCocoa & MVVM, I was still skeptical. There were so many crazy techniques I had to use in my CitiBike app to get trivial tasks done.

"I have to add a command to the view model that sets a property on the model? Why can’t I just set the property directly from the view?"

Because of this, I found writing functional MVVM code to be much slower than writing imperative MVC code. It didn’t seem worth it to me. Why should I throw out several years of experience writing imperative code?

It wasn’t until I dove head first into the TimehopThree code that I started to see the light. The original Timehop iOS app was written in just a few weeks. Compromises that were made to ship the app sooner started to come back to bite us and slowed our ability to ship new product. All of that technical debt became too much to bear. So when I started to see how the Timehop 2 code translated to TimehopThree (and understood it), I was sold.

After spending another few weeks with ReactiveCocoa and MVVM, I knew there was no going back. One of the great things about this approach is that you’re forced to put much more thought into your code. We take the approach of strictly giving objects only the information that they need by passing the necessary information in through the object’s constructor. This can be a huge headache when you find that a view very far down the chain needs to know something about the user, for example. You’ll quickly find yourself modifying the constructor method signatures of several different classes to get at that information. This could easily be solved by using singletons. Need the user? Just do [THUserSession currentUser] and you’re done. This is great if you want easy access to the user all over your app. But what happens after you log out and someone else logs in? Has the view that depends on that user object already been created? Is it showing stale data now? Using ReactiveCocoa, most of these questions are moot as you just observe the UserSession and react (pun intended) to its changes by updating the view to reflect the most up to date state.

I’ve learned a lot since my CitiBike app. I plan to release it to the App Store someday, but for now it’s open source on Github. TimehopThree is done and we released it a few weeks ago. We did run into some performance issues from relying so heavily on ReactiveCocoa, but nothing we couldn’t handle. After several months of programming with ReactiveCocoa and MVVM, I can’t imagine ever going back to writing Objective-C code “the old way.” I’m even holding off on Swift until ReactiveCocoa 3.0 is ready.

ReactiveCocoa’s learning curve is steep. If you’re unsure of whether or not you should get into it, start off by reading the readme. Don’t be discouraged if you don’t understand everything right away. If you want some more hand-holding, Ash Furrow’s Functional Reactive Programming on iOS was a huge help for me. And don’t forgot to put what you learn to practice. Start programming with ReactiveCocoa and MVVM. Build an app that has a similar structure to my CitiBike app, or Justin Spahr-Summers’ GroceryList. But most of all, remember to be patient. Like anything worth learning, you won’t learn this stuff overnight. And when you’re ready, feel free to give us a call.

Welcome to the 6th post from Tales of Timehop! This post is operations focused and touches on some non-product/engineering challenges to scaling a startup. This post was written by Jake Simms. 

When you are small, everything is easier. Everyone works at the same table. Everyone fits in the conference room. Everyone knows the detailed history of the product and the jokes. Everyone is on the same project working the same long hours for the same reason.

During that primordial ooze phase of pre-product/market-fit startup, a culture is born, communication styles are established, behavioral expectations are made.

That same feel is incredibly difficult to maintain as new people join. All that “tribal knowledge” locked up in the heads of the existing employees needs to be extracted and shared to the newer members of the team.

In the last year, Timehop has grown from 7 to 18 employees so we’ve experienced these growing pains first hand. Here are some things we’ve done to help tribal knowledge become company knowledge, and ensure that culture and core values scale with headcount.

Write things down. Put them in one spot.

Sounds obvious, but it doesn’t get done unless it’s prioritized. Our internal wiki homepage is a one stop shop outlining the tools we use (Slack, Dropbox, etc), how we use them (don’t @everyone in Slack unless it is urgent), who to talk to about what (Office manager for equipment, Head of Talent for HR concerns), inside jokes (why are there always half eaten mozzarella sticks in the fridge?), product history (how a random feature experiment turned into 10x growth), perks, benefits, organization structure, quarterly goals, team structures, etc.

It’s a lot of effort, but it keeps everyone on the same page, and helps new employees get up and running fast.

Make onboarding someone’s responsibility.

It can be easy to overlook the experience of a new employee when you’re adding multiple people a month, sometimes multiple people in a week. Someone needs to be the point person for this process. Currently, that’s our office manager Yanique.

One week before starting, employees are asked for t-shirt size, favorite snacks, favorite drinks, equipment needs, etc. Those items are waiting at their desk on their first day along with a few goodies like a card signed by the team.

On their first day, they are introduced to the wiki and the documents mentioned. In their first week, we schedule a small social event to welcome them, typically team drinks.

Over the next two weeks, they meet with everyone in the company for 30 minutes. This puts a names to a face and helps build relationships quickly. This may not scale forever, but you should do it as long as you can.

Establish decision making frameworks.

Like all startups, we want a high alignment and high autonomy culture, so we use OKRs and Core Values to empower employees to decide which ideas are worth pursuing and which aren’t.

OKRs (Objectives-Key Results) state the company’s current initiatives. This helps keep teams focused on what they need to accomplish this quarter. OKRs let anyone hold up a task and evaluate whether it’s the right thing to be working on -- whether it helps achieve the objectives.

Similarly, Core Values help frame decisions and allow anyone to say “philosophically, this is why I think this is right or wrong.” Core Values are the core beliefs that govern how a company operates. They state why the company exists and what is important to the people in the company. If you examine past people, process, and product decisions you will see patterns in the reasons why certain decisions were made -- those are the Core Values. Write them down.

Tess and Benny led the charge in establishing our Core Values, and Tess will be writing another post on those in the future.

Welcome to the fifth post from Tales of Timehop! This post is engineering focused and details a Facebook API adventure as seen through the Golang code of junior backend engineer Jessica Weinberg. This post was written by Jessica Weinberg.

One of the most important and time sensitive parts of our backend infrastructure is our fleet of importers. We’ve discussed them here before but to summarize they are essentially background running jobs that are responsible for retrieving all your content from whatever services you have linked to your Timehop (Facebook, Instagram, Twitter, etc.)

Facebook Events: Creating an Importer

While we were already importing Timehoppers’ Facebook photos, statuses and links, we were not doing so for events. Since Facebook is deprecating FQL in the coming months (which is what the other importers are written in), having to create the events importer presented an opportunity to start using the new version of the Facebook Graph API.

To grab a list of events, you hit the /events endpoint, which will give you a response with two fields: data and paging. Within data, you'll find an array of event objects that contain fields such as location, name, start_time, among others. This search can be further refined by providing time ranges, limiting the number of returned events or even specifying which fields you wish to receive for every event entry. This is where things started to get tricky...

The Problem: Too Slow/Erratic Behavior

We wanted to be able to pull more information such as the list of people attending the event — data that isn't included in a regular response. However, attempting to grab all past events of a user with this extra information, I kept running into problems. One of the problems was the erratic behavior; sometimes it would work, sometimes it would error out, sometimes it wouldn’t give me back all the data. After much investigation, only two solutions worked: a) only asking for ids of the events or b) narrowing the time range while requesting all the events’ fields.

This lead to our next problem: speed. Going with solution B meant that we would have to break a single request into multiple requests (each of them for a given time range) with pagination still potentially occurring within these. This was too slow for our needs as we’re currently getting a new signup every 2 seconds.

Problem Solved: Batch It

Looking for a speedy solution, I had learned that the Facebook Graph API has the ability to make batch requests. With this functionality, we were able to simultaneously batch up to 50 requests at a time. With the ability to batch, we could use solution A — fetch all of the IDs, split them up in batches of 50, and then send them off in parallel. With Go, we were able to take advantage of what the language is amazing at: concurrency.

To give some back story to my journey, I had JUST started learning Go and am also fresh to developing in general, so the concept of concurrency was still pretty new for me. This wasn’t going to deter me though, I was ready to Go.

The first step was understanding how exactly Go handles concurrency. What I learned was that there are two main things you need for concurrency — a channel (or channels) and Goroutines. A channel is a language construct that is fundamental to Go that allows Goroutines to communicate with one another. Goroutines are like lightweight threads that can execute units of work, i.e. functions, at the same time.

Since we will soon need to migrate our other importers to use V2 of the Facebook Graph API, it made sense to create a function, GetContentInBatches, for batch requests and make it accessible for any Facebook Graph API request.

GetContentInBatches function

This function takes in a few different arguments:

The Graph endpoint (i.e /me/events/attending).

Since and until (which are both unix timestamps, used to specify the range of time you want to pull data for).

Var args of the fields that you want back (i.e. for events some fields might be description, owner of the event, and/or people marked as attending that event). The function then returns an array of BatchResponses or an error.       

First, we would need to make one request out to the Facebook Graph API to obtain all of the object IDs for the events the user has attended. The URL for the request is passed into the function GetFacebookGraphRequest which performs an HTTP GET request to the Facebook Graph API and then unmarshals the response into a struct named FacebookGraphResponse or returns an error.

The GetContentInBatches function then either returns an error or appends the obtained object IDs to an array which is then split into arrays of 50 object IDs. Again, the max amount of requests you can have in a batch request to the Facebook Graph API is 50.

Once we have appended all the URLs for the batch requests into an array we issue these requests (HTTP POST) at the same time, and then collect the responses from each goroutine using channels.

By taking advantage of concurrency as well as using Facebook's batch requests, we were able to get the most data per event object as well as get all of a Timehopper's events much more rapidly.

_______

Welcome to the fourth post from Tales of Timehop. This post is engineering focused and shares the language and tool decisions we made the past year during our first major growth spurt. This post was written by Benny Wong and originally posted on Underdog.io's blog. 

Happy new year! 2014 was a wild and exciting year at Timehop. Our backend team doubled in size but our user base grew by 17x! Our awesome friends over at Underdog.io interviewed us last week about Timehop's tech stack decisions which support this growth. We wanted to repost and share our secrets to stability. 

Describe Timehop in 2-3 sentences.

Timehop is your personal digital time capsule. We pull in your data from Facebook, Twitter, Instagram, iPhoto and other services and show you your memories from years past – kind of like a personal “This Day in History”. Our goal is to create a service that collects your history, captures your present, and makes it useful, meaningful, and accessible for you.

What are your primary programming languages?

Our main language on the backend is now Go. A year ago, we were mostly a Rails shop. Since then we’ve grown our daily active users about 17x. As you can imagine, a lot of things stopped working well and we needed to rewrite some systems to handle the growth. We tried a few languages and Go was the hands down winner for us. The main reasons we chose Go were for its incredible performance, sane concurrency constructs, and type safety. On the mobile app side, we have both iOS and Android. They’re both built natively: Objective-C on the iOS side, and Java on the Android side. Of note, both apps embrace functional reactive methodologies with ReactiveCocoa on iOS and RxJava on Android. It’s a mind bending experience to learn it, but eventually leads to a much cleaner and extensible app.

What are your primary web frameworks?

We’ve built a lightweight framework for timehop specific stuff (like authentication, API versioning, etc), but for the most part, we only use the Go standard HTTP library and gorilla mux for our API. In general, we’ve found the Go standard library to be an amazing set of tools. We don’t have much of a web presence as most of our traffic comes through our API, but for what we do have we use Rails.

What are your primary databases?

We use PostgreSQL, Redis, and DynamoDB. PostgreSQL is tried and true; Redis is amazingly fast and simple to use. But DynamoDB is our work horse. DynamoDB is an amazing product and I can’t imagine scaling without it. Our largest table in DynamoDB is pushing 63 TB and our average range query is about ~5ms. We chose Dynamo because it fit our workload perfectly. We are write heavy with a constant flow of writes as users sign up and we fetch their data. We are read light-ish as we currently look at only 1/365th of our data set on a given day. The ability to independently scale storage, read performance, and write performance was exactly what we needed.

Which DevOps tools do you use?

We use Cloudformation and Chef heavily. When we begin scaling we were a small and nimble team with no dedicated ops team so we tried to pick established tools with a solid community. We didn’t have much time to experiment or build a ton of tools, especially since we were focused on shipping a fast, fun, and stable app to our users. We just hired a DevOps engineer so this may change!

Which part of your stack are you most excited about?

Welcome to the third blog post from Tales of Timehop. These will be product & engineering focused posts to share some of the challenges we’ve faced and the lessons we’ve learned here at Timehop. This post was written by Rob Zajdel & Sarah Wood. 

"If I had asked people what they wanted, they would have said faster horses." -Henry Ford

"I know you want the new Gameboy, but do you need the new Gameboy?  -A Mom

Two very important quotes, from two very smart people. Both have a lot of value for running a time traveling startup, but until we finish building the time traveling remote that will slow down time and let us rewind the day, we have a lot to do and not much time to complete it. This post will discuss the problems with not testing ideas before building and a quick and easy way we test new features.

Building Without Testing 

Like most startups, we have put a lot of time into building new features that we eventually killed. One of those features was Timehop Nearby. It allowed users to see their photos and posts from where they are standing. Sounds cool, right? We decided to do a one-week hackathon to get this feature up and running. 

After really long hours (and *three* weeks of work), Timehop nearby was ready to be rolled out to users. 

We thought this was one of our more brilliant ideas, and we built the feature based on two core assumptions - 1) users had content that had location data and 2) users cared about seeing old posts and photos in their current location. This turned out to be one big awko taco because we soon learned that very few people were using Timehop Nearby. Lots of work and no return. 

In retrospect, we didn't need to take three weeks to figure out that it wouldn't work. We weren't as ruthless with scoping as we needed to be- we had a small time frame, but didn't want to build an incomplete or slightly broken product. That quickly led to us analyzing geodata for all off the uploaded desktop photos from our existing user base- an INSANE amount of work that would ultimately only impact the 1-2% of users who have that software installed, and for which we had no metrics to track. Here's what we should have done- rolled that feature out to new users only, and not worry about updating our entire database unless it took off. Long story short, when we build a new feature now, we try to learn as much as we can while building as little as possible. 

Testing A Feature Already Built 

We feature a news item every day at the bottom of the app. This used to be sourced by USA Today, and during this time sharing was really low. Sharing Timehop content is really important for our growth, so we had to make some decisions about the news feature when our contract with USA Today ended (learn why sharing is important for growth by reading our last blog post). We could build an entirely new news section - that would hopefully increase shares, kill the news section all together, or do something different with the infrastructure already in place for the news. 

We started with a gut assumption- people will re-share old viral content, because it sparks nostalgia and is inherently more shareable (because it has already been shared). Back in the days of Nearby, we would have designed and coded a totally new section of the app meant to support that specific set of content. Instead, we hacked our old news uploading system. This meant that we couldn't have a few features we thought might be essential to the experience- like being able to play an embedded video. And because we didn't want to wait for those results, all of the shares of this "old viral" content still tagged USA Today in all of the shares. Results? Super fail. Cost to the team? About an hour of brainstorming, and no coding at all.

You may be thinking: But wait? Timehop still has the news! That's because we still had the gut feeling that it was important, especially to users without much content in the day. So we're still using most of that old USA Today code, but we're paying an illustrator to make those stories more Timehoppy. Shares for the news are still low so it does't directly impact growth, but we always get positive feedback about it. It's not worth our time to overhaul this feature, but it is important not to ignore gut feelings. After rolling this out, we asked users if they preferred the old news or the new news- over 80% liked it better this way.

Testing Before Building 

If you build too many assumptions into a product, then you will have a really hard time figuring out why something is or isn't working. Instead, do small iterative tests to see if something can be successful. 

Currently we are testing to see if users would want to edit the picture that gets shared containing their past post or picture. We have an inkling that the Timehop banner and watermark might be getting in the way and creating a less enjoyable experience. To see if it would be worthwhile to build a feature to change this process, we are testing the idea before actually building anything. 

Now, about 2% of users see "Edit" over the picture on the share page. The catch? They can't do anything with it, and they will just get a message that says coming soon with an option to send feedback. 

Releasing a broken feature to a very small percentage of users allows us to see if it would be used frequently. If our tests come back showing that users were tapping Edit a lot because they were interested, then we know it could be something worth wile to build. If the results come back sour we can turn off the Edit feature to the small percentage of users who saw it and return to the drawing board. 

Conclusions

Timehop's product guru, Sarah, says they key to testing is to learn the most in the least amount of time, and to be ruthless with yourself and expectations.

What can we get from Henry Ford and Mom? People do not always know what they need. People did not know that they needed a Model T until they saw one because they thought all they needed was a faster horse. Mom knew that the new Gameboy might not get used because the older Gameboy basically did the exact same thing.

In order to have happy engineers, users, and mothers: do not build anything without testing out the idea first, but leave a little room for those gut instincts in case a test does not come back with much success.

Welcome to the second blog post from Tales of Timehop. These will be product & engineering focused posts to share some of the challenges we’ve faced and the lessons we’ve learned here at Timehop. This post was written by Rob Zajdel.

  Anyone remember that? That was the name of the daily email service that Timehop started out as in 2011. Co-founders Jonathan and Benny created the service at Foursqaure’s first ever hackathon. Users received an email each morning telling them where they checked-in on Foursquare on this day...365 days ago. But would people actually want to read this everyday, or would they delete it faster than Clint Demsey's World Cup goal?  

A nice surprise to us was that people did not mind getting an email from abe at 4squareand7yearsago dot com: around 60% of users read it daily. We then created a web based version for Facebook posts (pastposts.com) followed by And7YearsAgram, the Instagram version. We eventually merged all of this into one daily email called...Timehop! All was well in time travel land.

In this blog post Timehop CEO, @jwegener will discuss why cutting 90,000 happy email users was the best decision we ever made.

---

It's true. We once cut 90,000 very happy users. Another thing that might be a little hard to believe is that it was actually a pretty easy decision. We had a very highly engaged email user base, but they were not helping with our growth. Users enjoyed opening the email, but we could not get them to do anything with it.  

Timehop.com

Here's the rub: users cannot interact with their email. To fix this we created timehop dot com. It was a web destination where people could interact around the past, see their friends' feeds, and use other "social" concepts that allowed users to be more interactive. It was exactly what we needed because it allowed users to share their posts with friends, which leads to us gaining more users. Problem solved? 

Nope.

The new challenge for us was getting someone from their inbox to Timehop's website. It's a lot harder than it sounds. We tried some nice ways and some sneaky ways to get users to Timehop. We once tried only sending users a portion of their Timehop in the email and giving them a button in the message that took them to our website where they could see the rest of their day in history. I realized this was a little mean, so we stopped teasing our users. 

The team and I went back to the numbers and saw that over half of our users were opening the email on a mobile device. At first I did not think much of that stat because the Timehop email looked great on a phone. Why would we make an app when a Timehop email already had a working mobile experience? When the growth we needed was still not happening, I realized we had to do something else. 

iPhone App

This is when the idea of an iPhone app made perfect sense. An app would allow for more depth, more interaction from our users, and it would hopefully have the same level of engagement as our daily email. We took the time to build the app, and even on day 1 in the App Store we had more new downloads than new users signing up for the email. iPhone users were also coming back each day to see their Timehop, just like those signed up for the email. Problem solved? 

No : ( 

Now our small team was running a daily email service and an iPhone application (no time for fun). It got to a point where I would get to the office in the morning and my engineers would tell me our emails did not send today, but we also have to keep working on important things for the app. It was pretty obvious that investing our time into the email would not help our growth because for a year we were stuck at about 200 signups a day. With the app, people can easily share their content- which helps drive growth. 

Shutting It Down

We ditched the email (and over 90,000 users) about 9 months after the release of the iPhone app. Here is the email we sent our users:

Dear Timehopper, You're receiving this email because you're subscribed to Timehop's daily email service that tells you what you did 1 year ago today on Facebook, Twitter, Foursquare, etc.

We wanted to let you know about an important change. We're sunsetting the Timehop daily email and pulling all our efforts behind the Timehop mobile app. We appreciate your support and hope you'll understand that as a small startup we have to pick our battles carefully.

We'll stop sending the daily Timehop emails in 5 days: Wednesday July 17th.

If you have an iPhone/iPod/iPad? Get our app: timehop.com/iphone If you have an Android or another phone, we don't currently have an app for you but hopefully we'll get there in the future. If you'd like to help us with this, we're hiring -- get in touch!

Thanks for your continued support -- and see you on mobile! Team Timehop

Problem solved?

NOPE!!!!!!

Below is a screenshot of my inbox which was full of hate mail from our users responding to the shut down of the email service. 

This all was a little heartbreaking because some of these users had been with us since 4square&7yearsago, and if they did not have an iPhone they could not continue receiving their daily Timehop. We could not make everyone happy with this decision, but in order to save our time traveling ship from sinking the email had to go. 

Was it Really the Best Decision? 

You bet. Cutting the email really allowed us to focus on creating a great mobile product (who doesn't love Abe?). It meant the team's focus was not split and the messaging around our product was the same. We used to send two sets of emails out to our user base: one with information for our email users and one with information about the app.

The decision also improved management of our infrastructure because it was always a bit fractured. The infrastructure to send emails is very different than the infrastructure needed to support a mobile app like Timehop. (Learn how we improved our infrastructure to fetch a first time user's social media history in seconds by reading our first blog post)

Now, more people download Timehop every two days than all people who ever signed up for the daily email.

Welcome to the first blog post from Tales of Timehop. These will be product & engineering focused posts to share some of the challenges we've faced and the lessons we've learned at Timehop. 

"Thanks for signing up -- you'll get your first Timehop email tomorrow!"

This little message greeted all new users for the first year of Timehop's life. After you had finished signup, our servers would import your history off the different social networks and create the Timehop experience you've come to know and love -- as long as that importing finished by the next day's batch of emails, all was well with the world.

But now that Timehop is an iPhone and Android app, new users expect the app to work instantly upon sign up. Which means that when a user starts the signup flow in the app, we only have a few seconds to finish the import process and deliver the first Timehop user experience. 

In this post, Timehop infrastructure engineer @kevrone shares the specifics of how we accomplish this!

Enjoy, Jonathan, Timehop CEO/Co-founder

--

Say it with me now:

First

Time

User

Experience

Or FTUE for short. (Phutooey!)

First, some historical perspective on why we have a special flow for FTUE at all…

Blank Days

When a user first signs up, they expect the app to give them some value right off the bat. Seems fair. But because we source content from third party APIs it can take some time to fetch all that data and serve it back to the app before the user finishes the signup flow and starts looking for content. If we're not fast enough with the fetch and delivery, they'll see the dreaded Blank Day and go into apoplectic fits of fiery hatred that they direct at our support email. Patient users will open it again the next day, but it’s likely (and understandable) that users might just lose interest or delete the app entirely. So it hurts user retention. No bueno.

Old FTUE

To avoid hate mail as much as possible, our first approach was to create a separate FTUE process that just focused only on grabbing the data we needed immediately: *today* for each of the last 7 years. Then we'd serve that to the user until the normal processes kicked off to make sure the rest of the year's data was abso-positive-lutely imported (about 1 hour, to be exactly imprecise).

All of that fetching was initiated directly in our Rails front-end, processed by delayed jobs, then stored in a temporary cache.  This approach worked well at first, but it failed to keep up with our current breakneck pace of new user growth. In the past few weeks, our delayed job queue was getting swamped. Many of these FTUE jobs got backed up and that meant many new users were seeing no content in their app the first time they used it. Again, no bueno. It also meant that we had a totally different logical flow for presenting content the first time, which created additional complexity.

So,

Blank FTUE rate for new users during this dark age: 80-90%, meaning these brand new users were seeing a sad "you have no history on this day" message (Did I mention no bueno?)

Scaling potential: (╯°□°）╯︵ ┻━┻

New FTUE

Welcome to the bright, bright future. Now what we do is offload all of those delayed jobs to Go processes in the backend and send "done" signals back to the Rails app to break user caches. We actually just piggy back off our existing importer logic (an interesting topic on its own that I’ll leave for a future post) and dedicate a server to process all the FTUE content asynchronously. With this approach, all the user’s FTUE content is usually in place before they finish the signup flow (a few seconds). And because we use the super-scalable DynamoDB as a datastore, scaling is just a matter of allocating more capacity (read: spending more $).

Now there's no reason our FTUE jobs should ever get backed up, but if they do, we'll be alerted within five minutes via Stathat. A lesson here for all you infrastructure folks is that any solution you come up with to an infra problem should be paired with metrics that verify the success or failure of those changes. It’s a small investment that will go a long way towards your overall psychological sanity.

These changes also means we've simplified our system because it keeps all the presentation logic in Rails identical for new and existing users, removes a layer of caching we don’t want to manage, and offloads processing cost from the front end to our service infrastructure (ie: Go instead of Ruby).

So,

Blank FTUE experience in this infrastructure paradise: 20%. (As not every user has content everyday, we wouldn't expect it to be zero. It's now inline with our expectations). It works! It really really works!

Scaling potential: ┬─┬ノ( º _ ºノ)

Here’s a nice Mixpanel graph showing that we started having more and more blank FTUEs as the infrastructure fell behind...

  ...and switching to an hourly view in Mixpanel demonstrated that the FTUE imports were massively backed up during peak hours of the day as shown by the spikes below:

The new infrastructure was a big win -- notice how drastically different May 7th starts looking -- no blank FTUE spikes! Hurray for a great new user experince!

Thanks for reading.

Welcome to Tales of Timehop! Here we'll be sharing the challenges we’ve faced and the lessons we've learned growing Timehop from a hack day project to something millions of people use (spoiler alert: we’ve learned a ton). We're considering this our chance to contribute back to the community:  

Here are just a few sample topics of what can expect going forward:

How we quickly test our assumptions before building

Running A/B tests and how to track metrics effectively

Our technology stack (Rails, Golang, DynamoDB, Redis)

DynamoDB best practices

ReactiveCocoa

And some juicy stories too:

How an accidental phone call closed our seed funding

How a tweet saved us $9,000

How a flower bouquet saved us from being homeless and nabbed us an awesome office space

How we prototyped a $100,000 feature in 10 minutes without coding 

How we spent 8 months and $500,000 building a feature we trashed

How shutting down Timehop's daily email was the best decision we ever made despite cutting out 90,000 users