Gilt Tech

What is Microservices Day? 

from microservicesday.com

Microservices Day is a conference by the enterprise, for the enterprise. It brings together people from organizations that have adopted (or are planning to adopt) a microservices approach to running their business and focuses on the issues that companies face. It gives participants a forum to discuss and share their experiences with microservices through advice, tips and tricks as well as driving forward both the technology and business community. Come join us at The Altman Building, 135 West 18th Street in New York on July 13th! The event will consist of a range of talks from industry leaders who have deep experience building and operating microservices systems at scale. Panel discussions and active audience participation will be a further part of the event, as the implications of enabling business to innovate at a faster pace than ever before will be a key discussion point. Come join the early adopters at the dawn of a new era in business computing!

Sean Smith is a Software Engineer from Canada. He currently works at Gilt as a Principal Software Engineer. He has worked on a variety of services at Gilt, ranging from third party warehouse systems integration to email content generation and delivery.

An analytics and event tracking sleuth.

In late 2014 Gilt received word that a core feature of Google Analytics (henceforth known as GA) would be removed in the next iteration of the platform. The ability to redirect collected data and events to a secondary endpoint had grown critical to our the analization of user behavior, patterns, and effectiveness of our user experience, as well as sales forecasting and predictive modeling. Our two most viable options for retaining this ability were to implement another tracking platform from a different vendor or create and implement our own, tailored to our specific needs.

https://github.com/gilt/Gumshoe

After taking a hard look at the heavy hitters of the analytics platform sphere it was decided that we'd take the divergent route and roll our own analytics and event tracking library. We won't go into detail on why we passed on big players like Snowplow, Omniture, CoreMetrics, et al. Rather, we'll focus on Gumshoe and its merits.

What Did We Need?

To anyone in development, marketing, or data sciences on the web, our list of needs will sound quite familiar.

First and foremost we required parity with GA's base page data. The industry has come to be reliant and has standardized on the kind of data that GA collects for various reporting, including vital year-over-year reporting. So no shockers there.

As performance-minded developers we also had a low page footprint in mind.

We desperately needed organized event names and data. One major shortcoming of GA was the lack of enforcement of any kind of standard in event naming and data. Arbitrary event names and varying event data formats produced migraines the size of Mount Fuji for our Data Scientists. It wasn't uncommon to have the same target event with several different names, in different applications, with different data in different delimited formats.

Related to the variance in event data, we were looking for a high degree of data integrity and confidence.

And last but not least, a low delivery failure or miss rate for the data collection.

Rolling Our Own

Events are events are events. In Gumshoe, everything is an event. There's no disambiguation between page views, virtual page views, or custom user events. Everything is an event, and it's a core tenant of Gumshoe.

Right off the bat, parity with GA was key. We started by breaking down the data that GA collected from each page view, and moved forward from there. A basic Gumshoe event consists of the following:

pageData eventName eventData

pageData consists of the GA parity data. eventName is self-explanatory; the name of the event we're passing. eventData is custom data passed by the consumer which should be associated with an event.

Wunderbar! We had the recipe for sending a page view event. But how do we set this up so that others can easily send these events to where ever they'd like? Enter the transport.

Transports are the mechanism(s) by which consumers instruct Gumshoe on how it should send the data for each event. One can configure Gumshoe to send to multiple transports, or a single transport, it's your choice. A brief example from the repository:

(function (root) { var gumshoe = root.gumshoe; gumshoe.transport({ name: 'example-transport', send: function (data) { console.log('Gumshoe: Test Transport: Sending...'); console.log(data); }, map: function (data) { return { customData: { someData: true }, ipAddress: '192.168.1.1' }; } }); })(this);

The send method is responsible for actually sending the data. Gumshoe bundles the https://github.com/gilt/reqwest library, which any transports can freely use. At Gilt we use reqwest.js to send the data to our backend event stack, which we'll cover in subsequent blog posts.

The map method allows consumers to extend the data which Gumshoe is sending through the transport. At Gilt we use this method to attach a giltData object that is sent along side pageData and contains vital Gilt-specific data for every single event that we send.

Using Gumshoe

Once you've got Gumshoe included on your page and you've specified a transport, you'll need to initialize the library. It's as simple as specifying the transport:

// tell gumshoe to use our transport window.gumshoe({ transport: 'example-transport' });

And sending an event:

window.gumshoe.send('page.view', {});

What's Next

In forthcoming posts we'll be talking more about the other goals we listed, specifically how we internally solved event organization, event naming standardization, data integrity, and the remainder of the stack that takes over once Gumshoe has delivered the payload.

The video from the recent Atlassian webinar featuring Senior Director, Heather Fleming is now available!

Introducing Gilt Tech's first official Summer Apprentice Program class! Here's our class described in their own words :D

Alex Luo

Has no middle name

Lived for more than two years in three different countries (China, US, Israel)

Walks to class in the snow in April because he goes to Cornell

Once volunteered by distributing medicine to poorer families in rural Dominican Republic

Voted “best eyes” for high school yearbook

Ate a cricket once

Fixed a graphics card by baking it in the oven

Broke a bone from accidentally falling off a building

Got kicked out of the public library once for “hacking” a computer

Actually can believe it’s not butter

Named by Time Magazine as 2006’s Person of the Year

Purposefully uses Comic Sans in documents to tick people off

Courtney Ligh

Courtney is a native New Yorker studying Computer Science at Dartmouth College. She is a future software developer and is excited about her summer internship here with Gilt’s Development Team. Besides coding, Courtney has a passion for food and prides herself with knowing all the best eateries around the city.  

Yogisha Dixit

The only thing you need to know about me is that I'm obsessed with chocolate. Literally...

Okay, maybe not. I'm interested in harnessing the power of computers to help make the world a better place. And I'm really looking forward to learning a lot and being challenged this summer.

That's it. I swear.

And a very special Welcome Back to

Helena Wu

Registration is open for this Webinar on May 27th!

Summary below from Atlassian.com:

A new Harvard Business Review Report asserts that agile development practices have steadily risen to become one of the most trusted and preferred methods of development across software teams in almost every industry. The study also discovered that by using agile frameworks, organizations can respond to market changes faster, deliver higher quality software, and gain a significant competitive edge. For these reasons HBR has dubbed agile development the competitive advantage for a digital age.

Join Jake Brereton, Sr. Brand Manager at Atlassian, as he discusses these HBR findings with Heather Fleming (Sr. Director, PMO, Gilt), Nate Van Dusen (Engineering Program Management Director, Trulia), and Maira Benjamin (Director of Engineering, Pandora); three seasoned agile experts who have successfully lead agile transformations and witnessed tangible, positive results firsthand.

In this webinar you'll learn:

How Trulia benefited from taking engineering agile principles and spreading them throughout their entire Enterprise

What agreements engineering made with Gilt leadership to improve their unique agile development process

Adrian Trenaman, VP Engineering at Gilt will be speaking tonight at our Gilt offices on how we adopted Solr at Gilt to power our personalized search results!

We will be raffling off copies of Solr In Action after the presentation! Drinks and sandwiches will be served.

Registration closes soon, so sign up now!  It’s free!

Google Trends offers a trove of data for analysis. It's not used nearly enough partially because good folks at Google did not provide an API to access the data. You can play with Trends in you browser, embed it into your webpages but it’s not that simple to get the raw data behind it to use it in your analysis.

There is a number of packages in Python, Perl, or R that pull the data for you but none of them did when I needed: compare hundreds of trends against each other.

You see, not contend with the lack of API Google returns trends on 1 to 100 scale so it’s hard to compare numbers for many different trends. You can plot several trends on the same graph but you will not be able to tell how they stand relative to another set of trends.

For example:

Above: “Game of Thrones” vs. “House of Cards”. Below: “Orange is New Black” vs. “The Newsroom” 

Since each set is rescaled there is no way to tell how “House...” stands against “Orange...” 

Gilt Tech is proud to announce the release of CleanroomLogger 1.0, an open-source logging API for iOS.

CleanroomLogger provides a simple, lightweight and performant logging API written in Swift that will be readily understood by anyone familiar with other logging packages like CocoaLumberjack and log4j.

CleanroomLogger Highlights

CleanroomLogger provides these key features:

Classify log messages according to one of five severity levels: verbose, debug, info, warning and error

See the source file and line number of each log message recorded

Silently ignore messages below a given severity threshold

Far more respectful of the calling thread than NSLog()

A trace() function that can be used to trace an application's code paths by logging the location of its caller

Configurable log behavior that puts the application developer in control of embedded code

Extension points that allow custom implementations for (1) log message formatting, (2) message filtering logic, and (3) the recording of log messages to an underlying facility.

Message logging

Each message sent to the log is associated with a severity value indicating the relative importance of the message.

You would send an informational message as follows:

Log.info?.message("The user has added \(count) items to the cart")

If a condition occurs that itself isn't an error, but that might signal future trouble, you might issue a warning:

Log.warning?.message("We're running low on space!!!")

Finally, when things hit the fan, you've got:

Log.error?.message("Oh my, this is extremely embarrassing")

Each log message is recorded with the file and line that issued it, so you can very quickly figure out what code was responsible for a given message.

Execution tracing

As you're developing and testing your code, you might want to understand the flow of execution your code takes. You can add trace() calls to various points in your code:

Log.debug?.trace()

The code above would print out the file and line number containing the trace() call, as well as the name of the calling function. The output looks like:

ModularTable.swift:364 — tableView(_:cellForRowAtIndexPath:)

Logging arbitrary values

Sometimes life hands you an NSError. Why bother constructing a new message? The NSError is the message:

Log.error?.value(err)

This function accepts Any?, meaning it'll handle anything you throw at it.

Extensibility

CleanroomLogger's extension points allow you to fully customize all aspects of logging without having to worry about the mechanics of how a logging engine is implemented.

The ability to provide one or more custom LogRecorder implementations is particularly powerful. You could implement your own LogRecorder to do things like:

Write messages to a file or set of (potentially rotating) files

Store messages with a warning or error severity in a database for subsequent diagnostic evaluation

Send messages with an error severity to a network endpoint to enable rapid response to codebase instability

And because more than one LogRecorder can be used at a time, you can multiplex your log messages to multiple destinations without writing any additional code.

About CleanroomLogger

CleanroomLogger is part of the Cleanroom project a set of open-source Swift projects launched as "an experiment in re-imagining Gilt's iOS codebase in a legacy-free incarnation that embraces the latest Apple technology."

Reserve your spot before they’re all gone!

Learn how we adopted Solr at Gilt to provide personalised search results, from early days using Solr 3 on just three nodes, to a more modern clustered deployment today running Solr 4 that powers all listing pages on Gilt through our daily pulse load at noon. We'll discuss what worked, what didn't, and where we're going next for search on gilt.com.

Join us on Tuesday, April 21 at 6:30pm for the NYCPUG meetup!  

David Steele in town to talk about PgBackRest and how to efficiently backup massive amounts of data.

Last week, Amazon made their Elastic Container Service generally available and also released a web UI to make it easier to create tasks. This service was exciting for our personalization team as we wanted to leverage ECS to simplify deployment of our jobs and have better control over failures.

Getting started with ECS is easy but requires your full attention because there are a lot of important details (did you remember to set the correct IAM roles for your container instances?). If you haven’t done so already, make sure you go through the Getting Started documentation.

Once you’re ready to run your task what to do if something fails or doesn’t go as planned? Here are some of our experiences debugging ECS tasks.

Start with a simple application

In our case we created a Scala main method with very few dependencies. A sleep method is sufficient to simulate a job running and should provide enough time to see our container running.

Make sure your cluster has at least one container instance

Your ECS cluster requires at least one container instance to run your container. You can check the web console to make sure there is at least one running.

If you don’t see your instances here make sure you’ve followed the Container Instance documentation. In particular that you launched an instance with the correct ECS AMI, that you have attached the correct IAM role and that you have pointed your instance to your cluster.

Inspect the ECS logs

Once your container instance is attached to your cluster there will be an ecs-agent running on the machine. (note: the ecs-agent is open source and on github so you may be able to find help on the ‘issues’ page if needed ). The ecs-agent emits logs that can help you debug. You will need to ssh to the container instance and inspect the ecs-agent log:

/var/log/ecs/ecs-agent.log

For example, when inspecting this log it can be useful to know that the agent has pulled your docker image:

t=2015-04-14T15:46:53+0000 lvl=dbug msg="Pulling image" module=TaskEngine image=<my-registry.my-domain>.com/<my-image>:latest status="Pulling dependent layers"

Make sure the container instance installed your docker image

You can check to make sure the ecs-agent successfully downloaded your docker image. ssh to your container instance and run docker images. Your docker image should appear in this list.

If you don’t see your image check the ecs-agent logs and make sure your container instance is able to contact your docker registry by curling the registry and printing out the available tags:

curl 'https://<my-registry.my-domain>.com/v1/repositories/<my-image>/tags'

Manually start your docker image

Checking to see if your image can be run from the command line is a great way to make sure the image can be run on the container instance. This also gives you an opportunity to inspect the application logs. For example,

docker run -it docker-registry.gilt.com/cerebro/svc-cerebro-job:latest bin/cerebro_ngram_job.sh production

Check for failed container processes

If your container can be run from the command line but is still not being run automatically through ECS Run Task then it’s likely you have something wrong in your ECS Task Definition. Check for failed container processes and inspect their container configuration.

To check for failed containers run docker ps -a.

Once you have the Container ID you can inspect the docker configuration. If you see an exit code other than 0 there may be an issue with your container configuration.

docker inspect <containerId>

For example, I noticed an exit code of 127 which indicates a bad command. This led me to fix a bug in the task definition CMD field that requires commands to be passed in json array format.

Finally, you can inspect the application log of the container.

docker logs <containerId>

For reference, here is a working task definition:

At Gilt, our open-source contributions generally come in the form of tools that we use internally that we think have value to the rest of the tech community. But like most other companies, we have internal systems that are very specific to Gilt. This makes open-source contributions tricky because our potential projects and tools are so tightly coupled to our systems. We wanted to bridge this gap and decided to do so through a tiny front end initiative we called Open Source First.

For example, our front end architecture consists of over 100 requirejs modules that are bundled and served by an in-house legacy build system that is as tightly coupled as a ball of rubber bands. And, our internal requirements make it necessary that we wrap requirejs which means we oftentimes aren’t using off-the-shelf versions of each module. These modules depend on each other in ways that would make spiders jealous. It works. It’s stable. And once the learning curve is conquered, we defend its design while sipping our orange Kool-Aid. But, it makes open-sourcing any of these individual modules just about impossible.

The Special Operations team (Kyle Dorman, Jose Sanchez, Archana Kumar, and myself) set out to create a pattern by which we could open-source modules that could be used by the greater tech community without sacrificing any characteristics of the module had it been developed internally. We used a project that we had just finished called OAFE (Open App From Email) as the basis of this experiment. OAFE is one of those mobile-web user experiences you see that asks “if you’d like to install the app.” Internally, we have a lot of discourse about whether this is a good experience for our users or not. As a result, our implementation of OAFE is a little complex. It tries to be the best experience for people who want to be redirected and people who want to stay on the mobile web. OAFE was a perfect candidate for an open source project because it needed to be implemented on two completely separate code basis (Gilt and Gilt City).

We created UBAR (Unidirectional Browser App Resolver) as an open-source first module that behaved as OAFE. Our goal was to write a front-end module that could be easily consumed both by our current system and externally. We did so without sacrificing any internal requirements or features. Are we expecting that other people are going to use UBAR on their websites? Frankly, no. So, why go through this exercise of creating open-source modules? This is where the learnings happened that ultimately forced us to think better and harder about our work.

5 Key Learnings

PRIDE Your code being publicly available instills a sense of pride. This is our work. This what we are capable of doing. This is how we do it. We want it to be great because we want others in the tech industry to appreciate it. As a result, we are quick to make fixes to bugs and quick to make improvements to make it feel better. Since it’s something we want to be proud of, we all felt an energy of wanting it to be great. It has been a great team-building exercise.

GIVING BACK We shared our creative process for implementing an open-source module so that we could inspire others to do the same. For so long, we have taken from the front-end community without giving back substantially. This is our attempt to give back, just a little. It feels good.

CODE LEGACY Regardless what happens with the feature, the engineers on the team have a really cool sense of pride for their work. The hard part about working on websites is that features are pulled down as frequently as they’re put up. This is our way of giving some legacy to the code.

WORKFLOW In the chance someone wants to use our open-source module, they can contribute to it. This allows input on our work from sources that we don’t necessarily interact with on a daily basis. One of our team members left the company during the development of the feature but since it was open-source, she was able to continue tracking its progress.

BETTER ARCHITECTURE Open-source first completely reshaped the way we thought about architecting this module, for the better. We did not want to remove any characteristics or attributes of the module from the open-source project. We wanted to eat our own dogfood. As a result, we thought a lot about how this module could be consumed by systems that were not our own. We created a patterns for how external users could easily implement tracking analytics, template rendering, deep-link requirements, and design. Through these abstractions, our code is easier to read and less tightly coupled than other comparable modules that we created.

Gilt was fully supportive of this effort largely due to the culture set forth by our technical leadership. And, that really moves us to our last point: this made us really excited about our work. It made us bond well as a team. It made us feel like we were making positive contributions to Gilt, to ourselves, to each other, and to the tech community. It felt like good technical culture.

And since we created patterns and frameworks, it’s a technical culture that we can share both internally and externally. Going forward, we’re going to try to open-source everything, first.

To view UBAR’s source code, please click here.

Senior Director, Heather Fleming (center) recently spoke about workplace culture and diversity at the “Let’s Talk Culture NYC” panel held at Paperless Post with Vox Media.  Other panelists included Millie Tran from Buzzfeed, Jackie Balzer from Behance/Adobe, and Kyle Macdonald from Carrot Creative.

Interested in having your own panel on culture?  Here are some of the questions from the discussion:

“Culture fit” can be used negatively or positively. How do we stop it from being used negatively?

Do you work with people from other teams and how do those cultures differ or influence one another?

Is it important to you to encourage healthy work/life balance on your team and at your company? How do you or other team leaders do this?

How do you try to get a diverse set of candidates for job openings? What issues do your team face when you’re hiring for a new position and what do you try to do to expand the pool of candidates?

Live in Dublin, or plan to hang out there in April? Tired of Java, and intrigued by this sleek and elegant Scala language you keep hearing about? Excited about learning languages for free? Then fill out the form below to express your interest in our upcoming free Scala course! Taught by some of Gilt Tech’s resident Scalasmiths, this one-day, beginners-level class will take place Thursday April 16th in our Dublin office. Our class will cover fundamentals such as:

Classes, traits and objects

Tuples and case classes

Monads

More!

The training day will last from 10 AM to 5 PM with a working lunch. Developers of all programming languages—but especially Java—are invited to participate. Just fill out the form below to submit your contact info, and we’ll let you know if you’ve been selected. Submitting this form does not guarantee you a seat. However, it does mean that you’ll receive details about the other courses we’re developing—including advanced Scala and Play—as soon as they become available.

REGISTRATION FOR THIS SESSION IS CLOSED

Gilt's Mockingbird open-source project for iOS was recently recognized by the popular Awesome iOS page, which maintains "a curated list of awesome iOS frameworks, libraries, tutorials, plugins Xcode, components and much more."

The MBNetworkMonitor and MBNetworkIndicator classes from the Mockingbird Toolbox project can now be found under Awesome iOS's section on Networking code.

(We've written about the MBNetworkIndicator class here on the Gilt Tech blog in the past.)

In other news, a little birdie recently told us that three of the Mockingbird repositories are already among the most popular Cocoapods — and one of those hasn't even been publicly announced yet!

Four Mockingbird repos are currently available for public use under the MIT license on GitHub. Two of these repos are post-1.0 and are fully documented:

Mockingbird Toolbox (MBToolbox)

Mockingbird Data Environment (MBDataEnvironment)

Two more repos contain pre-release versions and are awaiting full documentation:

Mockingbird Event Handling Module (MBEventHandling)

How do you create and maintain a great company culture? What does “culture” even mean? We often simplify the idea of culture (especially in tech) as pertaining to physical objects or events such as ping pong tables or happy hours. But what does culture mean in the context of scaling a healthy, happy organization that is productive, collaborative, and diverse?

Senior Director, Heather Fleming has been invited to be on the panel discussing these topics and we hope you’ll join us for the conversation!