OpenSky Engineering

At OpenSky we've gone through a few different iterations of how we manage changes to our software. This post aims to describe some of the changes we went through and why.

How did we historically operate?

Historically we had 3 branches named develop, release and production. Developers would work on changes in a branch created from develop. Once the work was done, the branch would be merged back in to develop. Then every 2 weeks, we'd move everything from develop to release and everything would be tested together before being merged to production. Then a tag would be created from the production branch and the tag would get deployed. Hotfixes and other small changes would sometimes be branched and merged directly to the production branch.

This approach works well but it had a few problems for us:

If we merged 20 different changes in to the develop branch and 1 of the changes has an issue, it would block the other 19 changes from going to production until the 1 issue was resolved. In some cases if the bug wasn't fixed quickly, we'd have a process to revert the change out of the release branch so that everything else can move forward.

If 1 of the 20 changes has an issue once it is deployed to production, it is a little more difficult and time consuming to quickly flip it back. It requires a git revert and a new tag to be created. This has to be done either way, but you can't just flip the one change back instantly without a full deploy process.

If you have a problem during testing or in production, it is hard to know which of the 20 changes caused the problem so everyone has to jump in to figure out if it was their change or someone elses.

We have two different paths & processes for how something can get to production. The question was always asked if a change should be branched from develop or production.

Other things were more complicated and messy as a result of having multiple branches that I can't remember :)

How did we improve this?

Over the years we invested heavily in our automated test and deploy processes. The goal was to make the processes very fast, stable and easy to rollback if something goes sideways. A developer should be able to push up a change and get feedback from the automated test suite in 2-3 minutes or less. It used to take up to an hour to get feedback from the tests because the tests all ran serially. We built a tool internally called PHPChunkit that allows you to run your tests in parallel chunks across multiple servers. On top of the speed issues, the coverage was not that great, so every change required lots of manual QA before it could go to production. The need to manually QA everything meant it was very costly to test and deploy individual changes in isolation. This is one of the reasons the above process to batch things up and test and deploy them together every 2 weeks was used.

We also improved the speed to deploy to production. The process used to be a series of automated steps that were executed by a human and would take up to an hour sometimes. Once this process was fully streamlined, optimized and automated we were able take a change to production in less than 10 minutes. This is including running the full test suite before going to production.

There was a bit of a cultural change that happened through all of this too. Due to limited manual testing resources, we mandated that developers must include unit and functional tests with their changes. If they fix a bug, the first commit should be a test demonstrating the failure and the 2nd commit should fix it.

Some of the tools we use are:

Jenkins - Job management.

PHPChunkit - Used to chunk up and parallelize the execution of our PHPUnit tests across multiple servers.

pr-nightmare - an internal tool that coordinates the work between JIRA, GitHub Enterprise and Jenkins. I hope to make this generic enough one day and open source it, but unfortunately it is not public yet.

Fabric - Fabric is a Python library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

How do we use branches today?

Once we had solid CI & CD processes, it was time to change how we worked with branches. We got rid of the develop and release branches and all changes get branched off of production. All changes are tested and deployed in isolation. Branches are usually short lived and are tied to a single developers work. We now deploy an average of 5 times per day and have deployed as many as 20 times in a single day. Our overall throughput went up, we have less issues in production and our fear of change decreased.

Now new developers deploy to production on their first day on the job!

Screenshots

Screenshot of GitHub Enterprise:

Screenshot of pr-nightmare:

We are hiring!

The 5th edition of Sensiolabs’s SymfonyCon took place in Cluj-Napoca (Romania).

Cluj, the European Youth Capital of two years ago, is a dynamic city with a large university center and a fastly growing software industry. The conference felt equally dynamic, with its varied choice of activities: workshops, lectures, a hackday, unconference speeches, community talks and social events.

Five members of the OpenSky team flew in through a thick fog to participate in the conference, for the opportunity to hear about Symfony 4 directly from creators and contributors, attend the good assortment of talks and get a feel for the international and local community. Symfony is at the core of the OpenSky technical stack, and Pablo Godel’s conference lecture gave a glimpse into how we use it and how we keep things running smoothly (and fastly paced too!), for our large-scale e-commerce applications.

Two days of workshops preceded the actual conference, which lasted for another couple of days, on the 16th and 17th of November. The talks were organized into three tracks, taking place at the same time in adjacent rooms: track advanced, beginner and PHP, with the criteria to split not always obvious.

The conference debuted with Fabien Potencier’s keynote.

At 9AM, Fabien Potencier delivered his keynote, not surprisingly in front of a room packed full of people. It was focused on the eagerly expected Symfony 4, lightweight, easy to start with, easy to turn into a full-fledged application. In a hands-down fashion, a live demo of installing a Symfony 4 application with SymfonyFlex was delivered in parallel with Fabien’s words. It showed a quick route from 'Hello World' to a full stack app, blink and you've missed it.

Package aliases in Flex recipes, turning commands into readable, easy to remember English, the Maker Bundle, that automates the creation of controllers, forms and other structures, were among the additions that make Symfony 4 so simple to start with. For existing users, which we suspect was the case for most of the people in attendance, Fabien declared a smooth upgrade path, from the very the first sentences of his keynote speech.

We were left to ponder on the difference that Symfony 4 will make in the adoption of new users and how soon we can upgrade our applications to benefit from the new approach.

Twenty six other talks, including 3 keynotes with a full audience, made up the rest of the conference.

There were worthy lectures in all of the tracks, making the selection difficult at times, and a few of them are mentioned below, in no particular order.

Nicolas Grekas discussing new features and performance of a central part of Symfony, ‘Dependency Injection Component v4.0’, attracted a high number of viewers. He spoke of features such as autoconfiguration (“automate your own config by defining your own conventions”), that every class is potentially a service and everything is by default private, of autowiring by service identifier, processing environment variables, of new features as it relates to performance and the compiled container. A slideshow worth revisiting slide by slide.

‘Lessons learned building the Composer internals’, held by Jordi Boggiano concluded the first day of the conference and gave an inside glimpse into challenges encountered while writing code or in the management of the development process. It was an insight into what it takes to provide good service for a software that is so widely used and on such a variety of systems and configurations. The presenter also drew on lessons learned in managing open-source contributions, and the sustainability of open-source projects with a high growth rate.

Marco Pivetta's 'Event Sourcing: The good, the bad and the complicated' was a real-life example of a code review marketplace where the domain of the application is modelled as a finite-state machine, with immutable states and transitions between them. With this model, the state of the application can be obtained by playing the entire sequence of events, which is persisted in a database. He went into details about implementing such a model in PHP, what are some of the solution’s advantages and what could go wrong.

‘Webpack Encore - Pro JavaScript and CSS for Everyone’ saw Ryan Weaver, in his usual gripping speech, present a simpler way to integrate Webpack into a Symfony app.

David Buchmann spoke of 'Decoupling an application with message queues', covering general notions of a message queue system, how AMQP-compliant message queues can be accessed from PHP, benefits and caveats.

‘API Platform and Symfony’ by Kévin Dunglas was an example of starting up a HTTP REST API application in a very short time, complete with CRUD functionality, documentation, and an administration interface for testing. The questions from the audience were related to authentication and authorization support, which were not included in the demo, and API versioning, which is not available.

The opening of the second day was the keynote on ‘PHP 7 and beyond: 7.2+’, where Sara Golemon gave an instructive overview of the new language features introduced in the minor versions of PHP 7 and the possible new features to be added in later versions.

Sarah Khalil’s ‘A year of Symfony’ keynote was a good wrap-up talk on what has been achieved in the last year in Symfony, marking the 1 billion download milestone, notable features added in 3.4, the upcoming Symfony 4, numbers of pull requests and issues fixed.

Symfony is a central part of OpenSky

Pablo Godel’s ‘Symfony at OpenSky’ gave a window into how we work at OpenSky to develop successful, large-scale applications with Symfony, working from offices on three continents. He went through OpenSky’s technology stack and the application organization into bundles, with general recommendations and lessons learned along the way from the early adoption of Symfony 2.0 in 2010. He has shown how the instrumentation of the development process from coding to production deploy ensures a fast deployment pipeline to release as frequent as necessary. OpenSky’s commitment to Symfony coding standards, best practices and good code coverage help maintainability and save development time.

There was good opportunity to deliver unscheduled speeches and meet the community.

For those that had missed the submissions deadline, there was a second chance to speak on stage, either in the Community Talks (7-minute speeches about the community or an open source project contribution) or in the Unconference. The latter filled the entire PHP track of the second day, and some saw this as a venue to ask questions, others talked about small subject. All in all, it was a good way to get people prepared to hold a talk and to get the community’s feedback in a more informal way.

Informal was also the mood at the happy hour, which took place in the evening of the first day in one of the clubs in Cluj, where there was mingling and even some dancing, which left us to wonder if the low number of women in the conference is representative of the Symfony community.

On October 19th and 20th the 2017 US edition of the SymfonyLive conference took place in the San Francisco, California. 

It was two days full of technical talks all around this popular PHP framework, which we love and use for all our marketplaces. 

In addition, we had two members of our team participate at the event, and one of them presented a talk about how we at OpenSky use Symfony. You can find the slides of the presentation here.

Day 1

On Thursday the day started with Fabien Potencier, the creator of Symfony, talking about Symfony 4 which will be released in November of this year. He talked about the changes from previous versions and dedicated a big portion of his talk doing a live demo of Symfony Flex, which aids in the installation and management of Symfony 4 projects. It’s worthy to note that Symfony Flex works already with Symfony 3.3 projects.

The next talk was also really interesting, where Nicolas Grekas from SensioLabs and of the top Symfony contributors, talked about the new features of the Dependency Injection Container shipped with Symfony 4 (and Symfony 3.4). Services are going to be private by default, helping keep the size of the container really small and improving the overall performance. Also, new hot features like aliases, auto-wire and auto-configuration are going to be very helpful for projects with lots of services, like our OpenSky codebase. We are really looking forward to start the migration path into Symfony 3.4.

The last session of the day was dedicated to lightning talks, where several attendees and speakers share talks limited to 7 minutes (controlled by timer). These are great talks that give the chance to people to present topics that normally would not have enough content for a full length session.

The day ended with a social event at The Armory Club in the Mission district where most of the attendees and speakers mingled and talked about all things Symfony. 

Day 2

The first talk of the second day of the conference was given by the one and only, Ryan Weaver from KnpUniversity with a great overview of Webpack Encore and how it can help take your frontend code in your Symfony apps to the next level. Ryan is one of the best speakers in the PHP world, and is always entertaining and great to watch. Not only that, but you are always guaranteed to learn new things.

Before lunch, our own Pablo Godel delivered his talk where he shared 40 minutes worth of things we have learned over the many years we have been using Symfony at OpenSky. The feedback from his talk was very positive and seems that everybody left with something useful for their own projects. 

Right after lunch, Pascal de Vink from The Netherlands talked about GraphQL, the new technology that is looking to replace REST for the API development dominance. One of the most interesting features for sure is the ability to define types in the data that your API will return.

The last full session of the day was about Kafka, presented by David Zuelke from Germany. David is a great speaker, with a deep and vast knowledge of the topics he presents. His talk was excellent and provided another view on how you can use Kafka as much more than just a message queue, thanks to its fault-tolerant and distributed approach to storage.

The conference ended with another set of lightning talks. One notable mention goes to the talk where PlantUML was demoed. This is a great tool to easily create diagrams of pretty much everything, including your PHP code.

Closing Notes

Altering Tables

In MySQL, append new columns to the end of the table to avoid locking the whole table. For example, avoid this:

ALTER TABLE users ADD COLUMN updatedAt DATETIME AFTER createdAt;

If createdAt is not at the end of the table, then the whole table must be locked in order to add the new column. Do this instead:

ALTER TABLE users ADD COLUMN updatedAt DATETIME;

You can also use pt-online-schema-change to alter tables without locking them.

pt-online-schema-change --alter "ADD COLUMN updatedAt DATETIME" D=database_name,t=users

Pagination

If you need to paginate deep in to a set of data, use an index and a where condition instead of using limit and offset. Limit and offset will get slower and slower the deeper you page in to a set of data. Here is an example script using limit and offset:

$perPage = 100; $page = 1; while (true) { $results = $em->createQueryBuilder() ->setMaxResults($perPage) ->setFirstResult(($page - 1) * $perPage) ->orderBy('id', 'DESC'); ->getQuery() ->execute(); if (empty($results)) { break; } // do something with $results $page++; }

And here is the same result using an index and a where condition:

$perPage = 100; $lastId = null; while (true) { $qb = $em->createQueryBuilder() ->orderBy('id', 'DESC'); if ($lastId !== null) { $qb ->andWhere('id setParameter('lastId', $lastId) ; } $results = $qb->getQuery()->execute(); if (empty($results)) { break; } // do something with $results $lastId = end($results)->getId(); }

Adding new Indexes

Add a new index to a production database without impacting performance for live traffic:

Demote and hide the primary node.

Add the index to the hidden node.

Promote the node back to primary.

Hide each secondary one at a time, add the index and unhide.

Avoid n+1

Avoid n+1 and fetch records in batches instead of 1 at a time. The following example fetches the cart for each user one at a time, resulting in 11 queries total to the database:

$users = $db->fetchAll('SELECT id FROM users LIMIT 10'); foreach ($users as $user) { $cart = $db->fetchOne('SELECT * FROM carts WHERE userId = :userId', [ 'userId' => $user['id'] ]); }

Here is an example where we fetch the users and their carts in two queries:

$users = $db->fetchAll('SELECT id FROM users LIMIT 10'); $userIds = array_map(function(array $user) { return $user['id']; }, $users); $carts = $db->fetchAll('SELECT * FROM carts WHERE userId IN (:userIds)', [ 'cartIds' => $cartIds, ]); $cartsByUserId = []; foreach ($carts as $cart) { $cartsByUserId[$cart['userId']] = $cart; } foreach ($users as $user) { $cart = $cartsByUserId[$user['id']]; }

Denormalization

Denormalize your data to avoid joins. Here is an example to demonstrate. Say you have a users table. Users can make comments and you want to keep track of how many comments a user has:

You fetch the user from the database like this:

$userId = 1; $user = $conn->fetchOne('SELECT * FROM users WHERE id = :userId', [ 'userId' => $userId, ]);

Now you insert some comments for the user:

$conn->executeQuery('INSERT INTO comments (userId, comment) VALUES (:userId, :comment)', [ 'userId' => $userId, 'comment' => 'This is a comment.' ]);

Now to get the number of comments a user has you could run this SELECT:

$numComments = $conn->fetchCol('SELECT COUNT(*) FROM comments WHERE userId = :userId', [ 'userId' => $user['id'], ]);

But if you simply increment an integer value on a column on the users table called numComments you can avoid the join or extra query:

$conn->executeQuery('UPDATE users SET numComments = numComments + 1 WHERE id = :userId', [ 'userId' => $userId, ]);

Removed Unused Indexes in MongoDB

Make sure you don't have unnecessary & unused or overlapping indexes. Every index you have is overhead for writes.

Find unused indexes in MongoDB:

use database_name; db.getCollectionNames().forEach(function (collectionName) { printjson(db[collectionName].aggregate( [ { $indexStats: { } } ] )._batch) })

Remove Unused Collections in MongoDB

Find unused collections in MongoDB using the following snippet:

use database_name; db.getCollectionNames().forEach(function (collectionName) { print ('collection ' + collectionName); stats1 = db[collectionName].stats().wiredTiger.cursor; total = stats1["next calls"] + stats1["update calls"] + stats1["insert calls"]; print (total) print ('------') })

SQL Style Guide

This tip provided by @treffynnon

Adopt a style and stick with it. Something like sqlstyle.guide, but any will do as consistency is key.

As we set out to build the next great social ecommerce platform we were struck with a rare and great opportunity to start with a clean slate. We were able to select each piece of our architecture without any predefined requirements. This let us review the state of the Frameworks in PHP and select the best one for our needs.

First we developed a set of requirements

Brace yourself, it’s a long list.

ORM

DB Obj relationships ie: :has_many

Good db migration mgmt

Good master slave config for db

Intelligent db trans support

Scaffolding

CRUD friendly

Business logic kept in model not controller

Controllers have ability to get heavy

No heavy loading of controllers

No race conditions for controllers (ie routing)

Template engine

Objects accessible in view

Widget/component support in view layer

Good header mgmt

Good mgmt of js and css files

All js and css be compressed

Ability to define explict routes or priorities not based on file system

Smart SEO urls by default

XSS & CSRF Protection

Good Form builders

Form Validations

No XML => YML or JSON

enforcement of coding stds

Convention and configuration

Ability to design by contract

Caching: Support for memcache and sessions

Configs cached not parsed

Cloud sensitive

Good role base ACL support (no global, system base)

Support for webservices

Easily loadable support for external libs

Profiler built in or easy to integrate (timers, number of queries, memory used, etc)

Localization (currency, dates etc)

CLI model access

CLI for testing

Unit testing

Actions are unit testable

Good file handling

Good exception handling – ie log4j

Support for multiple environments (dev, test, staging, prod)

Comparing Frameworks

Then we looked at any framework that someone on the team would be willing to evaluate. We ended up with 7 Frameworks to compare.

We only evaluated PHP specific frameworks as our team is primarily a PHP team and that is where our expertise is.

Symfony2

Symfony 1.4

Yii

Kohana

Zoop2

Lithium

Solar

Notice that Zend Framework, Cake, Code Ignitor or X Framework aren’t on the list. There were a good 10 or so popular frameworks that enough of our developers had used before and knew they didn’t meet our requirements, so we decided not to spend time evaluating them further.

Our Analysis

We performed this analysis May 2010. In the past 6 months both Symfony2 and Lithium which are under rapid development would score even better. It’s not my intention to keep this chart up to date as it’s a representation of historical analysis.

Further Analysis

At the end of our week of analysis we had a good overview of each Framework though there wasn’t a clear winner. What was apparent was that a “check” wasn’t equal to another check and some frameworks excelled in certain areas where other’s simply did enough to check the box. This made our analysis quite a bit harder.

Additionally there were some other factors that weighted heavily into our decision. Strength of community, Quality of development and Momentum are the three factors that played the biggest role in our decision making, but are not represented in this feature set analysis.

Ultimately we narrowed it down to 3 frameworks. Lithium, Symfony2 and Zoop2. All 3 were PHP 5.2+ and Next Generation Frameworks. Given the substantial changes accompanying PHP 5.3, we determined that either we went with that now, or had a painful migration in our future. We decided to bite the bullet now which left us to choose between Lithium and Symfony2.

Lithium vs Symfony2

This wasn’t an easy decision. Both are excellent frameworks developed by really smart people. Nate Abele, who started the Lithium project and was the lead developer of CakePHP, is’s a personal friend and an NYC local. He was the person who turned me, and consequently OpenSky, on to MongoDB. Nate is doing innovative things with Lithium and has a true passion for what his work.

Symfony2 is developed by Sensio Labs located primarily in France, on the other side of the world. At the time there were only 3 stateside Symfony team members. Symfony2 uses a bit more conservative approach to framework development, basing it largely on learnings from symfony 1 and established practices and patterns largely from the Java world.

Ultimately we went with Symfony2. It wasn’t a cut and dry decision and in spite of all our diligence still ended up as a bit of a gut call. Symfony boasted a larger developer base and community. This was probably the factor that ultimately pushed us in Symfony2′s direction.

Why Symfony2

Fast forward 9 months. OpenSky has embraced Symfony2 with open arms. We have become the 2nd largest contributors to Symfony2 behind Sensio Labs. We employ Kris Wallsmith and Jon Wage, two core Symfony2 team members. We’ve developed one of the first production applications using Symfony2 and the first ecommerce platform on MongoDB.

More on why you should use Symfony2.

Every class has an interface, every class can be typehinted as a method argument, typehint lets you specify the collaborator requirements. Not every typehint is the same… Example:

<?php class PaymentGateway { public function charge($amount, BankAccount $account) { // make a call to http web service here... $account->credit($amount) } }

<?php class BankAccount { public function debit($amount) { //... } public function credit($amount) { //... } }

The example above is fairly straightforward – we have a PaymentGateway class that requires a BankAccount collaborator for crediting amount to be charged in its charge function. Since we call credit() on the second argument – we typehint it to be of type BankAccount, which would prevent us from passing anything other than instances of that class. Let’s add some more requirements to our little example. Say we have checking and savings accounts, and we need to be able to charge both of them:

<?php class CheckingAccount extends BankAccount { } class SavingAccount extends BankAccount { }

Since both accounts are subclasses of BankAccount – the typehint still works fine. However, we have now opened the possibility for abusing the typehint… In case you didn’t know – classes in PHP don’t define a strict interface, and are open to change by subclasses:

<?php class SomeAccount extends BankAccount { public function credit($amount, $someOtherParameter) { //... } }

NOTE: if you have strict errors enabled, you would get one…

In the above example we changed the credit method signature by adding a second required parameter. We can now pass instances of SomeAccount class everywhere the BankAccount is expected and get nice fatal errors. Only interfaces and abstract classes/methods lock method signatures.

<?php interface BankAccountInterface { function credit($amount); function debit($amount); } class BankAccount implements BankAccountInterface { //... }

For our particular example, we could use an abstract class:

<?php abstract class BankAccount { abstract public function debit($amount); abstract public function credit($amount); }

Please note that in the example interface, I don’t specify method visibility. The reason is every method of the interface MUST be public and no explicit visibility means public in PHP.

<?php class SomeAccount extends BankAccount { public function credit($amount, $someOtherParameter) { //... } }

A class like the one above would cause a fatal error to be thrown: “Declaration of SomeAccount::credit() must be compatible with that of BankAccountInterface::credit()” or “Declaration of SomeAccount::credit() must be compatible with that of BankAccount::credit()”, depending on which approach you chose to use. If someone wanted to implement a BankAccount to be used with our PaymentGateway, they would have to keep the method signatures exactly the same as we defined them.

Hi everyone, in case you didn’t know – there was a ZendCon last Monday November 1st through Friday November 5th, 2010 in Santa Clara, California and most importantly – I got to attend and OpenSky sponsored my trip.

ZendCon is an annual conference organized by Zend the PHP company and held in California. Its known to be one of the biggest conferences in PHP world with attendees from all over the world varying from developers to top management, all using and interested in PHP. Zend makes PHP announcements – this year among other things, they announced that memory footprint of PHP had been reduced by 35% in the upcoming version – which are then followed up by different session from speakers from all over the world. These sessions vary in scope and context from detailed practical “how to”s of a framework or tool, to high level management advices and experiences.

As I already mentioned, this year I was thrilled to be among ZendCon attendees!

I arrived in Santa Clara on the evening of November 1st. This was my very first visit to west coast and I loved it. The first thing I saw when arrived at Santa Clara (after a 6 hour flight and a 2 hour ride on three trains – one of which had two levels) was a big Yahoo sign on one of the buildings which immediately felt like technology heaven.

During the next three days, I met interesting people from all over the world, including people I knew from twitter, but never met in person. I would like to especially thank Ryan Weaver and Pablo Godel, who are seasoned conference attendees, for making a great company and helping me get accustomed to the environment and people at the conference. I also met and spent a decent amount of time with people as: Jonathan Wage, Fabien Potencier, Nils Aderman, Dustin Whitle, Michelangelo van Dam, Jeremy Kendall and many many others.

I learned about Zend Uncon from Michelangelo during continental breakfast of the very first day of the main conference. Turns out that anyone could give a talk, even if it wasn’t included into the main session program. This is called the Uncon and there you can give a talk, do an interactive tutorial or round table on any topic you choose, all you need is to put your name in an empty slot in the Uncon schedule. Uncon sounded like a great idea and a good public speaking practice, so I registered… The session was amazing! Twitter was blown by quotations from my talk - tweeted, retweeted and re-retweeted. Attendees took pictures of me speaking and posted it on twitter too. Overall I was blown away by the success of my little experiment. As a result, my session was called the best Uncon session and I won a free lifetime license for Zend Studio 8 (they even wanted to call me up on stage during the final keynote, but I had a plane to catch that day).

Jonathan Wage, who joined OpenSky on the November 1st, 2010, suffered from poor organization on the conference part. He was unfortunate to get scheduled to arrive after his first session and leave before the second one. As a result, he gave only one talk of the two planned as part of the Uncon. His talk was, however, really amazing and well received by attendees and they wouldn’t let him go for a good 30 minutes after the talk was over, asking questions and expressing admiration.

Thank you ZendCon and see you again next year! Below are some pictures I took at the conference:

Unit testing is all about testing one piece of functionality in isolation. This is very easy when the subject of your test doesn't involve any dependencies. Take the following method:

Testing this method requires just an input and an expected output. This is a good case for using a PHPUnit dataProvider:

Unfortunately, it’s not always this easy. Let’s say we want to centralize this normalization logic in some sort of reusable Normalizer class, which we inject into the Tag class:

normalizer = $normalizer; } public function normalizeName($name) { return $this->normalizer->normalize($name); } }

The test above still passes, but we are now testing two things: the normalizeTag method and the Normalizer class; we are no longer testing our method in isolation.

Mock objects to the rescue!

To restore this isolation we create a “mock” version of the Normalizer class and tell that mock exactly what we expect will be done to it and how we want it to respond.

In PHPUnit 3.4, creating this mock object would look something like this:

getMock('Normalizer', array(), array(), '', false); $normalizer ->expects($this->once()) ->method('normalize') ->with($input) ->will($this->returnValue($expected)); $tag = new Tag($normalizer); $this->assertEquals($expected, $tag->normalizeName($input)); } }

Most of this is quite readable, thanks to Sebastian's nice mocking API. However, the initial getMock method is a bit gnarly. I've added a comment so you understand what's going on, which I'd rather not have to do.

In PHPUnit 3.5 we can use the new mock builder object, a contribution of Giorgio Sironi, which makes our test code that much more readable:

getMockBuilder('Normalizer') ->disableOriginalConstructor() ->getMock(); $normalizer ->expects($this->once()) ->method('normalize') ->with($input) ->will($this->returnValue($expected)); $tag = new Tag($normalizer); $this->assertEquals($expected, $tag->normalizeName($input)); } }

So there you go: unit tests, mocking, and the new mock builder.

People that RSVPed for the last Symfony NYC Meetup know, that it got canceled. We are very sorry, but the meetup speaker got sick and couldn’t make it. He did promise to give his talk on ESI caching as soon as he feels better.

Meanwhile, the meetup was moved to this coming Thursday, October 28th, 2010. The new topic is “Unit Testing. A guide to writing clean, testable code, that will be easy to maintain and extend”. This will be the repeat of the talk I gave to the OpenSky tech team a month ago. I will cover topics like unit-testing, dependency injection, mock objects and ways to refactor your code.

If you haven’t yet, please RSVP for the meetup and more importantly, come to hear me speak on the topic.

Edit: I uploaded the slides from that talk to SlideShare and embedded them below

Clean code

View more presentations from Bulat Shakirzyanov

Edit 2: We video-taped the full talk, which you can watch by clicking on the link below

At OpenSky, we concentrate on programmers’ skills and ways to improve them a lot. As part of that, we have Fridays dedicated to skill improvement – we, programmers, are allowed to write blog posts, prepare presentations¸ educate others on what we learned, refactor/improve our code or simply go through and add tests where they are missing.

Last Friday was very special. I finally took the time (the whole weekend before that) and put together some slides to give a long requested talk on unit-testing and the art of programming.

I wanted to cover as many aspects of testing and OOP in general as I possibly could, while preserving the structure of test-oriented talk, without concentrating too much on any single area.

I had no idea how to accomplish that when I started. However, the introduction section had given me a good understanding of how to relate all the different aspects, so it didn’t feel like I was jumping from topic to topic, not covering each in enough detail. I structured the talk around the properties of unit test – the properties we want to achieve in order to practice unit-testing. As a result of that, I was able to cover Dependency Injection (with the Law Of Demeter as a natural consequence), Mock Objects (leading me to Iterative Interface Discovery and Test Driven Development) and some general OOP techniques, that let programmers achieve much desired code clarity (Inheritance instead of Conditionals and Composition over Inheritance).

Ideas promoted in the talk were inspired by a number of different sources, and I feel I should list the references, since they were not included in the slides:

All definitions were taken from Wikipedia.org

The Clean Code Talks — Unit Testing by Misko Hevery (Google Inc.)

Mock Roles, Not Objects by Steve Freeman, Tim Mackinnon, Nat Pryce, Joe Walnes (ThoughtWorks UK)

The Clean Code Talks — Inheritance, Polymorphism, & Testing by Misko Hevery (Google Inc.)

The talk turned out to be pretty good and I received a lot of positive feedback on it from outside after posting it on slideshare. Even though I reserved 90 minutes to cover everything and answer all the questions, I was not able to do so, since we started later than planned. Therefore, I’m sharing the talk here and if anyone has questions or comments after watching the slides, feel free to post them here.

Some of you might, others might not know that our company, OpenSky, became the official sponsor of Symfony NYC Meetups, which means, that we’ll be seeing you the last Thursday of every month, here, at OpenSky HQ in NYC for the foreseeable future.

That was enough of the introduction, now to the point. Last Thursday, September 23rd, 2010, OpenSky held its first Symfony NYC Meetup. And, as anything OpenSky does, the Meetup turned out amazing. Our very own Kris Wallsmith gave an awesome talk, highlighting the differences between symfony 1.x and Symfony2. This was particularly useful, since my co-workers are the only people I know in NYC using Symfony2, and the majority of Meetup attendees are still in symfony 1.x world and hesitate to switch.

During the second, less structured part of the Meetup, we were enjoying pizza, that OpenSky kindly provided, discussing Symfony2 feature roadmap, current limitations of certain components and proceeded to other interesting, but hardly Symfony2 related topics like NodeJS, Android OS, Document-Oriented Databases (CouchDB and MongoDB) and PHP 5.3 Object Document Mappers (Doctrine MongoDB ODM and Doctrine CouchDB ODM).

In general it was a very interesting event and I feel sorry for those who couldn’t attend.

Since this is definitely not the last Meetup OpenSky is hosting, make sure to come in next time, to meet fellow symfony developers, share ideas and get to know OpenSky.

As a bonus, here is Kris’s video, where he talks about Symfony2.

The core symfony 2 team has just released Symfony 2 Preview Release 3. Of the 50+ different contributors, OpenSky provided 5 of them. Our developers contributed meaningful code commits which the core team approved and integrated with the release. We are excited to be a part of this thriving community and to be able to both use and contribute to such an excellent framework.

Here is a list of our team members and their commit counts towards this release:

39 Kris Wallsmith

16 Jeremy Mikola

6 Bulat Shakirzyanov (avalanche123)

2 Justin Hileman

1 Matt Fitzgerald (tiranog06)

http://www.symfony-project.org/blog/2010/09/13/symfony2-pr3-released-the-need-for-help

This article was written by Matthew Fitzgerald, Software Engineer, and Mike Zupan, Systems Engineer at OpenSky

The fix will go live next week (Sept 29)

One of the most important ways Shoppers find Sellers is through Search. Recently, we’ve found a hole in our search index. The problem we had was, Sellers would select Products with keywords like “beauty” in the Product Name but that Seller would not show up in the Search results for “beauty”.

The reason for this was our Seller search index only had direct information on the Seller, ie: Seller Username, Business Name, Bio, etc… The Seller index had no knowledge of Products the Seller was promoting. So any keywords the Product might have, wouldn’t return the seller in a search.

The Solution

We needed to modify our Seller index to have knowledge of both Products and Sellers and return the results of distinct sellers. The problem, our current Search technology (SOLR) didn’t support that behavior of Grouping Product Records by Seller and collapsing them into distinct Seller results.

The Tech Part

After some hefty Googling, I found that the feature to Group Search results does exist in a future version of the technology yet to be released. This feature became so popular that the developers of our Search technology released a “patch” to include this feature in the current version of the software. This is how I applied the patch.

Step 1

If you are like me, you hate Subversion and you love GIT. I run Ubuntu and using apt-get I installed a git subversion bridge.

sudo apt-get install git-svn

Step 2

Get some reading material, then start the checkout the latest stable tag of Solr

git-svn clone http://svn.apache.org/repos/asf/lucene/solr/tags/release-1.4.1

Step 3

Download and apply the patch

cd release-1.4.1 wget https://issues.apache.org/jira/secure/attachment/12448216/SOLR-236-1_4_1.patch patch -p 0 -i SOLR-236-1_4_1.patch

Step 4

Recompile it, Baby!

ant compile

Now you want to build the example project so we can test it

ant example

Now it is time to test. So go into the example project and the example data directory

cd example/exampledocs

Now you can import all the xml data

java -jar post.jar *.xml

Now start the server

cd .. java -jar start.jar

Now you should be able to hit the following url and get back grouped data

http://localhost:8983/solr/select? wt=json&indent=true&fl=id,name &q=solr+memory &group=true&group.field=manu_exact &collapse.field=manu_exact

Are you into PHP? This is how to implement it with the native PECL Driver.

$query = new SolrQuery('beauty'); $query->addParam('group','true'); $query->addParam('group.field','sellerId'); $query->addParam('collapse.field','sellerId'); $query->addParam('rows', 1000);

If you’ve ever used the Internet (how else are you reading this?), then you have used a search engine. Search powers everything from shopping to dating to research, and is so common that people use it without thinking about it… it is just natural.

Of course, when something is natural, you expect a comfortable experience and a good fit. You probably don’t wake up in the morning and put on someone else’s clothes; instead, you probably have a wardrobe of comfortable apparel that you piece together a look that feels good and feels right.

We don’t feel this is how our search experience at OpenSky has been. If you wanted to find products, you could get to them pretty well. But if you were a seller trying to find a great supplier to connect with, the experience was… lacking? discombobulated? Not cohesive, at any rate.

So we decided it was time to roll our own search experience rather than let our community search continue to be powered by [insert ginormous search engine]. Here is how we did it.

Search Technology

We use Apache Solr to power our custom searches. Solr is a search platform that utilizes the popular Lucene search library as its core, is blazing fast, and is open-source… hey, we like open source projects around here.

We decided to create a new Solr index for indexing and searching community data, rather than create a unified index for searching all data, the primary reasons being variance in relevant data fields and the ability to apply the unique schema to do more complex, targeted, and nifty search applications that we haven’t thought up yet.

There are certainly some benefits that can be gained from having a unified index, and we are currently cooking up some things which will make use of both unified and specific indexes, so we can make our search experience as useful and intuitive as possible. Expect more great things coming up as search continues to evolve.

Schema Considerations

Careful thought and consideration went into how we designed the schema. We are using the DisMax Query Parser for our general search, and one of the things we learned that hard way is that inconsistent field configuration options, such as stopwords or other tokenizer index / query filters, can seriously affect query results. Also, it pays off to tweak some default configuration options to your needs.

As a concrete example, the DisMax Query Parser defaults to requiring all search terms to be present in a field to generate a positive hit. Assume you have a document with title = “Essence of murtlap”, description = “Essence of murtlap is phenomenal!” Assume title is a “string” field with no tokenizer filters defined, while description is a “text” field with a stopwords configuration that prevents “of” from being tokenized, as it has high frequency and low importance.

In this case, a search for “Essence of murtlap” will not return any results. Why? Since “of” was not tokenized in the description field, 100% of the terms do not match, and so the entire document is a negative hit. However, searching for “Essence murtlap” would match. This may seem odd — after all, 100% of the terms DID match in the title field. For a mor thorough explanation, see one of several available resources.

I also decided to throw a tokenizer filter to remove leading and trailing punctuation. The inspiration for this was trying to match a word at the end of a sentence. Using the same example above, one would expect to be able to have a positive hit by searching for “phenomenal” — but one would be mistaken. Without some TLC the actual token stored is “phenomenal!” so, unless you include that punctuation in your search, you are going to be disappointed.

Fortunately, I found a very useful thread which presented a simple way to solve the problem using regular expression string filtering while indexing the document:

<filter pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2" />

Conclusion

If you decide to use Solr to facilitate your next searching project, don’t be afraid to get your hands dirty and play with the schema configuration. Solr has a lot of good documentation, is extremely powerful, and isn’t nearly as scary as it seems.

In the meantime, enjoy the new unified search experience and keep looking for more great things to come.

I’m Matt Fitzgerald and I’m a Software Engineer @ OS. OpenSky is a portal and as a portal, we injest a lot of content. As a result, we face a lot of problems around inconsistent user data. Often I’ll be tasked with cleaning up these issues after they pop up.

Recently, we discovered that our business entities (Sellers and Suppliers) had a lot of junk data in their OpenSky profile regarding information on their Facebook and Twitter accounts. We include links on the OpenSky platform that direct Customers to the Seller’s / Supplier’s profile page on Facebook and Twitter. Upon investigation, we found our Sellers / Suppliers were often confused when we asked them to save these identities into their OpenSky profile. As a result, a lot of Sellers / Suppliers, were entering incomplete or inconsistent data making these links confusing destinations for the Customer.

We went back and reexamined several parts of our application to see where the confusion began. The first pain point we identified was during our on-boarding process for Sellers / Suppliers. When Sellers / Suppliers entered their Facebook and Twitter accounts we simply required the input to be valid URLs that contained either “facebook.com” or “twitter.com” in them. Unfortunately for OpenSky, many of our users don’t really understand what a valid URL even means and as a result of requiring these valid pieces of data, we prevented a high percentage of users from successfully entering our community.

During testing, we found our users would enter a URL like www.facebook.com/The.OpenSky.Project and get rejected because only URLs with protocol (http://www.facebook.com/The.OpenSky.Project) are valid. In practice, the latter entry contained enough information for our application to correct the mistake, however, we simply stopped the user and advised them with error messages to correct the mistake. At first glance, this seemed reasonable, making the users enter valid data, however, on reflection we could sanitize a user’s data and keep them moving through our application with less impediments.

To make matters worse, other users understood their Facebook and Twitter Identities based on a variety of vanity urls, profile id’s, usernames, and SMS @reply tags. For instance, we asked people to enter a URL for their Twitter Identity like http://twitter.com/openskyproject and many only understood their Twitter Identity to be like @openskyproject. We even had problems with users entering links to their Facebook Photo Galleries rather than their main profile page.

We decided to remove these roadblocks by writing sanitizers that “smartly” extracted the relevant identity from all of the ways an identity can be known and saving only the relevant portion to our database.

Valid Facebook and Twitter Identities

Facebook Vanity URLs: http://www.facebook.com/The.OpenSky.Project

Facebook Profile IDs: http://www.facebook.com/profile.php?id=198143405897

Facebook Photo Gallery: http://www.facebook.com/profile.php?id=198143405897#!/The.OpenSky.Project?v=photos&ref=mf

Twitter SMS @reply: (@openskyproject)

Twitter User Name URL: http://twitter.com/openskyproject

Overview

The following solution involves PHP, Regular Expressions, and Symfony 2.0.

Sanitizing the Data

The first thing we had to do was write custom Validators in Symfony 2 to correctly extract a Facebook and Twitter Identity. Since there were many ways to build these identities, we had to search the data for the relevant portion and extract them. The second thing we did was pass these patterns to custom Form Fields. These form fields would first remove Facebook and Twitter domains, because these were actually not relevant to the validation process (The exact opposite of what we had first thought). So before the data was even processed, the value of the input field woudl be pre-formatted.

Facebook

class FacebookIdentity extends \Symfony\Components\Validator\Constraints\Regex { const GROUP = 'group\.php\?gid=\d+'; const PAGE = 'pages\/[a-zA-Z0-9_\-]+\/\d+'; const PROFILE = 'profile\.php\?id=\d+'; const USERNAME = '[a-zA-Z0-9\.]{1,50}'; ... some other Symfony 2 requirements here ... public function __construct($options = null) { parent::__construct($options); $this->pattern = sprintf('/^(%s|%s|%s|%s)$/', self::GROUP, self::PAGE, self::PROFILE, self::USERNAME); } ... some other Symfony 2 requirements down here ... } class FacebookIdentityField extends \Symfony\Components\Form\TextField { const STRIP = '/.*facebook\.com/'; protected function processData($data) { if (empty($data)) { return $data; } $strippedData = preg_replace(self::STRIP, '', $data); if (preg_match(sprintf('/%s/', FacebookIdentity::GROUP), $strippedData, $matches)) { return $matches[0]; } if (preg_match(sprintf('/%s/', FacebookIdentity::PAGE), $strippedData, $matches)) { return $matches[0]; } if (preg_match(sprintf('/%s/', FacebookIdentity::PROFILE), $strippedData, $matches)) { return $matches[0]; } if (preg_match(sprintf('/%s/', FacebookIdentity::USERNAME), $strippedData, $matches)) { return $matches[0]; } return $data; } }

Twitter

class TwitterIdentity extends \Symfony\Components\Validator\Constraints\Regex { const USERNAME = '[a-zA-Z0-9_\-]{1,13}'; public function __construct($options = null) { parent::__construct($options); $this->pattern = sprintf('/^%s$/', self::USERNAME); } ... some other Symfony 2 requirements down here ... } class TwitterIdentityField extends \Symfony\Components\Form\TextField { const STRIP = '/.*twitter\.com/'; protected function processData($data) { if (empty($data)) { return $data; } $strippedData = preg_replace(self::STRIP, '', $data); if (preg_match(sprintf('/%s/', TwitterIdentity::USERNAME), $strippedData, $matches)) { return $matches[0]; } return $data; } }