engineering @ openlabs

Last night as I was going through my git stash, when the mail icon on my Macbook started doing its tap dance suggesting the usual mundane result, there was an unread mail. The email was by my friend Jean from Copengo about something that I wanted to write since a long time. I quickly changed into my Star Wars t-shirt (sentimental reasons about the force) and started sabering my way through our internal workflow. For those of you, who are bored of technical jargon (I am one of them), a workflow is that which companies and individuals developing Tryton modules can adopt.

TL;DR; Here is the list of sabers that you will need to take down the Sith Lords (Stack):

Git - Version Control System

Git flow - A workflow for git

Github (Optional, but makes life easier) - Git Hosting Service

myOpenlabs for Project Management

Rietveld - For code reviews

Travis CI for Continuous Integration

Fabric for deployment

Newrelic and Sentry to make sure everything is up.

The abridged version of our workflow (The workings of Rebel Alliance)

Every module is a separate git repository owned by the Openlabs (A part of the Rebel Alliance) on Github.

Employees (and community contributors) fork the repository and send back pull requests.

The pull request is sent if the code passes the review, which is like winning against Master Yoda.

The pull requests are automatically tested by Travis, our own R2-D2, which tells us if the tests are successful.

If ‘all is well’ then the code gets merged.

Customer projects are deployed to staging/production servers and open source projects are released to the python package index.

Take a step back and repeat the same action with your light saber.

The workflow in detail

Just as I finished writing this post I realised that it had become way bigger than I had expected. So I have split it into multiple parts for easier reading and posting.

Part 1: Repository Management

Part 2: Project Management (using nereid-project)

Part 3: Code Review

Part 4: Pull Requests and Continuous Integration

Deploying Code like destroying the Death Star

It is important to deploy code easily to production and staging servers. While there are a host of configuration management tools, we prefer to keep it simple and pythonic using fabric. Most customer projects can be deployed with just `fab deploy_staging` and `fab deploy_production`. Documentation (using sphinx) is also automatically built and deployed to documentation servers with fab commands.

While the deploy_staging command checks out the develop branch of the code, the deploy_production command requires the exact version that should be installed. The version is identified from the tag used to mark releases.

The deployment commands are capable of upgrading the Tryton database too.

PS: Automate the most redundant task in your workflow…. Deploying code

Monitoring tryton (and nereid)

Monitoring the application is important to ensure that your application works for everyone (and not just for you) and monitoring the server they are hosted on is even more important to avoid downtimes and to debug if the armageddon (in this case failure of app to run) happens.

  Our preferred tool for application monitoring is Newrelic (Our very own Princess Leia) which also monitors our servers (for free). All exceptions are automatically captured from both Tryton daemon and nereid webapps and collected in sentry. 

Before sentry, we spent most of our time trying to reproduce mysterious errors users claimed to have encountered. Sentry keeps records of the stack traces and a sanitized context (values of variables), and client information making debugging a zephyr.

Proudly powered by travis CI

Pull requests are an integral part of our workflow and the trigger for Travis, our continuous integration engine. Every repository has a .travis.yml file which defines the steps to install the module and run the tests. Travis makes every build on a fresh virtual machine and the installation of the module is done from scratch.

In addition to running the tests, Travis scripts are usually configured to run flake8 and look for quality issues in the python code. Some repositories are configured to build on both sqlite and postgresql as testing on sqlite yields a lot of false positives.

Pull requests for older versions are obviously built with older versions of Tryton and dependencies. This relieves the package maintainer from the headache of having a mandatory older test environment and installing everything locally to test.

Why this process ? The ability to know if a pull request can be merged without manually pulling the code from the contributor and testing it is a significant time saver. Travis fits beautifully into our test driven development workflow.

See the screenshot below from the travis blog:

Github pull requests have support for lightweight reviews, but we still continue to use Rietveld (hosted on google app engine) for our reviews.

Here is a history lesson in technology, Rietveld is a web-based collaborative code review tool for Subversion written by Guido van Rossum to run on Google's cloud service.

Rietveld is a django app and could be hosted anywhere, but deploying on app engine is cheap and it works better for us because the logins are integrated with google apps (which we use for emails and chat within Openlabs).

Project Management  (The Home of the Jedi Knights)

Project management at Openlabs is done using myOpenlabs which is powered by Tryton and Nereid (They really like Jupiter). myOpenlabs is a suite of nereid applications which includes project management (powered by nereid-project, which we have open sourced) and account management for our customers to see their account position, download invoices and make payments.

A project management system helps reduce the coordination cost and is crucial when you have developers and customers across multiple timezones. (It is the wife/girlfriend you never had) The system helps keep the information transparent across the customers, developers and consultants.

The system is also tightly integrated with other tools like github. Commit messages which mention task id in the format #1234 are automatically linked to the task and displayed as an update.

Time Tracking

It goes without saying how critical tracking time spent on tasks is. Nereid project makes it super easy to mark time on tasks. Openlabs customers love this transparency into how their project is progressing on a task by task basis.

The central ‘official’ repository

The notion of a central official repository is just a convention. From a technical standpoint Git does not make a difference between each developer’s repository and the official one.

Fork the ‘official’ repository

As a thumb rule and tribute to nerds around the globe, nobody pushes code to the main repository. Everyone has to fork the main repository (in our case owned by Openlabs) and send the patches to the develop branch.

Branching for each Tryton Series

Once a new major version of Tryton (Yeah, they liked Jupiter) is made available (like 2.8, 3.0) the current code in develop is checked out to a branch with the name of the Tryton major version (3.0) and the develop branch is updated to the latest development version. (Think of it as a tree, the code is the fruit on the branches)

Minor Releases

Every minor release is tagged with the git tag functionality. The description of the tag is used as a CHANGELOG which Github renders seamlessly.

Customer Projects

Source code for customer projects (unlike Tryton modules) undergo significant changes within the same major version. In addition, customer projects always have a separate staging server where we deploy code for the customer to test and give us feedback. This is where gitflow shows its might like Han Solo coming to rescue of Princess Leia:

The staging server always has the code from the develop branch.

Once a set of features are completed (on develop branch) and approved by the customer, a major version of the project is released.

Sometimes, a critical feature or a bug might need to be fixed without waiting for the next version. In such cases, the (eye)patch is developed on develop branch,deployed to staging and on approval from customer it is cherry-picked into a hotfix release which includes this change alone and not other changes in the develop branch.

The Gatekeeper of our process (We call him C-3PO)

Though, we use a distributed version control system, the merging into the central repository is managed by a human-droid we call the C-3PO. C-3PO is also the package maintainer and is responsible for:

Merging pull requests

Making branches when new versions of tryton are out

Release management

You can read more about the forking workflow and how the gatekeeper works from the atlassian blog.

We have set up a github mirror for the tryton repositories.  The mirror is updated once every day between 01:00 UTC and 02:00 UTC.  The mirror uses the branching capability of Git to keep the different versions of the code (tryton's repository hosting has different repositories for different versions).

We just upgraded the core nereid module (the web framework) to Tryton 2.6 which uses the Active Record design pattern. The migration was easy since we had the wonderful guidelines written by Cedric Krier. Just as how the guidelines help in migrating tryton modules to 2.6, I hope this post helps in migrating nereid modules to 2.6.

1. Restful URLs

The 'methods' selection field was due for deprecation and has been removed. If you are migrating to 2.6.x from any version after 2.4.0.6 you should not face any issues with migration as the selection field is automatically migrated into the new format.

https://gist.github.com/4587773

Remember that prior to this version, the GET HTTP method was allowed by default (implicit), but the new design disallows all HTTP methods by default. So to achieve the same behavior as 2.4, you will have to explicitly enable the GET method.

https://gist.github.com/4587930

2. Active record pattern: Tryton 2.6 introduces the active record pattern and the model controllers which work on more than one record are class or static methods while the methods which act on a single record happen to be regular instance methods. The convention is extended to nereid too: If the handler works outside the context of a single active record, use static or class methods depending on whether your method delegates/uses other methods in the class. If not stick to an instance method. When you use an instance method, tryton expects the instance to be the active record itself and nereid allows you to do this again by convention. Just ensure that your url has an integer converter which generates a keyword argument 'active_id'. Example:

https://gist.github.com/4588022

Happy Hacking !!

The Tryton unconference at Liege was a lot of fun and one of the most exciting outcomes of it was the community's request for Nereid - the web framework for Tryton built at Openlabs to be made a part of the core tryton modules.

As a first step into the process, Cedric Krier made a review of Nereid in the context of its inclusion and made his comments on the mailing list. We have translated his comments into issues on github, and we are getting them done one by one.

A mail from Zodman in the tryton mailing list reminded me of a feature we had built for one of our largest customers - A module to export large amount of information in CSV for data interchange between Tryton and legacy systems. The goals of the modules were:

Export large amounts of data without eating all the memory.

Keep it simple and stupid (KISS)

API to add new reports in custom modules.

We have extracted the part which does this reporting to a separate module which is now available on github.

There's not much documentation out there and since we don't have time at the moment to write them, your best bet is a doc string in the module and this gist with a sample report, which exports tracking number for each order line.

The module also includes a Unicode CSV Dict writer, to save you from the fires of Unicode encoding/decoding dance (common in python 2.X).

Internationalisation (i18n) and localisation (l10n) are not features anymore, but very requirements of any modern web application. At Openlabs we love tornado :) and locale support in tornado needed some more love.... and we have given it babel!

Tornado-Babel adds the missing components for l10n and i18n in tornado.

Extract translations from your tornado templates using pybabel (like django and jinja)

Translate using gettext files (with all your favorite tools)

Use the powerful, date, time, number and currency formatting toolkit of babel.

Head over to the documentation now and start localizing your apps.

Happy hacking!

Welcome to the Openlabs Engineering blog.

We build software for small and medium enterprises around the globe. In doing this, we continually face challenges, and we successfully engineer our way through them by building new tools and making different softwares talk. 

We have started this blog to talk about tools, techniques and architectures we develop at Openlabs, which we believe would be fun to share. Being an open source company a lot of the work we do is already published under our github account. We hope to write more about them here.

You can expect to read posts on Python, OpenERP, Tryton, Tornado, Flask, Nereid and other tools we work everyday with.