Gemnasium

Gemnasium has been around for a few years now, and we’re thankful to have such a huge impact on your projects, private or open source, and the community as a whole.

Over the years, we have helped over 500 000 projects ensure that their dependencies are up to date and are not affected by known security issues on their dependencies. You read that right: 500 000. This gave us an incredible chance to talk with a lot of you and understand better your needs.

As we spoke with our users, it became obvious there were a lot of features requests that pointed us to an enterprise version of Gemnasium. Things like supporting GitHub Enterprise, GitLab, Bitbucket Server or supporting private dependencies hosted on your infrastructure. There is also a lot of people that would prefer us not to see their code ever for security reasons, and that’s fair.

Here comes Gemnasium Enterprise to the rescue

We worked hard toward that direction for the last year and started the beta recently with a few waves of businesses that asked for early access.

We’re happy to say today that we’re ready to go to the next step. Everyone who is interested in Gemnasium Enterprise can join the private beta!

Gemnasium Enterprise features

So, what are the key features that make Gemnasium Enterprise differ from Gemnasium.com?

It is installed on your servers and stays on your local network. We do NOT have access to your code.

Support for GitHub Enterprise

Support for GitLab (hosted on-premise and gitlab.com)

Support for Bitbucket Server is coming in the next few weeks

Support for private dependency registry coming in Q1 (we will support private registries like Gemfury and Gem in a Box for instance)

Easy setup with Docker that can be done in a few minutes (require a different way to install Gemnasium Enterprise? Get in touch and we’ll figure something out for you.)

Like the version on Gemnasium.com, the enterprise version also supports those key features: - Email or Slack alerts if there are security advisories affecting one of your dependency or one of your dependencies’ dependency - We can send you digests of the changes in your packages across your projects or for one project - We support Ruby's RubyGems, Javascript’s npm & Bower, Python's PyPi and PHP’s Packagist packages. Java/Maven coming in Q1.

Getting started with the beta

So you’re interested in learning more or get started with he beta? You can either just email us or fill the form on the enterprise page and we’ll get you started shortly!

We’ve covered a lot of ground in this series of articles, and are now on the final chapter. We’ve looked at the reasons we decided to make the move from a monolithic app to a micro-services approach, the criteria we used to select our toolset, and how we implemented the transition. Finally, we want to share with you how we use our new architecture, and share some of the code we have been writing.

Framework

Whether using a monolithic or micro-services approach, a decision must be reached on whether to employ a framework. There are hundreds of frameworks available, and every week or so, a new one is added. Even in Go, there are plenty of frameworks available. If deciding on a framework, or on whether or not to use one, you should be aware of their uses and benefits. Frameworks are mostly needed for discovering new services, and managing new and existing services (start, stop, etc.). They sometimes offer even more, including:

Circuit breakers

Rate Limiters

Logging

Metrics

Load balancing

Pubsub

etc.

Despite these benefits, using a framework means you need to include its code within your app, and for the micro-services approach, that can be a lot of overhead, and a big dependency to add. A great deal of code is dedicated to the framework configuration and events management.

For us, this overhead was an unacceptable burden, even with the benefits a framework could potentially provide. We prefer to keep it simple, and stay with vanilla Go. This allows us to separate the “infrastructure” code from our business logic code. Every service remains independent, and cannot be started without a registry service, or any other service. If we need to test communication between services, we use docker-compose files to run the minimal images. No frameworks for us!

Consuming Microservices

A service is often not self-sufficient, and needs to communicate with its peers to function. Because we don’t use a specific framework, we stick to plain old http communication, and every service is actually a REST API. This allows us to use curl for debugging. For the moment, we are keeping things relatively simple, but our no framework approach allows for a great deal of flexibility for later optimization.

Because of the flexibility of our approach, we will be able to optimize later, including solutions such as GRPC, which is based on Protocol Buffers (another opensource jewel from Google.) GRPC establishes a contract between services in order to serialize data. This has the advantage of fixing API specs with an IDL. It looks something like this:

message HelloRequest { string greeting = 1; } message HelloResponse { string reply = 1; } service HelloService { rpc SayHello(HelloRequest) returns (HelloResponse); }

This will be compiled, and the code generated will be included in the project. If the specs are evolving, the compiler will make sure nothing is broken.This has many advantages, including a big performance boost, because data is not serialized to standard JSON, but into a binary optimized format.

The ability to leverage such improvements in later implementations was a key factor in deciding our approach, however, for the moment, it is more convenient to use json directly.

Since Go v1.6, HTTP/2 has been part of the standard librarly for both the client and the server, allowing us to benefit from this performance boost in production, while staying with classic HTTP/1.1 on dev, for debugging. Building an HTTP/2 server in GO is as simple as:

import ( "log" "net/http" ) func handler(w http.ResponseWriter, req *http.Request) { w.Header().Set("Content-Type", "text/plain") w.Write([]byte("This is an example server.\n")) } func main() { http.HandleFunc("/", handler) log.Printf("About to listen on 10443. Go to https://127.0.0.1:10443/") err := http.ListenAndServeTLS(":10443", "cert.pem", "key.pem", nil) log.Fatal(err) }

That’s right, it’s the same code as a “classic” HTTP server using HTTP/1.1.

Configuration

Our configuration is kept as simple as possible as well. We follow 12factors methodology for our services, and all configuration is passed using environmental variables. That’s it, nothing else.

How do we set all the variables? The network info (addr, ports) are passed automatically by openshift to each container. Because of this, there is no need for a central registry.

To illustrate this, here is our postgresql library:

package pgcli import ( "fmt" "os" "strings" "github.com/gemnasium/secrets" log "github.com/Sirupsen/logrus" "github.com/cenkalti/backoff" "github.com/urfave/cli" "github.com/jmoiron/sqlx" _ "github.com/lib/pq" ) const ( Host = "postgresql-host" Port = "postgresql-port" Password = "postgresql-password" User = "postgresql-user" DB = "postgresql-dbname" SSL = "postgresql-sslmode" ) func init() { // Set credentials from mounted volume secrets.Load("/etc/secrets/postgresql") } func MakeFlags() []cli.Flag { return []cli.Flag{ cli.StringFlag{ Name: Host + ", pg-host, ph", Value: "postgresql", // service name Usage: "PostgreSQL host (use '/tmp' for unix socket on mac os, or '/var/run/postgresql' on linux)", EnvVar: "POSTGRESQL_PORT_5432_TCP_ADDR,POSTGRESQL_SERVICE_HOST", }, cli.StringFlag{ Name: Port + ", pg-port, pp", Value: "5432", Usage: "PostgreSQL port", EnvVar: "POSTGRESQL_PORT_5432_TCP_PORT,POSTGRESQL_SERVICE_PORT", }, cli.StringFlag{ Name: User + ", pg-user, pu", Value: "postgres", Usage: "PostgreSQL user", EnvVar: "POSTGRESQL_USER", }, cli.StringFlag{ Name: Password + ", pg-pw, pw", Value: "", Usage: "PostgreSQL Password", EnvVar: "POSTGRESQL_PASSWORD", }, cli.StringFlag{ Name: DB + ", pg-db, pd", Value: "postgres", Usage: "PostgreSQL database", EnvVar: "POSTGRESQL_DATABASE", }, cli.StringFlag{ Name: SSL + ", pg-ssl, ps", Value: "disable", Usage: "PostgreSQL sslmode", EnvVar: "POSTGRESQL_SSL", }, } } func Connect(c *cli.Context) *sqlx.DB { conStr := ConnectionString(c) log.Debugf("Connecting to db: '%s'", connectionStringWithPasswordRedacted(c)) db, err := sqlx.Connect("postgres", conStr) if err != nil { log.Fatalf("Can't connect to postgresql, please check your connection url. (err: '%s')", err) } return db } [...]

Every service is a command line tool, and a web server (default). The flags are added to a CLI (command line interface) command. Some options are passed in the CMD of the container.Creating microservices does not mean we need to isolate and duplicate code. We are able to take common libraries such as PGCLI, and share them amongst all services. As we are now using continuous deployment, any change to this library will trigger a new build of dependent projects (microservices), and deploy them right away.

Values are populated by a ‘secret’, which is basically a read-only configuration file shared by several containers. Credentials don’t appear in the container inspect info. Secret volumes are shared between services, like SMTP config. Each service has its own credentials, and is given a level of accreditation that allows it to access the database. Environmental variables like ‘POSTGRESQL_PORT_5432_TCP_PORT’ or ‘POSTGRESQL_PORT_5432_TCP_ADDR’ are populated for each container by Openshift.

This means we don’t have to manage the configuration of our service at all. As soon as we define a “postgresql” service (which is basically the IP of a LoadBalancer in front of our DBs), all containers will receive these variables automatically! As with docker-compose 2, containers share the same network, and a hostname named “postgresql” is now available. That’s what we love about Openshift and kubernetes; everything seems so simple once it is put in place.

Tests

Much like any other project, each microservice has unit tests. These tests can be more complex to achieve, since the microservices approach means more processes to start (in the right order) to go through all the layers. Because of this, and that Gemnasium.com and the upcoming Gemnasium Enterprise are now sharing a great deal of code and features, we use the latter for end-to-end tests. Because we use plain Go (mostly HTTP client/server), it is relatively easy to test new services.

Scaling

Our no framework approach applies to scaling as well. Because our microservices rely on centralized services such as postgresql and NSQ, we are able to run a large number of them concurrently, without any problems. As long as we do not write files in the common shared file system (which we do not), our system hums along smoothly.

As we mentioned, our containers are accessible through services, which act as load balancers. For example, the badges service we outlined in our previous article routes and load balanced traffic on all containers running the badges server. Even with a hundred running simultaneously, nothing would change in our code (and the performance hit would be negligible as well!).

Monitoring and Alerting

The more you do with any app, whether monolithic or microservices, the more metrics will be needed when debugging, or even just monitoring. There are numerous things that are important to keep track of, number of signups, failures during project sync, notifications (by channel), and more. Gemnasium uses Graylog for this, because we need a tool that resides in our network, and keeps our logs secure within the network as well (logs should never be sent elsewhere). Graylog is an opensource and free log aggregator with tons of features (including dashboards, alerts, search graphs, and more).

Graylog tracks alerts for us based on number of occurrences. Airbrake alerts us of the first error, in order to allow us to respond. We have set up alerts and notification in Graylog based on triggers.

Embrace failure

Lastly, as developers, our approach has to account for the possibility of failure, and deal with it appropriately. Even if kubernetes is smart enough to avoid sending traffic to an invalid pod, things can quickly become unstable or unavailable. To us, accounting for failure means not only being reactive and aware, but actually embracing failure as part of the roadmap to success. We need to anticipate failure in every workflow, particularly:

Idempotent tasks (for example: do not insert twice if runs twice, or do not send the same email twice, etc)

Check writes (DB, NSQ, etc.)

Retries 

Hot retry (inside the running instance, with exp. backoff) 

Cold retry (instance can be killed, and the backoff with it, allowing a retry of every task that should be finished already)

Summary

Our overarching advice: Keep it simple at the beginning, and optimize further down the line, if necessary. This advice serves equally well for any project. Introducing complexities too early in the project’s lifeline merely compounds difficulties later on. Ensure any complexities added are well-considered, and add significantly to the end product - a framework’s benefits must outweigh the overhead and dependencies incurred, for example. Even in planning a large shift such as ours from monolithic to micro-services, the question should always be - “does this make our transition easier, or can I see enough benefits to warrant any difficulties?”. You will recall that our decision to select Openshift was based on exactly this principle. It provided a strong framework, and good documentation, allowing us to implement our first services successfully the first time around. Our decisions were based on what was before us, rather than what might be available in later versions of other products. Stick to what works, and innovate based on a firm foundation.

As we sign off on our final instalment of this blog series, we hope that our experiences in moving from a monolithic application to a microservices approach proves helpful to your organization, not only if you are preparing for a similar shift, but also if you are determining if such an approach is right for you.

Genesis: Making the Transition

For those who have been following this blog series, you’ll know of Gemnasium founder Philippe Lafoucrière’s visit to the WAQ conference in Quebec city, and his stint as guest speaker with Jean-Philippe Boily of Metrics Watch. This conversation provided the impetus for this blog series, which covers Gemnasium’s shift from a monolithic app to a micro-services approach. For those unfamiliar with the topics at hand, I recommend a review of the previous two entries.

Having outlined the reasons for the transition, and the criteria we used in the selection of our toolset, it is now time to cover how we made the transition.

Choosing a Starting Point

Now that we had our toolset, OpenShift, it was time to decide upon process, and what to move first. This was not an easy decision, as micro-services architecture was completely new to us. We wanted a quick win in order to validate our choices made thus far, to give us confidence in our selected path, and to work out our processes going forward. We also wanted to be sure that nothing would be broken by our efforts, so we had to select a non-critical feature to start. We did, however, want to begin with something that was identifiable as distinctly Gemnasium.

Once we considered these factors, (a signature service that was not critical to the app) the answer as to what would be moved first became obvious: badges. Since the beginning of the project, Gemnasium had provided badges to our users, indicating in a small and portable image the status of a project. These badges were typically inserted into project ReadMe files, and thus each hit on the project page was a hit on our stack. Thanks to shields.io, these badges have now become pretty standard, and their appearance has been unified across heterogeneous services, such as Travis CI, Coveralls, Jenkins, and others.

Example:

https://gemnasium.com/mathiasbynens/breach_core.svg

This feature was small, relatively simple, and had clear code boundaries within our monolithic rails app. Badges rendering was done in Ruby-on-Rails, within the main app, and while it wasn’t slow, it certainly wasn’t as fast as it could be (a ~30 to 50ms / hit, when cache was invalidated). If something went wrong with the badges server, we could simply remove the routing to the micro-service within seconds, and be back to the original state of the app. These factors made badges the perfect test case for our new approach.

Modus Operandi

Having chosen our test service for breakout, we should take the time to explain the original app configuration, and the changes we ended up making. As a ‘monolithic’ app, this was the (very simplified) original configuration:

As you can see, we provided load balancing and redundancy within the application, but it was a self-contained single entity. Our database servers were mirrored, ensuring a full copy of our data was always available. Two load-balanced front end servers provided our services, with optimum load decided by our load-balancer. It was a simple, elegant solution, very suitable for a smaller application. Unfortunately, growth happens, and sometimes the answer is not simply to throw more resources at the problem. But we already covered the reasons for the change in our previous article.

Step 1:

To begin our migration to the new architecture, we had to first stand up our Openshift cluster, and connect it through our firewall to our stand-alone app. The red wires represent the flow of web traffic through our application. Once communication was opened between the Openshift cluster and our application servers, we were ready to make our first transition.

Next, we had to route traffic through the new Openshift cluster. This helped us to validate that Openshift could work as our new front-end, and handle the traffic our application generated, as well as ensuring there would be little to no downtime in the transition. This included connecting the databases to the new cluster, and testing that the data was accessible through Openshift. Both clusters had to operate in tandem, or the badge services could not be safely be removed/replaced without disruption.

Now we could create badges as a micro-service, and add it to the Openshift cluster. After just a few days of learning go, we had badges working as a micro-service. 

The results were impressive; not only was the rendering fast, but the resource consumption was remarkably low.

The time to render a badge (both for .png or .svg files) was so fast that we still haven’t had any need to implement a cache around this service. Rather than a 10 to 50 ms hit, it was reduced to between 1 and 5 ms. The low resource consumption (as evidenced in the screenshot above) allowed us to run 2 containers in a load-balancing configuration for redundancy, just in case a node was failing in our new cluster. The impact on the main app was also immediately visible. We had less requests to handle, and our CPU had more resources to spend on other features/pages.

Continuous Effort, Continuous Deployment

With our first service moved, and already seeing results, we knew we had made the right choice in toolset. Moving to OpenShift from our more ‘conventional’ architecture was far from a slam-dunk. We needed to put every component into a docker image, and run it as a container. It took a great deal of time to do so, but after a few months, our new architecture was ready.

The moment had arrived, and it was time to deploy our statically compiled Go program. We’ll explain in part 4 how we went from this executable to something running in our cluster. We now had our own Continuous Delivery platform, without a single line of code leaving our infrastructure, which was a big win. OpenShift proved smart enough to deploy only running instances of our services, and only when required. We added some probes in the badges service, so that traffic would be routed to it if (and only if) the health status was green. Updates are using a smart rolling strategy, where new containers are started, checked, and add to our LB (again if status is green), before starting to remove old versions:

This meant not only secure deploys, but also absolute zero-downtime deploys. As we were producing new versions several times a day, this was a strong requirement.

Since implementation, all changes approved in master branches (because we now have more than one project, due to ongoing efforts to deliver Gemnasium Enterprise), are deployed within a few seconds directly to production, saving us a great deal of time. Having every component in docker would make shipping to our clients easier and safer.

Criteria for Architecture

In this segment of our larger piece, we will cover the criteria we used to create and plan our architecture. Moving from a monolithic app to microservices isn't just a matter of code, it is also and above all a question of architecture. Without a cohesive plan, and a strong understanding of the potential pitfalls and gains of a large transition such as this one, it is bound to falter. As you (may have) read in our previous article, this project had us move from classic servers on VMs to something completely new to us. Every decision could change a multitude of later processes, without any assurance that it was the right path. For that reason, we made sure to create and provide a full specifications document, and shared it with the team to gather a large amount of feedback, and gain support and buy-in for our plan. Sharing this document also provided the team with the big picture view, and allowed them to influence key decisions.

Within the full specifications document shared with our team, the following key points were covered:

Security

As Gemnasium is an application responsible for delivering security advisories, it was vitally important that no matter what path we chose, that the security of our application remained a top consideration. After all, who would trust the advisories provided by a solution that was not itself secure? With more than 500K projects, we have access to a lot of accounts, tokens, and data. Because of this, we take security very seriously.

This consideration was tested early, when we considered the first batch of new tools. The first tools considered were Docker and its siblings, such as docker-compose, swarm, etc. While the Docker registry was freely available and opensource, there was no way to authenticate users at this point. We couldn’t yet see a way with these solutions to secure access to the registry, or to provide for different roles (readonly, readwrite, etc.). Another drawback in those we tested did not have anything that would isolate projects completely from each other, or even isolate the components themselves. For these reasons, we couldn’t justify a shift to these tools.

When we discovered Openshift, on the other hand, we found a solution in which the registry was only available with proper authentication. It provided compartmentalization, strong authentication/authorization, and still had the flexibility we needed for building enterprise grade architecture.

Deployment

A simple, process-driven and easy deployment was important to us. We had been using Capistrano for years. We also loved heroku-like deployments, where when code was pushed, a specific remote would push and deploy the app. We wanted something similar, simple and neat. However, we had a few other considerations that forced us to make a move.

First of all, our new architecture had to support rollbacks, both manual and automatic, like we did with Capistrano. This change alone involved a lot of changes to our processes, so we needed a solution that wouldn’t add to the workload. Unfortunately, while most of the solutions we tested did provide at least a partial response to our criteria, or were close to meeting them, the gaps were significant enough that they required creating and maintaining a lot of in-house scripts. This meant more work for us, and no one wants that if they can avoid it.

Once again Openshift proved the solution (for our requirements). The health check of deployed containers practically guaranteed clean deployments. If something was failing, the previous version would still be in place. Their deployment strategies allowed us full control over how the new Docker containers were deployed. For example, we could set the “Rolling” strategy to have two new containers up, with at least two running at once. Best of all, like Capistrano, it offered zero-downtime deploys, which in turn ensured smooth database migration. Because we could keep at least two versions of the codebase running at once, it was impossible for us to introduce breaking changes.

Scaling

Scalability was another strong consideration. As anyone in IT knows, while VMs can be easily duplicated, but it is by far not a point and click process preparing and updating VMs, or removing old architecture, as would be required for our new approach. We needed a solution that provided more than just vertical scaling (adding more resources and containers). The vertical scaling approach has a ceiling (the node’s available resources), and was not resistant to a node failure. With only one instance running, and all resources tied to the overtaxed node, nothing was available to restart the node elsewhere.

We wanted something simple, but with the ability to scale with a few operations if needed. We were shifting our processes to micro-services, but there were some pieces that would still require a bit of heft. In what is likely sounding fairly repetitive at this point, Openshift again had everything our organization needed (they’re not paying us, we swear). A few clicks enabled us to scale containers up or down as needed. As we exploded our giant app into a multitude of smaller pieces, we could scale exactly to the requirements of that fragment, without scaling the whole app. On top of this, the underlying kubernetes layer, responsible for orchestrating all these containers, also provided real-time monitoring, ensuring that the exact desired number of replica were running.

Logging

If you’re making any kind of wholesale changes to your app, knowing exactly what is going on, or at least being able to dig into the problem at a moment’s notice, is vital to your progress. Checking the logs, whether to confirm your suspicion, or to dig into the bowels of your system, is a go to staple for any troubleshooting coder. Unfortunately, prior to 2015, and the introduction of Docker logging drivers, nothing was available that would send these logs directly to a central location. Aggregation of these logs was a requirement for us, because no-one wants to hunt for the logs, only to then have to hunt through the logs for the issue. One hunt at a time, folks.

Our solution was simple, if a little old school. If details were present in environmental variables to send logs to a specific location, we mirrored those instructions and sent them to the centralized location as well. With the help of a “log” service (http://kubernetes.io/docs/user-guide/services), it was relatively easy to configure this for all of our components at once.

Backups

No solution can be considered ready for go-live if a backup solution is not in place, and well tested. More importantly, above and beyond backup, a disaster recovery plan should be in place. When a critical failure occurs, whatever the cause, you need to be able to restore things to status quo in a very short time-frame. There is no time to figure out where the required data is backed up, where it should be restored, and what to do next. One challenge of the micro-services approach is that if it is the failure of a single (but key) service, finding the back-up files specific to that piece might be problematic, if not adequately planned. In particular when many of the services tested did not provide centralized data stores. Additionally, no matter which solution we looked at, the question of what to do next in a recovery situation was not really covered at all in the documentation.

In Kubernetes (hence Openshift), all persistent data is available through persistent volumes. This mechanism is automated, well-documented, not to mention very complete. Many volume types and shares are supported, including NFS, GlusterFS, and even Ceph. With all the persistent data in one place, it is extremely easy to backup, restore, and monitor. What to do next is also simplified, reduced to scaling down replicas to zero, restoring the data, then scaling back up again to the original number of replicas. Even if it is not spelled out, planning your recovery is made simpler simply by the features provided.

Monitoring

Another key consideration when choosing your architecture is monitoring. It is a bit of a challenge to monitor microservices, because it is difficult to predict where and when a component/service will be running. In most cases, all you need to do is check the load balancer for expected responses from the services behind it. However, this won’t tell you how many instances of a given service are running. If it is only one instance, your app could be circling the proverbial drain, awaiting that last minuscule push/pull to edge into disaster (or at least downtime) territory. Good monitoring is pro-active, and should let you know of such impending problems. Ideally, it should be able to trigger automated responses, such as restarting services if they stop, and notifying an admin when this happens.

In our case, metrics are largely provided by the app directly, and fed into a centralized group of servers dedicated to that purpose.

Of the solutions we looked at, only Openshift provided key metrics and monitoring out-of-the-box. Openshift automatically provides hawkular metrics, and made graphs available directly to the console. While this provided a convenient and attractive dashboard, the metrics provided did not go beyond CPU and RAM consumption. For us, this was not enough, so we came up with our own innovations, which we will cover in part 4 of this serial blog. 

Documentation

You can have the most brilliant, innovative solution in the world, but if you don’t tell people how to use it, it will fail. Adequate documentation is where nearly all of the solutions we tested failed. Granted, many were in the early stages of development, and now probably have more complete documentation, but at the time we were testing these solutions, every problem encountered resulted in a great deal of wasted time. As is often the case with opensource projects, solutions are very clear for the authors of the project, but until there is a larger community surrounding the project, the holes in the provided information are simply not filled in. If planning to shift to a new methodology and a new platform, solid documentation is vital. Hands down, Openshift won this round. RedHat provided dedicated teams to create online documentation. Even in the beta stages, we had not seen a solution with documentation as precise and complete as this. Even better, it was constantly evolving, with new details being added in tandem with each version (a monthly basis approx). This alone might have made our decision for us, even if a couple of the other pieces had been missing, simply because with adequate documentation you know what is missing or incompatible, what is on the roadmap, and can plan accordingly.

Stability

We spent a great deal of time evaluating potential solutions, and for some it took several weeks before we finally had to toss them aside. In many cases, we liked the solution, but could not justify the delays they would incur to fix bugs or integrate a needed feature. Choosing a promising and shiny startup software is always tempting, but what if the project is discontinued? You are back to square one, because all the tests you’ve done, and the development and planning surrounding this solution are now obsolete. We needed to make a long-term investment, and master it asap. This meant we had to see a similar investment from our chosen solution. RedHat’s support of Openshift development swayed us in this regard. This support made it clear that Openshift was in it for the long haul. The underlying layer, Kubernetes, had just been opensourced by Google, and was starting to get a great deal of attention from the development community. To us, this meant that the solution had both the required support, and a promising development path for the future.

Summary

The final decision to go with Openshift was a watershed moment, based on the first feedback from our IT team. The feedback was simply: “We went through the Getting Started part of the documentation, which includes installation, and our test app is working fine”. This was the first time that everything had gone well in our testing from start to finish. That was all it really took, we had our new tool. It was time to figure out how to fit it to our existing app, and the new services within. With the help of services and endpoints, we could start putting components into Openshift, while still being able to communicate with legacy components outside the cluster. This was the core of our new solution, because it allowed us to migrate one piece at a time, and rollback easily if there was a problem. The benefits of the Kubernetes configuration of our app also allowed us to look ahead to our upcoming Enterprise Edition of Gemnasium.

Gemnasium’s founder, Philippe Lafoucrière, recently had the opportunity to attend and speak at the Web à Québec 2016 conference in Quebec City a month past. The WAQ conference, as it is known, is the largest gathering of francophone entrepreneurs in America, with guests arriving from not only Québec, but also France, Belgium, Africa, and the Caribbean.

Because not everyone could attend the event, and the topic shared by Philippe is essentially part of Gemnasium’s life story, we thought our readers might find it of interest.

Philippe’s presentation, co-delivered with Jean-Philippe Boily of Metrics Watch, covered Gemnasium’s journey from a monolithic single application using Ruby on Rails to a more distributed micro-services approach using Go. This journey is not uncommon, but the reasons for such a shift are not often publicly shared or explained thoroughly. Why was the shift made? What were the implications of the move? How was the transition performed? These are all questions that any developer might want to learn about before attempting their own such endeavor. As this is a rather in depth topic, we’ll split it up into several articles for easier consumption:

Why the Transition? (this article)

Criteria for Architecture

Making the Transition

Anatomy of a Micro-Service

Gemnasium is proud to announce our new partnership with Node Security Project. Node Security Project(NSP) is an integral part of the Node.js community, dedicated to ensuring that security is a core priority for Node. The Node Security Project offers tools to ensure the security of Node projects, as well as a place to report new security threats, as well as the solutions to them. This partnership offers Gemnasium projects using npm (node package manager) the added protection of NSP security advisories.

This partnership was a natural fit for Gemnasium, enhancing our core capabilities by allowing us to monitor your project even more effectively, covering project dependencies, updates, and even better coverage of potential security vulnerabilities.

Gemnasium has always been committed to providing value and ensuring our customers see return for their investment. This has led to some tinkering with our pricing model in the past, in order to ensure that our customers could get the features they wished, without paying more than necessary. Originally, our pricing model was based simply on private projects hosted on Github. In order to provide those with public projects access to premium features, we added an option to ‘buy’ features, labeled as ‘slots’. A ‘slot’ would be purchased, for example, if adding an additional private project to an existing package, if adding a report (such as Brakeman for Ruby), adding the auto-update feature, etc.

Unfortunately, this pricing model was not as customer-friendly as we had hoped and led to a number of questions and concerns in our support queue. There were enough such concerns to cause us to revisit this pricing model, and change it for the better. And usually, simpler is better, so that is the direction we decided to go.

A few weeks ago, we decided to remove the usage of ‘slots’ on Gemnasium and returned to the previous pricing model of charging only for private projects. All existing accounts were moved to the old pricing behavior transparently, ensuring that no loss of service occurred. The response thus far has been favorable. We’ve effectively lowered prices. Consider our Gold tier - previously it offered 50 ‘slots’. It now offers 50 private projects. As we mentioned above, the ‘slots’ system charged for premium features such as alerts and reports, meaning that a single private project could cost 3 slots or more. Because these features are now included, this not only potentially saves the client money, but it also makes billing clear to the customer.

Gemnasium’s pricing is now based on a tier system. It consists of 4 tiers (larger plans available on demand):

Each tier offers a free trial, allowing your team or organization to determine the value of the product to their organization, as well as what usage level they actually incur. All free trials offer the full stack, including unlimited collaborators. This means that if you select a free trial of Gold, and realize your organization rarely exceeds 15 private projects, then switching to silver is a simple decision.

Not only can you try before you buy, our full-featured product is now free for all public projects. Previously, public projects required ‘slots’ to be purchased for auto-update alerts or reports. This did not fit with our company vision. Gemnasium has always fully supported open-source development, and our pricing model had to align with that commitment. By ensuring that public projects are offered the full stack, including unlimited collaborators, for free, we ensure that the innovation and ingenuity of open-source development with Gemnasium is without any roadblocks.

On the other hand, any private projects will now require a paid subscription. Clients can register for free, and add as many public projects as they want, as described above. However, any new private projects will require them to select a pricing tier. Existing free tier clients taking advantage of the former free tier offering (which included 2 free slots), will still be able to continue working with their free private project. These users will have access to the full stack, including unlimited collaborators. However, these benefits are not transferrable to another private project, should they delete the current one. Upon adding a new private project, the existing free tier client will be required to select a pricing tier as well.

Hello readers! It has been a while since our last post, and we apologize for that. We will be renewing our efforts to continually inform you and others in the development community by posting on a regular basis. Not only do we hope to share our successes, but also provide industry insight, and help our users make the most out of our product. But this blog report isn’t just to renew our commitment to our readers, it is to announce an exciting development for Gemnasium.

Gemnasium now supports Bitbucket!

Atlassian’s Bitbucket is a web-based hosting service for projects using Mercurial or Git, and is designed to help professional teams collaborate on code with inline comments and pull requests. Bitbucket passed the one million user mark in 2013, making it one of the most popular developer choices available. Bitbucket support has been a long awaited feature, with users clamoring for it for over two years. We are extremely proud to now be able to tell new clients and old that Gemnasium is now able to serve clients on two of the most popular hosting platforms available.

Limitations

Adding support for Bitbucket has not been without its challenges, and for that reason, we have held off on announcing our progress till we had it right. With that said, it is a new feature for us, and there are some limitations (in comparison to our GitHub support) our users must be made aware of. These limitations include:

Gemnasium is unable (as of yet) to clone private repositories hosted on Bitbucket.org, so reports generation is not yet available.

While the auto-update feature is available to Bitbucket projects, Gemnasium cannot yet create a branch and a pull request based on dependency files.

Python support is limited to the dependency files compatible with PIP, such as requirements.txt and the like. (Gemnasium cannot process other dependency files in setup.py because it would need to clone the repo prior to running the script.)

The Bitbucket API does not allow us to search through the sub-directories of a repository in an efficient manner. Thus, recursive file search is not yet supported.

Mercurial repositories are not supported.

We plan to include all of the above features to Bitbucket repositories based on Git, but have no plan to support Mercurial repositories.

Challenges

Our primary challenge with regards to many of the above limitations has been the way Bitbucket handles access to repositories: There is no way to access a repository with a private token, like Github. An SSH key is needed to fetch the code, and as of now, we cannot create and share these keys in Gemnasium. Also, creating and sharing these keys would make Gemnasium a part of each and every user’s projects, and we’re not certain how our users would feel about this. Of course, we think that Gemnasium should be everywhere, but we can’t just make that decision for you.

Because some users may not want to give us full access to their repositories, we have provided the following guidance on how to create a webhook for bitbucket manually.

We'd like make it clear that there's no way to create a repo webhook without getting access to private repos. That's because one has to grant simultaneously both the "webhook" and "repository" OAuth scopes otherwise Gemnasium won't be able to create the hook, and the "repository" scope gives access to all the user's private repos.

In order to support Bitbucket, we also had to make changes to how Gemnasium handles URLs. Prior to Bitbucket support, all URLs were composed with two part URLs, formatted in this manner:

https://gemnasium.com/[org]/[project]

The organization and project would be the same as listed for Github.com. In other words:

https://gemnasium.com/gemnasium/gemnasium-parser

would become

https://gemnasium.com/github.com/gemnasium/gemnasium-parser

This format still works, and is still used for GitHub repositories. We have simply (well, not so simply) added support for URLs with a different source (bitbucket.org) appended. This means that:

https://bitbucket.org/gemnasium/test-public

Is available at:

https://gemnasium.com/bitbucket.org/gemnasium/test-public

(All projects from Bitbucket.org must use this new url scheme) This might not seem a drastic change, but it was a definite challenge for our team because all of Gemnasium’s processes were based on two-part URLs.

Open Source Support and Continued Improvements

It has been asked for a while, and we are now excited to announce the availability of Slack notifications for your Gemnasium projects!

While a large number of users is still using email notifications, more and more are migrating to team chats like the popular Slack. 

To start using Slack notifications, go to your projects settings, in the Hooks tab.  

Responding to the big demand on our support, we have decided to start working on the Enterprise Edition of Gemnasium. This version will be available on premises, running on your own servers. It will allow your Gemnasium instance to reach private repositories hosted on Atlassian Stash or Github Enterprise, and many other things the SAAS version is not able to achieve today.

We have planned to start a private beta in early 2016, but until then, we would be very glad if you could help us building the next Gemnasium for your enterprise:

You can subscribe to the Gemnasium Enterprise newsletter here: http://enterprise.gemnasium.com/ or fill our online survey directly: https://gemnasium.typeform.com/to/i2Gqqp

If you're familiar with Linux distributions, you probably know the concepts of "downstream" and "upstream"; these refer to the package you install and the source project, respectively. For instance, postgresql.org hosts the source of Postgresql RDBS and it's available as a package for Debian, Fedora, etc. When "upstream" is fixed and gets a new version, many packages have to be fixed too. One issue, many packages. Things are different for librares written in Ruby, Python or Node as the developers in charge of the project also build the package. Totally different story, right? Well, not so true.

Ubiquitous JavaScript

On August 24th 2015, a hacker named Yan who goes by @bcrypt demonstrated that it was possible to hide malicious JavaScript code that would be activated when minified by uglify-js. On the very same day, it's no surprise that this vulnerability was announced on nodesecurity.io. Shortly after that, @reedloden pointed at that uglify was also available as a Ruby wrapper called uglifier. One issue, two packages. But wait, there's more.

The ultimate example of JavaScript being ubiquitous is the a jQuery plug-in named DataTables. When an advisory disclosing a XSS Vulnerability in DataTables was reported published on September 9th this year, we at Gemnasium.com went straight to the GitHub repository and found these dependency files we immediately recognized: package.json, bower.json and composer.json.

From there, it was obvious that the same source code was available as 3 different packages:

datatables, a npm package

datatables/datatables, a PHP Composer package

and as a Bower package

3 in a raw. But there's more. Searching in Gemnasium's database, 2 more packages were found:

js.jquery_datatables, a Python package

jquery-datatables-rails, a Ruby gem

When preparing this article, we realized we missed a plugin for one particular CMS: collective.js.datatables is a "Plone Integration of jquery.dataTables plugin".

In this particular case, the versions don't match. Version 3.1.10.5 of the Python package integrates version 1.10.4 of the jQuery plugin - and it's affected.

In the end, we had to file 6 advisories for one vulnerability issue, covering all the packaging system we currently support here at Gemnasium. That was a good catch and it was worth the efforts. By the way, the advisory on nodesecurity.io is still the only one in sight.

Embedded C code

Interpreted languages like Ruby or Python sometimes need some piece of C code to boost their performance. Ruby and Python developers can call shared libraries using FFI but sometimes it's so easy to copy-and-paste C code.

The LZ4 algorithm has a C implementation, and it's small (around 2000 SLoC). In July 2014, it was reported that one could exploit the Python lz4 package. Looking around, we discovered that the same piece of C code was embedded in a Ruby gem appropriately named lz4. Same C code, same flaws.

Be aware of forks

When a project is forked, it's possible that a vulnerability ends up in several packages. In that case, our security team has to dive in as a fork may or may not be vulnerable. Actually, it's quite common that the fork gets a fix when the original project remains vulnerable. For instance the advisory on ldapauth provides no solution whereas ldapauth-fork is now fixed.

Looking around

Every now and then, we at Gemnasium.com go through the source code of some package that's been reported to be vulnerable. Not only we want to better understand the security issue, we also want to be sure about which versions are affected and which versions are fixed. This can be quite easy when the source is on GitHub, and when each individual version corresponds to a git tag, or very difficult (and frustating) when there's not relation between the two. While some package managers do it right, others don't really integrate with VCS. Spoiler: the more recent the package manager is, the better it is regarding this!

Venerable Python

Python certainly deserves some respect as it predates git and the popular VCS we're using nowadays. So it's no surprise that Python doesn't integrate with git tags. To make a new package version available to the users, a Python developer edits setup.py, updates the version number, creates a new distribution, and publishes the "dist" on Pypi.

Actually there are a few tools that makes possible to publish a package version and create a tag on the repo altogether, namely bumpversion, changes and zest.releaser. So there are some options to do it right.

Ruby can't make it, but it can learn

The build process that results in a Ruby gem is driven by a Gemspec file. Among other things, the Gemspec contains the version number defined a string. "gem build" knows absolutely nothing about VCS like git. No luck here.

A prolific Ruby developer who goes by Postmodern addressed this issue a few years ago and proposed Rake tasks to create git tags automatically. Ruby is a dynamic community, so we can be sure that there are other tools in the wild to do this as well.

Bump, tag with "npm version"

The version of a npm package is stored in a JSON file appropriately named package.json and it contains the version number. As explained in the great getting started video, the developer can either update the version number manually or rely on npm version to do it, prior to publishing with npm publish.

npm version is a good ally as it will create a git tag by default:

If run in a git repo, it will also create a version commit and tag. This behavior is controlled by git-tag-version (see below), and can be disabled on the command line by running npm --no-git-tag-version version. It will fail if the working directory is not clean, unless the --force flag is set.

This is a lot better than Python and Ruby! Still, one can do it wrong and edit the version manually.

Packagist is aware

PHP Composer builds a package out of a composer.json file that's similar to npm's package.json. Even though this JSON file used to contain the version number, Packagist.org now provides a better way to manage the versions:

The easiest way to manage versioning is to just omit the version field from the composer.json file. The version numbers will then be parsed from the tag and branch names.

So the developer can focus on the source repo, and Packagist will automatically serve new package versions based on the git tags. Easy.

Bower needs your tags

Bower goes even further: package versions are git tags. There's just no other way to publish a new version. See the instructions to register a package:

Your package must be publically available at a Git endpoint (e.g., GitHub). Remember to push your Git tags!

This is by design: Bower.io is a lightweight registry that can't even list the versions of a package.

Go will enter the dance soon

While there is no (official) package manager, nor central registry in go, the use of go get is far from flawless. There is no way to specify a specific version. At best, go users can use gopkg.in to lock to a tag or branch. This is even the official way for some packages like gobrake, the airbrake go client:

import ( "errors" "gopkg.in/airbrake/gobrake.v2" )

This will install the last commit of the v2 git branch. Only issue with this solution: the namespace of the package is not visible. gopkg.in/airbrake/gobrake.v2 will resolve to https://github.com/airbrake/gobrake/tree/v2.0.2 . Gopkg.in is therefore acting as a registry, with all the security issues implied. The go core team is finally decided to address this issue, and currently working on it. It’s a big step forward, and we are waiting for this issue to be closed to support Go in Gemnasium.

Reducing the gap

The trend is certainly good: new packages have tags on their repository (be it a git repo or not), either because it's good practice, fully integrated, or even mandatory. That reduces the gap between a package and its source code, and this certainly a good thing when one has to debug.

Listing the libraries a project depends on is generally not enough: one has to be more specific about the versions that are fully compatible with the project. This task is both critical and difficult - especially when it comes to versions that have not been published yet. Hopefully, we've got this wonderful convention called SemVer. For instance, if a project is compatible with version 1.0 of a library "z", we can assume that it's also compatible with 1.1, 1.2, 1.3, etc. But we also assume that's it's not compatible with version 2.0 and greater. This scenario is so common that package managers came with a syntax for that: one can go for a joker and set the dependency requirement to 1.*. Along with the joker, the caret and tilde operators make our lives easier. As times goes on, the syntax gets richer and richer. Actually, it has become so powerful that it sometimes goes wild.

Today package managers make it easy to install a library at a particular git branch or commit (assuming we can access the source code of the library). This is so convenient that PHP Composer (PHP's main package manager) goes a step further: if the source code of a library is managed using git, all the git branches are automatically turned into so-called "versions" users can easily install. Nice, don't you think? Well, depending on branches and commits certainly has drawbacks that justify the tedious process of releasing versions.

Until recently, a paid subscription was required to access the details of a security alert on Gemnasium.com. Today we've got good news for all free Gemnasium users: you can see the security alerts without paying a dime. Alerts management also went free. Not familiar with this feature? Let's recap.

Last year in February 2014 we introduced a new feature called trusted domains. Basically, a trusted domain makes possible to share a Gemnasium account with users having emails matching the domain. For instance, ACME's dev leader would trust "acme.com" domain to share ACME's projects with *@acme.com: [email protected], [email protected], etc.

"trusted domains" were introduced to easy team management in medium sized companies. But there were drawbacks we were not aware of when releasing this feature. Eventually, we decided to remove the trusted domains altogether. Here is why.

Why using Docker?

Since the first version of Gemnasium, we knew it wouldn't be possible to generate all the data we needed from the files provided. Gemspec files, for example, are pure Ruby files, and the only way to get all the metadata is to execute a ruby interpreter. Docker was not available at this time (2011), and the files were parsed like text files, leaving behind the metadata using ruby code (ie: spec.version = someGem::VERSION). When Python was introduced in Gemnasium, it became obvious that the "parsing era" was over, and some new architecture was needed to run isolated code. To determine requirements, we needed to run Python code, no parsing-based solution would give results. A lot of solutions were tested, and our first attempt was based on OpenVZ. But Docker and lxc-containers were by far the most promising, and we decided to give it a try. We instantly discovered a lot of advantages:

Blazing fast starting time (~100ms)

Ready to use images (a real toolbox!)

Resources management

Network management

Nice HTTP API (+clients)

And docker became the toy-for-everything in the team. While Docker was presented as a standard solution to ship applications, we saw another usage, perfect for our requirements.

Docker-based workers

When we started to evaluate Docker, not only did we run the tutorials, but also looked at the code (remember Docker is open-source: https://github.com/docker/docker). We tried http://golang.org earlier, and the code was easy to read, with a reasonable codebase size in early versions. We decided to write our workers in Go, like Docker, because we could build something quickly, and use inspiration from existing code. There's a nice go client available, and very active (many thanks to Francisco Souza), so we started from there. The initial idea was quite simple: Un-queue some jobs on a dedicated "docker" queue, where security and isolation are needed. Go was a perfect match too for that job, since we could take advantage of the wonderful go routines out of the box, to handle concurrent job executions nicely. We created in 2013 our docker-based worker: Gembosser. Gembosser is in charge of listening on a queue (only "docker" at this time), and perform the jobs in that queue.

The process is very simple:

Grab a new job

Pull the corresponding image (each job "class" is mapped to a local docker image)

Start and run the container, passing the environment necessary for the job

Fetch the result inside the container (logs + copy files)

Send the results + metrics back to Gemnasium, through the internal API

Loop until death

We are running this loop with a specific number of routines, no more than 8 per server. Some jobs can take one second to complete (the whole loop above), and some a few minutes.

When we need to update docker on the servers, we simply send a signal to the application, which will wait all the routines to stop, and don't start new jobs. When all jobs are finished, the program exists, and we update the daemon, before restarting Gembosser.

Docker containers are a great way to isolate processes. Our jobs running inside containers access only what they need, and can't get back to the message queue, or the internal API. Gembosser is ensuring the containers configuration, and we can add as many servers as needed in the cluster. Funny inception story: Docker is running inside vmware (vsphere) VMs (our own private cloud at OVH). If we need to process more jobs, we just deploy new VMs from the Gembosser-server template.

Almost 2 years later

Gembosser is running in production since mid-2013, and now process more than 100K jobs per day, without any problem. We are monitoring the number of running containers, which is very constant now (we have more queues today, and some background jobs are launched when workers are free). It wasn't always easy: we had to follow carefully the API changes (hopefully it's stable now), and had many issues with zombies containers, blocking routines, and concurrent issues. We introduced timeouts in Gembosser, and fixed all the bugs (located in our go code), but the main issue was some containers not starting under heavy load. It was fixed in 0.7, and now the only limit seems to be the capacity of our servers and imagination.

We are using Docker A LOT, for different purposes:

Development environments

Testing: our CI server is only running docker containers, nothing is actually installed in it.

Packaging: if tests are running Ok, we build Debian packages inside Docker. It keeps our CI server clean again.

Delivery of projects in our professional services activities. Private repos are a great way to ship applications.

Debugging: Imaging this failing job you want to debug? Most of the time, you'll need the whole application stack to reproduce the error. Not with our architecture. The job might be run in the exact same conditions with docker run [...].

The future

We are working on providing automated reports to our customers. Again, Docker is a perfect solution to run such jobs: It's safe, and we just need to create a few extra images to handle new features. Before the end of the year, we will propose a new feature to go further: "Bring your own reports". Don't find the report you need for your project? Create it!