Scalingo

Adding a postdeploy hook to your Application

To add a postdeploy hook to your application , you just have to add to your Procfile (or create a Procfile file), a new type named postdeploy. The command specified for this type of container will be executed in a one-off container at the end of each deployment.

postdeploy: command to run

The command will be executed synchronously after the deployment. If the hook fails, the deployment will also fail, the old version of your application will be kept until a new successful attempt of deployment. Here is an example of logs you can get when a postdeploy hook has been set:

Examples

The postdeploy hook is executed in a one-off container and has exactly the same environment, which means you'll have access to your the environment variables of your application, and variables from the runtime environment (APP, CONTAINER, CONTAINER_VERSION). You can use these variables directly in the postdeploy line of the Procfile, or in a script which is called by it.

Applying Rails migrations

postdeploy: bundle exec rake db:migrate

Keep in mind that migrations may fail, design them knowing that, you will find an article from Codeship explaining this process.

Notifying a third-party service

Newrelic

postdeploy: newrelic deployments --revision=$CONTAINER_VERSION

Rollbar

postdeploy: curl https://api.rollbar.com/api/1/deploy/ -F access_token=$ROLLBAR_TOKEN -F environment=$RAILS_ENV -F revision=$CONTAINER_VERSION -F local_username=scalingo

Run a custom script

postdeploy: ./scripts/postdeploy.sh

You can event chain taks by using logical operators like && or ||

postdeploy: ./postdeploy-step1.sh && ./postdeploy-step2.sh

One-click Deployments

A while ago, we released our way to add a button on your open-source project website or Github page. Thanks to this hook a lot of new possibilities are available as you can achieve operations to initialize the database or anything required by your application in an automated fashion.

Note that to ease the process, if you were already using this feature from Heroku, we've added a compatibility layer to use the scripts.postdeploy attribute of the JSON app manifest as a standard postdeploy hook on Scalingo.

What you should not do

Building assets: The postdeploy hook is not the right place to build your assets. Keep in mind that the action is executed in a dedicated one-off container, the other containers from your application won’t be impacted by any file change. All operation relative to assets should be done in the build process.

Conclusion

You'll find more information about the way this hook is being executed in our documentation. This is a first stone to a bigger picture, next step will be related to Github:

Raise your hand if you want to test the new Github integration! #PaaS #Github pic.twitter.com/8MK6IM9KtE

— Scalingo (@ScalingoHQ) October 7, 2016

Detailed timeline of the events

At 00:05 CEST, Leo's (our CTO) phone rang. Not the usual tune when someone is calling, but a PagerDuty alarm telling us that something was wrong with our platform. After a short analysis, it was clear that the problem was network related, not related to our software stack, and we had to get the responsible for the level below us, our infrastructure provider.

It is not a secret that we're not working with a public cloud like Amazon Web Services or Google Cloud (which are not exempted of downtime either), but we are relying on a smaller French structure which is based on 2 datacenters in Paris and in Strasbourg. We've made this choice to have a better control on the price/performance ratio and because we knew those guys (yes, we keep on telling them that their website should be in english too) for a long time.

So at 00:10 CEST, we reached their support, actually engineers who have also been woke up by something wrong. They acknowledged our problem, telling us it is impacting several of their customers and that their team is on it. Ok, all right.

Then, started the 'dark' period, this period where you feel blind as there was nothing we could do, except waiting our infrastructure got juice (i.e. network) again. As we had no problem with our infrastructure, applications and databases kept running as expected. Harassing their team would not have helped us more as the problem has been acknowledged and the level of stress of everyone was, obviously, very high.

We got a first message at 00:30 CEST, that the cause might be related to a BGP router blocking the traffic, resulting in the fact that our infrastructure was in the dark, but still, no solution, stress was increasing minute after minute. Then again, we could just wait for them to solve this.

At 01:00 CEST, they called us back to declare they got the real problem, which was not related to the BGP router, but to the firewall/load balancer positionned just after. They were in the process to find a workaround as quickly as possible to recover the service, and to solve it.

It is only around 01:40 CEST that our servers got the internet back and that all applications and the platform itself went back online. All the information we have at the moment from our provider is that the problem came from their load balancers (which are positionned just before of our servers). These have been shortcut to give internet back to our infrastructure. Once the real problem has been solved later, we got back to the initial networking configuration.

What have we learned

First, our emergency notification system is working all right, a few minutes after the problem declared itself, someone was awake, ready to get things into hands. That is a good point, even if we test it, it is critical that such alarm happens correctly in real situations.

Then, in the hosting world, we have positionned ourselves in the Platform as a Service area. It means, that we 'use' an infrastructure provider, install our software stack, and provide service to customers in order to ease their life, the choice of this infrastructure provider. To the question "are we working with the good guys?", we answer yes. The contract we've signed with them mentionned the following:

There won't be any downtime lasting more than 4h

There won't be more than 12h of downtime per year

The emergence of Pokemon Go apps

The 6 days following the initial release of PokemonGo, some players/developers from all around the world have been taking time to reverse-engineer the game in order to understand how the mobile application communicates with Niantic Labs servers. They finally released a first Python third-party API client. With a few additional days, clients in any programming language were available, resulting in a very large amount of applications being created by the community. The main problems people are trying to solve are: "What are the pokemons around me?" and "Where are the rare pokemons I'm tracking for days?"

The 16th of July (2 days after the release of the python library), the project PokemonGo-Map got started, to propose a simple python application to visualize pokemons around a specific location given as configuration. The server pretends being a specific user at a specific location walking around and discovering pokemons, displaying them on a GoogleMap map.

The app was several weeks on Github trending page, making its popularity rocketing to the hype world of the game Pokemon Go. A lot of people have been willing:

To use the application

To deploy the application

To contribute to the application

From that point, we assumed Scalingo could help to solve the problem number 2.

One-Click Deployment

In December last year, we've released our 1-click deployment feature. It let you put a button on your webpages, or Github project page to allow people deploying an open-source project in a single click.

We made a pull request to the map project which has been accepted. From that point people coming on the project page were able to deploy the application in less than two minutes on the platform.

SQLite - Houston there might be a problem

Almost 1,000 instances of the application got deployed in a few weeks. We knew in advance that all these people would be doing it, because it's free, enjoying our free trial, and in exchange they would discover the advantages of a platform like Scalingo.

After the first tens of deployments, the tech team got alarmed: disk usage has raised abnormally, after a short analysis, the PokemonGo-Map applications were using, in average, 10MBps of disk bandwidth. Let's make a simple computation: 1000 * 10 = 10GBps, even spread on multiple servers, it is a problem.

Why is the application using so much IO? One answer: SQLite.

Usually, we discourage the use of SQLite not because of the disk usage but because it does not fit the PaaS model. SQLite writes its data in a single file, usually suffixed '.db' alongside an app. In the PaaS model, disk around your app is temporary and any deployment or restart operation would result in data loss as the '.db' file would disappear.

As the data gathered from Niantic API are only temporary, representing pokemon states at a given time, it was not a problem to use SQLite, even if restarted or redeployed, data would be fetched again.

Here is an example from one of our server monitoring, do you guess when was the IO pike?

Solving the IO problem

From that point, we had two solutions:

Stopping the one-click deploy feature for this app

Finding a way to handle such IO load

Stopping deployments would be radical but we were under the spotlights, it was not interesting from a business point of view to stop that now, this solution should only be our last resort.

Our tech team took a glance at the project to get how it is working, to analyse if there was something do-able to reduce the disk usage. However it is not our policy to modify the source code of what is deployed on the platform, we should accept anything and the platform should adapt to the app needs.

One of us made the following assumption: "what if the SQLite database was in memory instead of being on the disk". That's an idea which was worth thinking about it. The data from the SQLite database was never really large (average of 10MB), putting it in RAM would increases the app memory consumption reasonably and decreases the IO to… 0.

From certain point of view, it is a dirty hack, from another it is an elegant solution to a very precise problem. A few hours later, we were deploying in our production cluster a hook detecting if the application is a PokemonGo-Map, and if it's the case, creating dynamically a TMPFS directory where the SQLite database is created by the app.

> "TMPFS is a temporary filesystem that resides in memory and/or your swap partition(s), depending on how much you fill it up. Mounting directories as TMPFS can be an effective way of speeding up accesses to their files, or to ensure that their contents are automatically cleared upon reboot." - Archlinux Wiki

Conclusion

After restarting all the concerned applications, the disk usage dropped back to normal and we were able to handle more and more deployments without fearing to decrease performance of our customers production applications.

It's not in our habits to develop custom things for specific apps, but it is sometimes required to keep our infrastructure stable and its cost normal (to make our business model actually works).

Enabling PHP 7 for Your App

Getting your application executed by the powerful PHP 7 is as simple as specifying it in your Composer file. The important key in this JSON file is require:

{ "name": "my-php7-app", "require": { "php": ">=7.0" } } `

We use our semver webservice to solve the required version.

If you already have an application running, just update your existing composer.json file with a newer version. Then, git commit and git push this change.

An Example of PHP 7 App

In order to start a PHP 7 application, you first need to write a simple PHP code. For this example, we will just show a simple phpinfo page. The index.php file is:

<?php phpinfo();

Then we need to specify the composer.json. The simplest one would be:

{ "name": "php7-support", "require": { "php": ">=7.0" } }

Next we must generate the composer.lock file. We can install composer locally and generate this last file with:

php composer.phar install

Eventually create a Scalingo application, commit and push your files:

scalingo create appname git remote add scalingo [email protected]:appname.git git add index.php composer.{json,lock} git ci -m "Test PHP 7 support" git push scalingo master

This application is now available online at appname.scalingo.io:

What’s New in PHP 7

Improved Performance

The most awaited change in PHP 7 is the use of the new Zend Engine 3 which provide better performance. Indeed, this new version is twice as fast as the Zend Engine used in PHP 5.6. More performance improvement will be more easily achieved in the future thanks to a better code organization for a migration towards a just-in-time compiler.

Do not be afraid anymore to use the assert construct when you develop a new code. The new assert has zero cost when in production. More information on this in the documentation.

Enhancement for the developers

Return Type Declaration

PHP now provides a way to specify a type for the return value. Be aware that the behavior changes depending on the strict typing mode you use. More information on this in the documentation.

Anonymous Class

Support for anonymous classes was added in PHP 7. This simple syntactic sugar add a very convenient way to create simple one-off objects. More information on this in the documentation.

Improved Exceptions

In older PHP versions, handling fatal error use to be nearly impossible. In production, such an error results in a blank screen for the client. In PHP 7, an exception is thrown rather than just stopping the script. Some unrecoverable errors still exist such as out of memory.

To develop this feature, a new Throwable interface becomes the common ancestor for both Error and Exception.

Much more things

Much more new wonderful things were released in this version such as new operators, easier to use Cryptographically Secure PseudoRandom Number Generator (CSPRNG)… Read the full blog post and changelog for a comprehensive list of novelties.

Available extensions on Scalingo

The following extensions have been upgraded on the process:

APCu with v5.1.4

imagick with v3.4.1

mongo with v1.1.7

When not to use PHP 7

PHP 7 being a recent version, some extensions do not work yet. You should not use this version if you need one of these extensions:

igbinary: support for PHP7 is in a different project. It is not yet working.

libevent: not meant to be used with PHP >= 5.4.

memcached: not compatible yet.

redis: should work but in another branch. Not yet released in master branch.

Global view

The metrics view is accessible from the Scalingo dashboard just underneath the Containers section. This page contains two main parts: application graphs and containers graphs. You can adjust the viewing period to 3h, 12h, 24h, 48h and 72h in the past. We try to autodetect your timezone from your browser and you can still change it later if needed.

 Application graphs

The application graph displays global data that are not container specific: events and routing metrics.

The events graph is here to show you all the events that happened during the viewing period. This can help you link an application behaviour with events that happened on the platform, e.g. spot a deployment that contains a memory leak or follow your application behaviour after a scale up (or down).

A lot of events are available on your application timeline but we think that only a few are relevant on the metrics view:

Restart event

Deploy event

Scale event

We'll add new event types in the coming weeks (you'll find the complete list of events in our developer documentation).

The router chart is here to show you how many requests your application gets per minute. It displays you two different kinds of data. The first one is the number of requests per minute, the famous RPM, and the second one is the number of server error responses generated by your application (HTTP responses in the 500 range).

Note: 504 errors can be generated by our proxy if your application takes more than 30 seconds to respond. More information is available in the routing documentation.

 Container graphs

The container graphs use the container types defined in your procfile.

For each of your container type, two charts are shown. The first one shows the CPU usage and the second one the Memory and SWAP usage of this type of container. For the memory chart, we stack your memory usage (in blue) and your SWAP usage (in red). That way you can easily see the total memory usage of your application.

Protip: Is your application slow? Check your swap usage! If your app swaps a lot it will significantly alter your application performance. You'd better reduce your memory usage or use a bigger container size.

Note: The swap line is only shown if the swap usage exceeds 2% of your container memory limit.

If you have more than one container of a specific type, these graphs will show the mean CPU usage / Memory consumption of all your containers of the same type.

Detailed view

If you have more than one container of a type defined in your procfile, you'll be able to access a detail view of your container. The graphs are exactly the same except that they are now per container (and not per container type). It will help you spot a bugged container and therefore simplify the debugging process.

API Integration

Like every other feature, metrics are fully integrated in our API. The endpoint to access the data is documented here.

Metrics are collected every minute. However to prevent high resource usage of our metrics stack, the API will send you 300 points max per graph. To have the minute precision, your viewing period should not exceed 300 minutes.

How does it work?

Data comes from two different data sources. The first one is the Scalingo frontend servers (3). Since the beginning, Scalingo uses Nginx to proxy and balance requests to the application containers.

In order to collect and store request informations, we use the NGINX Lua Module which let us intercept many Nginx events and run Lua code. Of course, we kept in mind that this code will be run for each request sent to our platform so the code must run as fast as possible and the codebase must be as little as possible. That's why the Nginx bindings only send raw data to a Redis server which is used as a buffer. Every minute our router metrics aggregator (6) will collect data in the Redis buffer, aggregate them and send them to our metrics storage adapter (5).

The second data source is the application containers. To get the resource consumption of the containers we use our in house solution called Acadock which was introduced in an older blog post. Once per minute these data are collected on each node by our container metrics aggregator (4) and sent to our metrics storage adapter.

We rely on InfluxDB as a backend to store all metrics data. In order to be independent of this backend we implemented an intermediate service called Metrics Storage Adapter that translates internal API calls to InfluxDB queries. This layer speaks JSON and provides a standard protocol to access and retrieve metrics data.

By the way, we're very happy with InfluxDB. It perfectly suits our needs and proved to be very reliable. We did a POC (Proof of Concept) nearly 9 months ago between OpenTSDB, Prometheus and InfluxDB. Even if at the time Influx was not the most performant timeseries database, it has proven to improve very quickly.

Finaly, when your browser displays metrics graph, it calls our API (7). The request is proxied to our metrics storage adapter through an authenticated endpoint.

In the future

We already plan to improve a few things to this new Metrics component. For example, we'll soon add real time updates, response time graphs and an alerting mechanism (the ability to trigger notifications when your containers reach a certain value).

Plus, since we use InfluxDB as an internal product, why not make it available to all our customers as an add-on?

Earn 20% of what they use

If a person signs up with your referral link and spends something on the platform, each month you'll get 20% of what they spent in credit, during 12 months after their account creation.

REFER SOMEONE NOW!

How to refer a friend?

Go to your profile (click on your avatar in the upper right corner) and choose "Referrals". You will find there your personal referal link. Copy it and share it with your friends, customers or followers.

Referral stats

The program is completely transparent and you will find your personal referral statistics on the same referral panel.

See the t-shirt in action

How to get MongoDB 3.2

It only takes a few seconds, sign up on the platform, create an application, choose MongoDB addon and that's it, your new and shiny MongoDB 3.2 database has been provisioned, this is the new default version of the addon, you've nothing special to do.

Upgrading an existing database

Go in your database dashboard (from the addons page of your app dashboard), and an upgrade card will be available. Upgrading to MongoDB 3.2 from 2.6 requires 2 steps. First you need to upgrade your database to version 3.0, then a second upgrade will be available to get to 3.2. Once this second upgrade has been done, you'll see that your data has reached the latest available release.

Enabling WiredTiger Storage Engine

WiredTiger Storage Engine is activated by default for new 3.0 and 3.2 databases.

Migrating to WiredTiger requires the database to be taken down for a while (from seconds to minutes according to your database size). This is the time to migrate existing data to the new storage format.

By migrating to WiredTiger, the authentication scheme will be switched from MONGODB-CR to SCRAM-SHA1. You have to be cautious, your application may be incompatible, resulting in authentication problems. Have a look to the driver compatibility list first.

What's new in the MongoDB 3.x releases?

WiredTiger Storage Engine

The biggest and most famous change in MongoDB has been the integration of a Storage Engine named WiredTiger. This new engine delivers 7-10x greater throughput for write-intensive applications with more granular document-level concurrency control. You can expect a huge performance gain if your application tends to be write-heavy.

This storage backend has also changed the way your data are stored on our disks. Data files take now up to 80% less space on disk thanks to compression. Our addon offer let you use 5 times the memory size in disk, you can now store much more data for exactly the same price.

WiredTiger engine is designed to use more memory than before. It uses aggressive cache and compression strategies to reduce I/O operations and to improve performance. The MongoDB default cache size starts at 1GB for WiredTiger and that doesn't include memory needed for file system cache or other runtime requirements like index storage. According to the plan chosen we'll determine the amount size for this cache for most databases.

Partial Indexes

MongoDB let you create indexes with a given condition to index only documents matching this condition:

db.restaurants.createIndex( { cuisine: 1 }, { partialFilterExpression: { rating: { $gt: 5 } } } )

The above request will create an index which only consider restaurants where the rating is greater then 5.

Official Documentation

Document Validation

So far it wasn't possible to set any constraint at the database level about the values your applications have been sending to MongoDB. All the logics had to be included in your app.

It is now possible to create collections where only some type of documents can be written.

db.createCollection( "contacts", { validator: { $or: [ { phone: { $type: "string" } }, { email: { $regex: /@mongodb\.com$/ } }, { status: { $in: [ "Unknown", "Incomplete" ] } } ] } } )

Official Documentation

Query optimization debugging

Debugging query optimization has been done much easier by improving the explain() method, you can now get much more details about the internal operations of the database for a given query. Official documentation

It has now several 'verbosity' modes to get the information you need:

// 'queryPlanner': MongoDB runs the query optimizer to choose the winning plan // for the operation under evaluation. db.collection.find(query).explain("queryPlanner") // 'executionStats': MongoDB runs the query optimizer to choose the winning // plan, executes the winning plan to completion, and returns statistics // describing the execution of the winning plan. db.collection.find(query).explain("executionStats") // 'allPlansExecution': MongoDB runs the query optimizer to choose the winning // plan and executes the winning plan to completion. In "allPlansExecution" // mode, MongoDB returns statistics describing the execution of the winning // plan as well as statistics for the other candidate plans captured during // plan selection. db.collection.find(query).explain("allPlansExecution")

Much more things

These were the features which were the most user-facing but much more has been released, We encourage you to have a look at the official Release notes for MongoDB 3.0 and 3.2:

https://docs.mongodb.com/manual/release-notes/3.0/

https://docs.mongodb.com/manual/release-notes/3.2/

A few words about Ruby on Rails and Meteor

Ruby on Rails

If you are using Mongoid in your Ruby on Rails application, ensure you are using at least Mongoid 5.x before using MongoDB 3.2, otherwise your app will fail to authenticate to the database.

Meteor

Using MongoDB 3.2 will work normally, but you won't be able to use the latest features of the database as the driver used by Meteor is not up-to-date. Partial indexes and collection validation won't be available.

A few days ago, the Meteor team announced that they will fully support MongoDB 3.2 in their next release: Meteor 1.4. Read the announcement.

Oplog is obviously also available for this release.

Nothing's hidden, it's a Docker image

As our other database addons, they are based on Docker and the images we're building are public and you can try it on your workstation:

docker run -p 7000:27017 \ -v /tmp/dbdata:/var/lib/mongodb \ scalingo/mongo:3.2.6-2 /mongo user password adminpassword

That's it, your database will be running of the port 7000 of your workstation. Docker hub page: scalingo/mongo

Version choice and future upgrades

Next step in our database service is to let you choose the precise version you want to use for your application. You may want to try your application against Mongo 2.6, 3.0 or 3.2, but so far you can't choose a precise version of the database, we impose the last stable supported release.

We'll also add a maintenance window to apply minor and security upgrade the day you choose and at the time you want your database to be upgraded.

Scalingo's Context

We are using Docker containers at two different steps of the workflow we’re proposing. First comes the build step. A container is created, your code and all its dependencies are gathered thanks to the buildpacks, resulting finally in a Docker image, stored in a private highly available registry hosted in our infrastructure. Each time, our orchestrator sends a request to start a container of your application, one or multiple servers of our infrastructure get the image we’ve previously built and run it, that's the run step.

In this case, we’ve covered what we called application images, images which are dynamically built according to the deployed app. We’re also handling another type of images: database images, pre-built image to run the different types of database handled by the platform: MySQL, PostgreSQL, MongoDB, Redis, Elasticsearch.

Structure of application images

When an application is deployed, we create a container based on what we’ve called a builder image, this image is open-source and you’ll find it in the Docker Hub, named scalingo/builder. Each time any user deploys a new release of their application, a container is created based on this builder image. Once the build of the app and its dependencies is done, another unique layer is added to the image resulting to the application image. As soon as the image is sent to the registry, this last layer only is sent as all the others layers are common.

The builder image is a common base to all the applications hosted on the platform, as a result, it should be a generic image which is unspecialized. That's why it is based on a stable Ubuntu LTS environment (currently 14.04 LTS).

In this builder image, we've installed different libraries and softwares which are commonly used, in the build process of application or to be used by a human when running one-off containers. You'll find:

Build essential utility (GCC, make, autotools and more.)

curl, git, telnet, ssh-client, openssl

NodeJS, Ruby, Perl, Python, Java (system version, nothing specific to app)

Imagemagicks

MySQL, PostgreSQL, MongoDB, Redis clients and development librairies

Making these default libraries and softwares optional has been part of our reflexion. We know that all application won't use imagemagicks for instance, but we've considered that managing a set of builder images instead of one is not worth it. Actually, a common question asked is: “why are you using a unique image instead of multiple images based on the technology of the application, wouldn’t it speed the deployment process?”

Moving one image all around a cluster is much easier and convenient than handling myriad of images. Even if we sacrifice a few megabytes of disk space for this precise image. Once it has been fetched on a hosting node, we’re done. When a new application has to start, the application layer is fetched and nothing else. It contains everything required to run the contained application. As an example, imagine we want to create a specialized image for ruby applications, we should be maintaining the images for all versions of ruby since 1.8.7, it would represent more or less 30 images to prepare and maintain through the time.

People often argue about the size of Docker image, but in this case we’ve considered, having one common and large base image is the good way to go. There is no difference for us if the image is 800MB or 1000MB, servers get it once and use it thousands of times. Here is the schema of the structure of our builder image:

Database Images

With database images, we had another approach, as we control what will be running in the containers, we don’t need to build these images on a really generic stack, all it has to do is to run the database we’re looking for and do it well.

As well, we’re working with five types of database, each one having multiple versions, so to avoid filling our disks with database images, we’ve started looking at how to reduce their size significantly. In this field, Docker images based on Alpine Linux instead of Ubuntu or Debian seem to win the race.

Alpine Linux is a lightweight linux distribution weighing nothing more than 5MB. It is way less than the standard Ubuntu image, which is, as we’ve seen in the schema, 200MB heavy.

As a result, our redis image is now weighing 14MB. Previously, we never took care of the image size, it was 364MB large. Reducing the size of an image by a factor of 30× without loosing any feature of what the image should do, is definitely worth it. Working with such images gives us more flexibility are we're able to migrate databases faster.

All our database Docker images are available on the Docker Hub, don’t hesitate to give them a shot.

Conclusion

Context

The need to access protected infrastructure by VPN has been picked up by our customer MF-TEL and they agree to sponsor the work of developing an addon. You can been very thankful to them because they also agree to let us share this work publicly. They have been using this addon for a few months, and today, we’re making it Generally Available.

What kind of VPN?

Currently, you can connect to a VPN IPSec (Cisco) in mode client-to-site. The addon is using the tool vpnc, so any VPN compatible with this client will be working correctly.

Setup of the addon

Provision the addon

First, you need to provision the add to your application. This can be done through the dashboard or with our command line tool.

scalingo scalingo-vpn-ipsec vpn-ipsec-standard

Setup the configuration

From the addon tab of your dashboard, click on ‘Link to Dashboard’ under the VPN addon icon. You’ll arrive on the dashboard of the VPN IPSec addon. It contains a text area in which you’ve to write your vpnc.conf file.

Usually such configuration file looks like this:

IPSec gateway [server] IPSec ID [VPN group] IPSec secret [shared secret] Xauth username [username] Xauth password [password]

Once you’ve validated the configuration, the environment variable SCALINGO_VPNC_CONF will be added to your application, the app will be restarted. You’ll be able to see in the logs:

2016-02-03 13:59:31.321065492 +0100 CET [web-1] -----> Starting VPN connection 2016-02-03 13:59:31.321223270 +0100 CET [web-1] -----> VPNC started in background (pid: 70)

Automatic reconnection

Our container runtime will check in real time if anything happens to the connectivity between your Docker container and the VPN it’s connected to. If it is disconnected you’ll see in your application logs:

2016-03-02 19:36:59.95866371 +0100 CET [web-1] ! VPN disconnected, attempt to reconnect

Then, the system will immediately try to reconnect to the VPN. In the reconnection fails because of a problem of your infrastructure, or because of networking problems, we’ll attempt to reconnect every 30 seconds.

How does it work?

Your application is running in one or several Docker containers. By default these containers are connected to the external world with one network interface. When you are using this VPN addon, we automatically create a second network interface used for the VPN connectivity. The following schema represent this method.

Security consideration

One of the first question you may be asking yourself is: “how safe is it?”. We’re ensuring that the VPN interface is only present inside the network namespace of your application containers. A network namespace is an isolated virtual environment linked to a container. It contains all the network interfaces dedicated to this precise container. It means that for other containers sharing the same host, there is no way to see of sniff anything related to your VPN.

Q&A

Can I setup site-to-site VPN?

No, currently site-to-site VPN is not compatible with this addon, it is only working in client-to-site mode.

What happens if the connection fail, will my app be unavailable?

No, it will be written in the logs that we have not been able to connect your container to your VPN, but your application will be started correctly. You should check the configuration and ensure all the fields are correct.

Where can I find an comprehensive documentation of the vpnc.conf file?

The Ubuntu manpage about vpnc contains all the properties which can be set in the configuration file.

Conclusion

This addon only contains a single plan. Once provisioned you'll be able to configure a VPN connection for your app. We'll see in the future, thanks to your feedback, if additional features or plans are required.

Context

Since the beginning of our commercial offer in Frebruary 2015 we have only been accepting credit card payments through Stripe as payment gateway. Some users have been complaining about that, wishing to use less credit card dependant payment methods. That’s why we’ve decided to add Paypal as payment solution.

Why PayPal?

People and companies often have a PayPal account, they know how it works and in most of the case, they trust this company to keep their banking data safe.

PayPal let you use directly your bank account as payment source, thus you are now able to pay on Scalingo without the use of a credit card, by using your PayPal account directly.

Pre-paid credit

PayPal payment is not similar to credit card payment because you have to buy credit for your account which will be used in priority to pay your invoices.

It means that if you have both a credit card and pre-paid credit, we'll use the credit until there is no more, and pay the rest with the configured card.

If you are only using paypal credit and that there is not enough credit to pay your monthly invoice, you'll be notified and will have 3 days to add enough credit to your account to pay the pending invoice.

About recursive payments with PayPal

PayPal has a way to achieve monthly automatic payments of variable amount of money, in our case the price of the platform usage each month. However this feature is reserved to companies using PayPal for a long time.

The other method to have monthly automatic payment is to handle it with a monthly fixed amount. This last solution does not really fit our needs as the amount may change from one month to another. In one case you would over-provision your account or in the other case you may under-provision it.

As a result, we've decided to apply the following model: automatic payment by card and pre-paid credit with PayPal. If one day we've the opportunity to enable the feature 'Reference Transactions' with PayPal, allowing us to smoother experience for our customers, of course we'll do it.

Workflow

To buy some PayPal credit, you have first to go to your "Billing Profile" and click on the PayPal tab.

Then choose 'Add Credit', fill your billing profile, choose the amount your want to credit your account (minimum is 10€), and finally, click on the PayPal checkout button.

A few seconds later, you're redirected to PayPal confirmation gateway to approve the transation, once it is done, you're redirected to Scalingo's dashbaord, the system will check the transaction, and that's it, the credit has been added to your account.

What's next?

Scalingo's approach to Platform as a Service is close to CloudControl model. Both services are based on buildpacks and Procfile. Those two technilogies have been introduced by Heroku years ago and have become a standard for PaaS providers. Thanks to these similarities, the migration process only takes a few minutes.

Even though we think this migration documentation is comprehensive enough, don't hesitate to search information in our documentation http://doc.scalingo.com and to contact our support to help you in the migration process.

1. Create an account on Scalingo

Just go at https://scalingo.com and follow the instructions to get an account.

2. Get used to the environment

Dashboard

Scalingo is proposing different ways communicate with its service, the more commoon is to use the web dashboard https://my.scalingo.com. Here, you'll be able to create and configure your applications.

Command line tool

The command line interface (CLI) is pretty simple to install. Just go at http://cli.scalingo.com and follow the instructions. It is essentialy like installing a single binary on your system, there is no dependency to install, it just works.

You will find its documentation at following page: Introducing the CLI.

Everything which can be done in the web dashboard can be done with the CLI, and even more, like accessible your databases with helper commands or running one-off jobs for your apps.

Advanced usage: API

If you need to get things done automatically, you can use our open API. Its documentation can be located at http://developers.scalingo.com. Everything which can be done through our web dashboard or our CLI can be done with this API as we're also using it.

3. Create your applications

Clicking on '+' in the dashboard or with the CLI:

# Create an application named cloud-app scalingo create cloud-app

4. Install required addons

Your application often need a database to run correctly. Most kind of databases are provided directly by Scalingo as addons. As previously you can do it on the dashboard by going in the 'Addons' tab of your application, or with our CLI:

# List all the addons available scalingo addons-list # List all the plans for an addon scalingo addons-plans scalingo-postgresql # Provision the 1GB plan of the PostgreSQL addon scalingo addons-add -a cloud-app scalingo-postgresql 1g

Scalingo is providing the following addons:

PostgreSQL

MySQL

MongoDB

Redis

Elasticsearch

[Private Beta] VPN IPsec client

[Private Beta] Shared file system

You'll find more information about these addons at https://scalingo.com/addons

If an addon does not have an equivalent on Scalingo, you can use the environment variables of your CloudControl app, directly in your Scalingo app.

5. Configure your account to push your code

Before pushing your code, youd should setup your SSH key to authenticate your GIT pushes. With the dashboard, go on the 'Keys' page and upload your public SSH key (usually $HOME/.ssh/id_rsa.pub). Or obviously, you can do it with the CLI:

# Add a key named 'workstation-key' located at $HOME/.ssh/id_rsa.pub scalingo keys-add workstation-key $HOME/.ssh/id_rsa.pub

6. Push your application

We're using the same workflow as CloudControl, so you first need to add a remote to your git repository:

# Add Scalingo remote for 'cloud-app' git remote add scalingo [email protected]:cloud-app.git # Push your code git push scalingo master

That's it, your application will be live in a minute.

7. Move your data from CloudControl to Scalingo

CloudControl didn't rely on their infrastructure to host the databases and third-party providers were providing those. They gathered in their documentation all the different method to export the data of the database of their the different third party provider.

To import them on Scalingo, we'll advise you to follow the more comprehensive guide located in our documentation.

8. Handling the custom domain names

What is called 'alias' on cloudcontrol is named 'custom domain names' on Scalingo. To migrate your app you've to modify the configuration of your app on Scalingo, and also the DNS field at your domain name registrar dashboard.

# Add the 'my.cloudapp.com' domain name to you application scalingo -a cloud-app domains-add my.cloudapp.com

For more informations about domains on Scalingo, take a look at our dedicated domain name management page.

Something else specific to your app?

App Side Bar

The app side bar received multiple enhancements to help you working with Scalingo Dashboard. You can now sort your apps by last deployment date or by name, click once for an ascendant sort, twice for descendant sort.

If you are working with collaborators on your apps, sometimes you might want to hide apps shared with you, it is now possible, just click the MINE button.

Browsing through a long app list is tedious, so we implemented a smart filter on the app name. A few key strokes is enough to find your app quickly. You can fuzzy search through your apps by typing just few letters.

Little indicators are added to help you quickly distinguish apps that are stopped and apps where you are collaborating. Apps are considered stopped when they are scaled down to 0 containers.

Other improvements

Some users told us they often don't know the container specifications for each container size, so we added a bit more information when you are about to change your container's size.

To read more about Kinto, the concepts and how to use it please refer to the Kinto documentation. If you are ready to test Kinto read on!

Setup Kinto on Scalingo

Kinto developers did a good job on making Kinto really easy to setup. Deploying Kinto on Scalingo is as simple as clicking on the Deploy to Scalingo button (read more about our one-click deploy solution). Just give your Kinto instance a name and you are ready to deploy it.

If you want to make change to the code of the deployed application you can git clone it. The command is git clone [email protected]:my-kinto.git where my-kinto is the app name you gave earlier. You can now make change to the cloned project such as updating it when a new version of Kinto is released. To deploy your changes just do a git push scalingo master and your app will be updated.

Interacting With Your Kinto Instance

Once deployed, your Kinto instance is immediatly accessible at for example my-kinto.scalingo.io. You can query information about your Kinto instance by doing:

$ curl -X GET https://my-kinto.scalingo.io/v1/ {"project_name":"kinto","capabilities":{},"settings":{"readonly":false,"cliquet.batch_max_requests":25,"batch_max_requests":25},"project_docs":"https://kinto.readthedocs.org/","http_api_version":"1.3","project_version":"1.11.0","url":"http://my-kinto.scalingo.io/v1/","cliquet_protocol_version":"2"}

And start to interact with it, to for example create a new bucket and list your buckets.

$ curl -X POST https://my-kinto.scalingo.io/v1/buckets -D '{"data": {"id": "blog"}}' -u "user:pass" {"permissions":{"write":["basicauth:6754e15aa495a872aafd3de3a70f7de4a214d0655f3bce89f5b7fe669ff51b28"]},"data":{"last_modified":1454427199729,"id":"E58Ud-Ih"}}

curl -X GET https://my-kinto.scalingo.io/v1/buckets -u "user:pass" {"data":[{"last_modified":1454427199729,"id":"E58Ud-Ih"}]}

Setup Parse Server on Scalingo

parse-server is a Parse-compatible Node.js module made for Express. It is not a stand alone solution but Parse do provide the parse-server-example app that can run out-of-the-box after a git clone.

To setup an instance of Parse Server on Scalingo 2 options are available, the automated one-click deployment and the manual way. We recommand you to use the automated solution to setup your Parse Server which is a great base project customizable in the future.

Automated One-Click Deployment

The automated solution is to use the Scalingo deployment button:

This button will deploy the official parse-server-example project which is suitable to be used for existing Parse app migration. A MongoDB instance will be provisionned automatically for you during the process.

Follow these simple steps to setup your own Parse Server:

Click on the Deploy to Scalingo button.

Give your future Parse instance a name.

Fill the APP_ID and MASTER_KEY. If you are migrating from an existing Parse app you can use the same values here. Otherwise, you can choose any string.

You are now ready to deploy the application, click on the Deploy Application button.

If you want to use the deployed application as a base to your project you can git clone it. The command is git clone [email protected]:my-parse.git where my-parse is the app name you gave earlier. You can now make change to the cloned project such as adding some Cloud Code. To deploy your changes just do a git push scalingo master and your app will be updated.

Manual setup

To follow the following steps you will need the Scalingo CLI. If you prefer using a GUI, steps 2 to 4 can be done in Scalingo Dashboard.

Clone the parse-server-example project and change directory to it.

Create an app on Scalingo: scalingo create my-parse

Add the Scalingo MongoDB addon: scalingo addons-add scalingo-mongodb free

Setup MongoDB connection string: scalingo env-set DATABASE_URI='$SCALINGO_MONGO_URL'

Your are now ready to deploy your own Parse Server: git push scalingo master

Migrating from Parse to Scalingo

Once you've setup your Parse Server on Scalingo it's time to migrate your Parse-hosted app. Be sure to read the official Migrating an Existing Parse App guide first to understand the process. The following points are the big steps that must be done to successfully migrate to a Scalingo hosted Parse Server.

Migrating Client apps

To allow your client apps (iOS, Android, web, etc.) to access your own Parse Server, you'll have to change the server URL in each of your client apps to the Scalingo app URL (e.g. my-parse.scalingo.io). Instructions for each SDK are detailed in Using Parse SDKs with Parse Server.

By default, Parse Server will serve the Parse API on /parse. To change this, or if you are using older client SDKs, you can run scalingo env-set PARSE_MOUNT=/1. This command will set the environment variable PARSE_MOUNT to the value of /1.

Migrating Database

On Scalingo, databases are not publicly accessible for security reasons, so the process described in the official migration guide will not work. Instead, you'll have to manually transfer the data to your Scalingo MongoDB instance. Here are the steps:

Go to the new Parse Dashboard > App Settings > General, and click on Export Data.

Wait for the email containing the link to download exported data.

Change directory to your Parse Server project

Start a tunnel to your Scalingo MongoDB using Scalingo CLI:

$ scalingo db-tunnel SCALINGO_MONGO_URL Building tunnel to my-parse-4727.mongo.dbs.appsdeck.eu:31068 You can access your database on '127.0.0.1:10000'

Prepare downloaded files by deleting the { "results": and } that wrap the array containing your collection data.

Import each collection with the following command and change the options accordingly (--db, --collection, --file, --username, --password). To find out the password to your database you can use the commandscalingo env | grep MONGO.

$ mongoimport --db my-parse-4727 --port 10000 --username my-parse-4727 --password 6O65xJH3_k2qu0DeIfxg --collection yourcollection --file yourcollection.json --jsonArray connected to: 127.0.0.1:10000 2016-02-01T18:45:47.073+0100 imported 9999 objects

Final Words

By following the official Parse migration guide and this blog post you should now have a working Parse Server. As mentionned by the Parse team, the migration will not be easy for everyone, you should conduct tests to ensure your data is consistent and your Cloud Code still work. We hope by providing a quickly deployable solution on Scalingo will help you ease the transition.

An interesting Parse alternative is the Mozilla backed Kinto that we will discuss in the second blog post tomorrow.