#libcloud

For some time, Ansible has been my configuration management of choice and we use it for both Screenly and YippieMove. Since both of these services are running on Google Compute Engine, we're using Ansible's dynamic inventory for GCE.

(Behind the scenes, this dynamic inventory is using Apache Libcloud, which is a great Python library for interacting with various providers.)

When I first followed Ansible's Google Cloud Platform Guide I did run into a fair bit of trouble with authentication.

As it turns out, Libcloud is very picky when it comes to the environment variables that you need to set. To solve this, I whipped together a little script that I call on to set the appropriate variable environments for each project I'm working on. This saved me a lot of headache.

The script looks as follows:

#!/bin/bash export GCE_PROJECT=your-project export GCE_PEM_FILE_PATH=~/.gce/$GCE_PROJECT\.json export GCE_EMAIL=$(grep client_email $GCE_PEM_FILE_PATH | sed -e 's/ "client_email": "//g' -e 's/",//g') gcloud config set project $GCE_PROJECT

Just change GCE_PROJECT to match your setup, and then run:

$ source /path/to/script.sh

You can now run Ansible with the GCE inventory file.

As an added bonus, this also configures gcloud to the same project.

The Mist.io open source project, provides a touch-friendly control panel for managing your clouds. Whether you’re using KVM hypervisors, public or private clouds, Docker containers or bare-metal servers, you can manage everything from a single point.

Mist.io supports all popular cloud providers, but there are some that we didn’t get around to adding yet. If your favorite provider is not supported, don’t worry because it’s fairly easy to add a new one. To illustrate this, we’ll walk you through the process of adding a new provider to Mist.io. We’ll use the example of Host Virtual, a highly flexible provider with datacenters in 20 locations around the world, we’ve recently added support for.

The only prerequisite in order to add a new provider to Mist.io is a libcloud driver for that provider. In case there isn’t one, check this comprehensive guide on how to write one. The list of providers supported by libcloud is really big and many are already covered. The list grows fast so before you get your hands dirty send an email to the mailing list in case someone is already working on it.

The first step is to fork the mist.io repository on github. Then install it locally by following the instructions. Make sure you use the fork’s url where required. After a few minutes Mist.io should be up and running on port 8000.

Now create a new git branch and start hacking:

$ git branch hostvirtual_provider $ git checkout hostvirtual_provider

Time to write some code. Add the new provider to src/mist/io/config.py under the SUPPORTED_PROVIDERS_V_2 section.

The Host Virtual entry looks like this:

# HostVirtual { 'title': 'HostVirtual', 'provider' : Provider.HOSTVIRTUAL, 'regions': [] }

Please use a descriptive title. The provider attribute, should correspond to the libcloud provider name. You can find a list of all libcloud providers and their names here. The regions attribute is usually empty and applies only to providers like Amazon which treats regions, eg EC2 Sydney, EC2 Ireland etc, as separate clouds. The attributes ‘title’, ‘provider’ and ‘regions’ are mandatory, but you may add as many additional ones as you need to.

Now edit methods.py and add the provider to the add_backend_v_2 function:

elif provider == 'hostvirtual': backend_id, backend = _add_backend_hostvirtual(title, provider, params)

Define the actual function below:

def _add_backend_hostvirtual(title, provider, params): api_key = params.get('api_key', '') if not api_key: raise RequiredParameterMissingError('api_key') backend = model.Backend() backend.title = title backend.provider = provider backend.apikey = api_key backend.apisecret = api_key backend.enabled = True backend_id = backend.get_id() return backend_id, backend

Host Virtual requires only one key for access to its API. Others may require certificates, usernames or other info as you can see on other _add_backend functions in the same file.  

Now search for the connect_provider function and pass the API key to the libcloud driver object.

elif backend.provider in [Provider.LINODE, Provider.HOSTVIRTUAL]: conn = driver(backend.apisecret)

Each provider allows you to perform different actions depending on the state of the machine. For example, most cloud providers do not allow starting a terminated machine. Libvirt on the other hand, which is used to manage KVM hypervisors, allows restarting. You can specify the actions that can be performed for each machine state on the get_machine_actions function.

In the case of Host Virtual tagging in not supported:

if conn.type in (Provider.RACKSPACE_FIRST_GEN, Provider.LINODE, Provider.NEPHOSCALE, Provider.SOFTLAYER,Provider.DIGITAL_OCEAN, Provider.DOCKER, Provider.AZURE,Provider.VCLOUD, Provider.INDONESIAN_VCLOUD, Provider.LIBVIRT, Provider.HOSTVIRTUAL): can_tag = False

Next step is to write the code for provisioning VMs. This is handled by the create_machine function. Most providers require name, image, size, location and finally a public SSH key. The latter is used in order to be able to connect to your machine once it is up and running. If you are unsure on what your provider requires, you can check out the corresponding libcloud driver and see if the create_node function takes any extra arguments.

In some cases the provisioning process is complicated, but fortunately Host Virtual’s API is straightforward so you just need to add this:

elif conn.type == Provider.HOSTVIRTUAL: node = _create_machine_hostvirtual(conn, public_key, machine_name, image, size, location)

and the respective creation function:

def _create_machine_hostvirtual(conn, public_key, machine_name, image, size, location): """Create a machine in HostVirtual. There is no checking done here, all parameters are expected to be sanitized by create_machine. """ key = public_key.replace('\n', '') auth = NodeAuthSSHKey(pubkey=key) try: node = conn.create_node( name=machine_name, image=image, size=size, auth=auth, location=location ) except Exception as e: raise MachineCreationError("HostVirtual, got exception %s" % e, e) return node

The final step is to create the relevant “Add Cloud“ form in Mist.io’s UI. Open src/mist/io/static/js/app.js and find the section starting with “var PROVIDER_MAP”. For Host Virtual the entry is rather simple, with only a title and and API key field:

hostvirtual: [ { name: 'title', type: 'text', defaultValue: 'HostVirtual', }, { name: 'api_key', type: 'password', }, ],

For the final touches add the provider’s logo. You’ll need a 48x48 png file which you should put in the src/mist//io/static/images/sprite-images directory.

That’s it!

Once you test and make sure everything is working fine, commit the code and push everything to github

$ git add src/mist//io/static/images/sprite-images/* $ git commit -a -m ‘Added hostvirtual provider’ $ git push -u origin hostvirtual_provider

Open a pull request from mist.io’s staging branch to your fork and branch. We’ll review the code and contact you if we have any questions or suggestions. If everything checks out, we’ll add it to the Mist.io staging branch, test it a bit more and push it to master.

최근 libcloud를 써볼 일이 있었다. 많이 써보지는 않았지만 여러모로 좋은 라이브러리라는 생각이 들어 소개해본다.

Libcloud는 boto와 비슷한 기능을 공유하는 라이브러리인데, boto가 Amazon Web Services API의 클라이언트 라이브러리라면, libcloud는 AWS 외에도 Microsoft Azure, Google Cloud Compute 등 다양한 업체의 서비스도 함께 지원한다는 점이 가장 큰 차이점이라고 할 수 있다. 어차피 각 업체마다 대응되는 같은 용도의 서비스가 있는데, 크게 EC2 같은 가상화 인스턴스 서비스 (그야말로 클라우드의 primitive라고 할 수 있는…), EBS와 같이 거기서 쓸 블록 스토리지 서비스, S3와 같은 오브젝트 스토리지 서비스, ELB 같은 로드 밸런서 서비스, Route 53 같은 DNS 서비스 등이다.

그래서 가령 libcloud를 이용해서 EC2 인스턴스를 새로 만들고, 만들었던 인스턴스를 삭제하고, S3에 이미지를 올리는 스크립트를 짜고 나서, 맨 위쪽에서 드라이버 종류만 AWS에서 다른 업체로 바꾸면 그대로 돌아간다. (실제로는 각 프로바이더마다 확장하는 기능이 존재하는데, 네이밍 컨벤션에 따라 그런 함수나 인자명은 앞에 ex_가 붙는다.)

Libcloud는 현재 25 종류 이상의 드라이버를 제공하고 있고, 이 안에는 심지어 KT ucloud까지 포함되어 있다. 강대명 씨가 드라이버 코드를 기여했다고 한다. 나는 앞으로도 쓸 일이 없겠지만… 전략적으로 돈이 없을 때 ucloud를 쓰다가 사업이 번창하여 돈을 많이 벌면 AWS 등으로 옮길 때 유용하게 쓸 수 있을 것이다. ㅋㅎㅎ

이 외에도 boto보다 libcloud가 더 나은 점이 많다.

API가 boto보다 더 깔끔하고 쓰기 쉽게 되어 있다. 여러 클라우드 서비스를 오갈 일이 없이, 그냥 AWS만 쓴다고 해도 boto의 대안으로서 나쁘지 않다.

Python 2에서만 돌아가는 boto와 달리 Python 3에서도 아주 잘 돌아간다.

더미 드라이버를 제공하기 때문에 단위 테스트 짜기가 훨씬 쉽다. boto를 쓰면 mock을 만들거나, 아예 libcloud가 해주는 그런 레이어링을 내 애플리케이션 코드에서 만들어야 하는데 이러다가 풀고자 하는 문제에 집중을 못하게 된다.

The use of Web APIs for communicating with cloud providers has been increasing over the last few years and every computing and web services provider is expected to maintain a well documented API for their clients. It enables developers, system administrators and engineers to  automate tasks and assists in the deployment of servers and services; and as multi-cloud setups become the norm and cloud management tools a necessity, a well documented API significantly enhances the user experience.

As one would expect, there are many tools in different programming languages that facilitate the communication with each cloud API.  On the Python world, libcloud is the de facto library to communicate with one or more cloud providers. It abstracts away differences among multiple cloud provider APIs and provides a unified way to manage cloud resources.

Mist.io makes heavy use of libcloud in order to support a wide range of providers in a unified interface. You can try it out for yourselves by logging in mist.io.

Libcloud was initially developed by the awesome CloudKick team. Today it is an Apache foundation project. It has a healthy community, with lots of contributors, updated documentation, mailing lists, a ticketing system, a continuous integration platform and an established user base.

Libcloud allows users to manage four different cloud resources: Cloud Servers, Cloud Storage, Load Balancers as a Service and DNS as a Service. Cloud Servers is the oldest and more mature part of the library and currently supports more than 26 providers.

Many cloud providers eg Amazon, Rackspace, Linode, DigitalOcean etc, are already supported by libcloud. In this post we want to illustrate how easy it is to add support for new providers, by documenting the steps we followed to add support for a cutting edge US-based cloud provider NephoScale.

What should we implement on our compute driver

Our goal is to create a compute driver that will provide the most common sever management actions. Actions that are universally supported by libcloud for all the cloud providers:

list_nodes: provide a list with our nodes, including information such as public/private ip addresses, node state -running, rebooting, stopped etc- and some extra metadata such as the image, location etc

list_images: list all the images the provider supports for faster node creation

list_sizes: list different plans, providing data such as CPU, ram and disk size. Some providers include the price of each size plan

list_locations: Most cloud providers allow us to specify the different zone where we can deploy nodes

and the following node actions:

create_node: create a node specifying it's size, image, name and location

deploy_node: create a node and deploy a public ssh key for server access, and optionally run a deploy script after the node is initialized and can be accessible

reboot_node/shut down_node

start_node/stop_node: start a stopped node, or stop a started one

destroy_node: Delete the node

We will implement the above for the NephoScale driver, plus some extra functions that will allow us to create, list and delete keypairs. Keypairs in NephoScale can be ssh or password keys, for server or console access. Digital Ocean also allows the listing/creation/deletion of keys through it's API, while other cloud providers do not offer that functionality. Other examples of provider specific functionality include tagging, where only a few cloud providers (eg AWS, Rackspace) provide. In this example and in order to be as generic as possible, we will target functionality that is available for most -if not all- cloud providers.

Getting started with the compute driver

We'll need access to our provider API. In this scenario, the Nephoscale API.

We will use the Digital Ocean driver as a template as it is one of the most recent, well written and up-to-date drivers. There is an effort by libcloud to standardize the basic functionality for all the compute drivers, and update the oldest ones. It is also very important when cloud providers contribute code or review the drivers created for them in libcloud, or even better maintain them as their API changes. 

Compute drivers are located on libcloud/compute/drivers and they all inherit from the base compute driver NodeDriver on libcloud/compute/base.py. The driver is well documented and we can learn a lot from reading the comments and of course the code.

Add the driver on types.py and providers.py

We'll start by adding our provider on the DRIVERS dict on libcloud/compute/providers.py and on Provider class on libcloud/compute/types.py. It's pretty straightforward as we can see from the existing drivers.

Create driver file and start writing the code On libcloud/compute/drivers/ we create a file with the name of our provider: nephoscale.py in our case.

After the initial imports that we copy from the Digital Ocean driver, we specify the API endpoint of our cloud provider

API_HOST = 'api.nephoscale.com'

and a dict with the states that the nodes can have. This differs from provider to provider, with some providers having just a few states -running, stopped - and others having intermediates ones -e.g. rebooting, shutting down etc.

NODE_STATE_MAP = {     'on': NodeState.RUNNING,     'off': NodeState.UNKNOWN,     'unknown': NodeState.UNKNOWN, }

Writing the NephoscaleNodeDriver

We then need to create a NodeDriver based driver

class NephoscaleNodeDriver(NodeDriver):     """Nephoscale node driver class"""     type = Provider.NEPHOSCALE     api_name = 'nephoscale'     name = 'NephoScale'     website = 'http://www.nephoscale.com'     connectionCls = NephoscaleConnection     features = {'create_node': ['ssh_key']} def list_locations(self): ...

Here we specify the type (same as on libcloud/compute/types.py), the connection class (NephoscaleConnection) and a features dict with the way deploy_node will try to authenticate to the created node after create_node has run . Inside the NephoscaleNodeDriver is where all our functions will live. We'll write all functions we want to implement there, list_images, list_nodes, reboot_node etc. To keep consistency, we make sure that list_nodes returns a list of Node objects, list_images a list of NodeImage objects, list_sizes a list of NodeSize objects and list_locations a list of NodeLocation objects.

Writing the Connection class

NephoscaleConnection is the Connection class that will handle connection, authentication and send the request to the NephoScale API endpoint. All drivers implement a Connection class that is responsible for sending the request data, adding HTTP headers or params and also encoding the request body.The API host endpoint is specified there, along with the HTTP headers or params that are used to authenticate for each provider.

NephoScale API calls require HTTP Basic Authentication with the user/password base64 encoded on every request, so we'll add it to add_default_params. 

class NephoscaleConnection(ConnectionUserAndKey):     """Nephoscale connection class     Authenticates to the API through Basic Authentication     with username/password     """     host = API_HOST     responseCls = NephoscaleResponse     def add_default_headers(self, headers):               user_b64 = base64.b64encode(b('%s:%s' % (self.user_id, self.key)))         headers['Authorization'] = 'Basic %s' % (user_b64.decode('utf-8'))         return headers

Other providers handle authentication with requiring the params on each request -API key, password, secret etc.

DigitalOcean for example requests the client ID and API key on each request, and we pass it with add_default_params

class DigitalOceanConnection(ConnectionUserAndKey):     ...     def add_default_params(self, params): params['client_id'] = self.user_id params['api_key'] = self.key return params

Inside our Connection class we can override encode_data method, so that it encodes data as requested. For example we can set it to encode data with json.encode, if the API endpoint expects json encoded data. 

The Connection class that gets overridden on our driver lives in libcloud/common/base.py. 

Writing the Response class

All drivers also need a response class that handles responses of the API endpoint, parses body and returns Exceptions or the actual return content.  Response classes for the libcloud drivers derive from JsonResponse or XMLRPCResponse, depending on the response type of the cloud provider -Json or XML. Both derive from Response class that lives in libcloud/common/base.py. We want to make sure here that in the case of errors we throw the correct exceptions - e.g. InvalidCredsError for failed authentication.

class NephoscaleResponse(JsonResponse):     """Nephoscale API Response"""     def parse_error(self):         if self.status == httplib.UNAUTHORIZED:             raise InvalidCredsError('Authorization Failed')         if self.status == httplib.NOT_FOUND:             raise Exception("The resource you are looking for is not found.")         return self.body

Notes on create_node and deploy_node

Most of the drivers of libcloud implement create_node to return a Node object, of the newly created host. Function deploy_node is not overridden , but instead gets called from the NodeDriver, since it does a lot of things that need not be rewritten, such as checks that the node public ip is up, that the machine can be accessible, that it can authenticate via password or ssh key, and optionally runs a deploy script.

The default behaviour of create_node is to send the request for the node to be created, and get a response with the created node id. Most cloud providers we have used reply with the ID of the created node, along with information about the state of the node, public ip etc. NephoScale does not respond with an ID once it receives the request for a node to be created, but instead replies with a status ID that is not very helpful. So as soon as we send the request and receive a valid response, we'll be waiting for a bit and try to get the machine information by requesting list_nodes and checking if the machine has appeared. When we have the machine Node, we return create_node. In this way the behaviour of create_node stays consistent with the other drivers and deploy_node works as expected, out of the box.

Once again, reading the comments of NodeDriver on libcloud/compute/base.py for deploy_node and create_node functions will help us implement create_node and make sure deploy_node will work.

Keep consistency, hard code things when necessary

One of the cool things with libcloud is that it standardizes things. It enforces the use of basic entities -nodes, sizes, images, locations- for all cloud providers it supports.

SoftLayer for example does not provide a size entity through it's API, but rather expects that we specify CPU, RAM and disk size when we create a node. The way libcloud driver is implemented, one can either pass these manually, or simply select one of the sizes provided by list_sizes.

Ideally all size/image/location related data should be fetched by asking the provider. When this is not possible, for example when the provider does not implement a request, we need to hard code these settings. For example the pricing info for some of the providers isn't returned while asking for the sizes (list_sizes) and thus is hard coded on libcloud/data/pricing.json. Or the info regarding sizes for AWS is not available through an API call. The obvious disadvantage with hard coding things is that they have to be maintained to reflect the current status of the provider's API.

Working with our driver

While writing the driver, it is always good to test it:

user@user:~/dev/libcloud$ python >>> from libcloud.compute.types import Provider >>> from libcloud.compute.providers import get_driver >>> driver = get_driver(Provider.NEPHOSCALE) >>> conn = driver('user','correct_password') >>> conn.list_nodes() [<Node: uuid=e20bdbf7ef6890645f5b217e0bd2b5912b969cc1, name=nepho-7, state=0, public_ips=['198.89.109.116'], provider=NephoScale ...>]

We can use the connection class for GET or POST requests according to the API, for testing and debugging issues. E.g to implement the list_locations for NephoScale, we need to see what the response is after requesting the locations (via https://api.nephoscale.com/datacenter/zone/)

>>> conn.connection.request('/datacenter/zone/').object {'success': True, 'total_count': 2, 'subcode': 0, 'message': 'Your request was processed successfully.', 'data': [   {'datacenter': {       'id': 1,       'airport_code': 'SJC',       'name': 'SJC-1',       'uri': 'https://api.nephoscale.com/datacenter/1/'},    'uri': 'https://api.nephoscale.com/datacenter/zone/86945/',    'name': 'SJC-1',    'id': 86945},   {'datacenter': {      'id': 3,      'airport_code': 'RIC',      'name': 'RIC-1',      'uri': 'https://api.nephoscale.com/datacenter/3/'},    'uri': 'https://api.nephoscale.com/datacenter/zone/87729/',    'name': 'RIC-1',    'id': 87729}], 'response': 200}

Contributing to libcloud

Having created the compute driver for our cloud provider and having tested it's functionality, it is time to commit it to libcloud. In this case we need to write tests -ideally for all functionality, add some fixtures and make sure the tests pass. Libcloud contains unit tests for nearly everything and enforces testing for new functionality. Testing on libcloud deserves another post by itself.

Until then, details for contributing can be found can be found here.The process involves creating a ticket to libcloud’s JIRA issue tracker and opening a pull request to libcloud’s github repository. The code will be reviewed and most probably comments and suggestions will arise.

So have fun developing your driver!

To check this driver in action log in to mist.io

libcloud[http://libcloud.org]

Apache Libcloud is a standard Python library that abstracts away differences among multiple cloud provider APIs.

fog[http://fog.io/]

Whether you need compute, dns, storage, or a multitude of other services, fog provides an accessible entry point and facilitates cross service compatibility.

Just getting started working with cloud resources? You are not alone, and having so many complicated options makes it hard to know where to start. fog delivers the knowledge of cloud experts to you, helping you to bootstrap your cloud usage and guiding you as your own expertise develops.

By coding with fog from the start you avoid vendor lock-in and give yourself more flexibility to provide value. Whether you are writing a library, designing a software as a service product or just hacking on the weekend this flexibility is a huge boon.

With a rapidly expanding community and codebase the advantages of fog just keep coming. Join us and together we will realize the future of cloud computing.

jclouds[http://www.jclouds.org/]