jclouds: multi-cloud library for the JVM

Everett Toews is hitting the road, getting some hands on keyboards, and really teaching developers how to use jclouds. He'll be giving a workshop titled "Solve the Cross-Cloud Conundrum with jclouds" at Gluecon this year. The session is on May 22 from 10:30 to 11:30 am in Breakout 2.

Gluecon

What is Gluecon? In their own words, "Cloud, Mobile, APIs, Big Data — all of the converging, important trends in technology today share one thing in common: developers. Developers are the vanguard. Developers are building in the cloud, building mobile applications, utilizing and building APIs, and working with big data. At the end of the day, developers are the core." This session will get developers doing what they do best. The workshop includes an overview of jclouds, its terminology, and its community then it quickly gets down to development. First, we'll install jclouds for use on the command line during the workshop. Then we'll work through an example of using jclouds to start virtual machines in multiple clouds. Rackspace will be offering free cloud trial accounts to the attendees of this session. It's their way of helping you get started in the cloud and say thanks for coming out.

Conclusion

jclouds 1.6 represents 7 months of refactoring to essentially "reset" the project 4 years after its founding.  The goal was to do less magic and be easier to code and troubleshoot.

No doubt, we've added new apis, including ones from OpenStack, Rackspace, AWS, DynECT, and UltraDNS, so if any of that interests you, move on to details in the release notes.

Most exciting is our move to the Apache Incubator.  Watch out as details will unfold this week as we progress to our next level of open source!

jclouds community is an international group with over 100 contributors since we started in early 2009.  The idea of holidays vary, but we decided "Mistletoe" as an appropriate codename for jclouds 1.5.4.  Here's why.

Chef and Karaf

We've noticed an attraction between systems that use chef and those that use karaf.  For example, it isn't uncommon to have a deployment pipeline that incorporates the two.  Since jclouds has subproject for both chef and karaf, it seemed natural to cozy them together.  Thanks to Ioannis and ignasi, karaf and chef are now a couple!

For example, the following syntax can be used in karaf directly, or via the jclouds cli to start a node in any supported cloud and bootstrap chef recipes.

karaf@root>node-create --adminAccess --recipe chef/java::openjdk karaf  

What else is in 1.5.4?

We've a number of other highlights in jclouds 1.5.4.  Many thanks to the code contributors jclouds, jclouds-chef, jclouds-karaf, and jclouds-cli for putting them together. Also to our review team for helping ensure quality: Andrew G and Matt for code, and Becca on docs.

Rackspace Cloud Load Balancers

Everett modernized support for "rackspace-cloudloadbalancers-us" and uk is now available.  This includes syntactic sugar such as auto-pagination and easy transforms.

FluentIterable<HostAndPort> sockets = clb.getLoadBalancerApiForZone(zone) .list().concat().transform(converter);

Nova Server Diagnostics

Some implementations of Nova have diagnostic information available.  Thanks to Leander, you can now discover this capability at runtime.

Optional<Map> diagnosticInfo = novaApi.getServerApiForZone("az-1.region-a.geo-1") .getDiagnostics(serverId);

S3 Multi-Object delete

Maginatics have BlobStore containers with over a billion objects in them.  S3 containers (buckets) can now be cleared with 1/1000 the requests, thanks to Andrei's additional support of S3 multi-delete.

DeleteResult result = s3Api.deleteObjects(container, keys); assertEquals(result.size(), keys.size());

What's next

Released on 2012-11-14, jclouds 1.5.3 includes minor fixes, and a few important updates:

new openstack-cinder and rackspace-cloudblockstorage-us/uk providers

add new Asia Pacific (Sydney) Region [ap-southeast-2]

new TagApi for ec2, in a revised syntax similar to openstack-nova

handle network failures in large container (1M+) deletes in blobstore

jclouds-cli and jclouds-karaf now provide a more intuitive file-based interface for reading and writing blobs.

Many thanks to the contributors in this release of jclouds, jclouds-chef, jclouds-karaf, and jclouds-cli. Also many thanks for the diligence of our review team: Andrew G and Matt for code, and Becca on docs.

Moreover, we're indebted to BuildHive for checking over 500 pull requests to date! On a build side, we currently validate both JDK6 and JDK7 on each pull, thanks to Andrew P.

Please update to jcloud 1.5.3 and check the new features. Particularly, try things out using JDK7. If you have questions, ping our user group or tag your question with "jclouds" on stackoverflow.

At jclouds, we've been running our Jenkins continuous integration jobs in CloudBees' DEV@cloud for a while now (CloudBees has a FOSS programme). We also have an active and ever-increasing contributor community, which amongst others means...lots of pull requests.

So we were very interested to hear about BuildHive, another CloudBees initiative that integrates with your GitHub account to automatically trigger build jobs for pull requests, and quickly signed up.

We did not regret it. BuildHive has served as a crucial initial filter to allow us to concentrate precious reviewing time on working changes and to remain confident, as pull requests are updated, that we are not going to break the build.

Of the last 100 builds, we've had 7 with errors, 2 with warnings, and 1 incomplete, so a roughly 10% efficiency gain in terms of focussing pull request review time. In future, we may also use BuildHive to provide "style feedback" and similar metrics for pull requests - most likely some feature suggestions for CloudBees to come here!

Over 6 months in the making, and just in time for JavaOne, jclouds 1.5 is formally released. 

Many of you have been awaiting (or helping build) massive new support for OpenStack based clouds. There's even more, such as our new Jenkins Plugin and command-line utility.

I encourage you to check out our release notes and give the newly minted version 1.5.1 a whirl.  If you haven't seen our new website, check it out and let Becca know how you like it!

For those of you looking to meetup, you can find us at the Jenkins User Conference,  JavaOne, meetup at Abiquo during VMworld EU, the OpenStack Summit, Java/Cloud2Days, and RWX.  You can also meet us on irc freenode #jclouds and our google group.

The problem of large result sets is something cloud providers want you to have. For example, what better sign of cloud life is there than Amazon S3 having a trillion objects stored so far. The way most cloud apis deal with listing your million (or even thousand) things is to paginate the response.

For example, you get a first page with maybe 500 records in it, and a "marker" you can use to get the next.

Sounds pretty straightforward, and most jclouds abstractions do this dance for you.

For example, our BlobStore api has an interface PageSet which holds a bunch of results and the underlying marker.  This also deals with the fact that sometimes "marker" is called "nextToken" or other unnecessarily different names :)

3 years since the birth of PageSet, you can imagine folks could get a better idea of what they *really* want.

Here's a few comments:

Marker isn't necessarily a String! - Andrew Gaul notes that especially in native implementations, it isn't the case that Marker will always be a String. Imagine you are making an in-memory store. It is very likely the marker will be an Object in this case.

Set is the wrong type for results! - Tim Peierls notes that uniqueness isn't a fundamental concern of API results.  In fact, it can get in the way of streaming.

Iterating through a bunch of pages is monkey-work! - Jeremy Whitlock found dancing through each page to get a complete view of metrics was tiring, and made a CloudWatch function for that.

Ensure it is possible to opt-out! - Toni Batchelli notes that iterating across network calls can lead to inconsistent state.  Particularly clojure users will want the option to manually control pagination.

This feedback underscored jclouds general concern to make things easier, yet still allow control. The status was tracked in issue 1011 and our jclouds-dev google group.

Through several iterations  and many thanks to Tim P for the design, we have a new type: PagedIterable, which extends Guava's fantastic FluentIterable.

Here are a few examples of how it can be used:

lazy advance through all your metrics:

FluentIterable<Metric> allMetrics = cloudwatch.getMetricApi().list().concat();

advance only until we find the load balancer we want:

Optional<LoadBalancer> firstInterestingLoadBalancer = elb .getLoadBalancerApi().list() .concat() .firstMatch(isInterestingLB());

get only the first page of database instances

IterableWithMarker<Instance> firstPage = rds.getInstanceApi().list().get(0);

The above syntax is being worked through relevant apis.  In order to try it out, grab jclouds 1.5.0-beta.7 (releasing today), and use any of the following methods:

cloudwatch/aws-cloudwatch - MetricApi.list()

elb/aws-elb - LoadBalancerApi.list()

iam/aws-iam - UserApi.list()

rds/aws-rds - InstanceApi.list() SecurityGroupApi.list() SubnetGroupApi.list()

Other apis and abstractions will be caught up while we finish the 1.5 release.

We are very near beta, I promise!  In the mean time, we just cut a new codebase with a bunch of cool new toys.  Here's a few.

Thanks to Adam Lowe, we are getting even deeper into OpenStack with more Keystone support than ever. Setup your code to pull org.jclouds.labs/openstack-keystone, and you can do stuff like this:

ContextBuilder contextBuilder = ContextBuilder.newBuilder("openstack-keystone"); RestContext keystone = contextBuilder .credentials("tenantId:user", "password") .endpoint("https://keystone:35357") .build(); for (String regionId : keystone.getApi().getConfiguredRegions()) { AdminClient adminClient = keystone.getApi().getAdminClientForRegion(regionId); for (Tenant tenant : adminClient.listTenants()) { // ... } }

Also pro, is our new Amazon CloudWatch support from Jeremy Whitlock. This is our first complete renovation of an AWS api to have the same look/feel as our new OpenStack stuff.  Just add a dependency on org.jclouds.providers/aws-cloudwatch and you can do this!

ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-cloudwatch"); RestContext cloudwatch = contextBuilder .credentials("accessKey", "secretKey") .build(); for (String regionId : cloudwatch.getApi().getConfiguredRegions()) { MetricClient metricClient = cloudwatch.getApi().getMetricClientForRegion(regionId); for (Metric metric : metricClient.listMetrics()) { // ... } }

And for the jenkins users, we also have an api for remote job and computer control, at org.jclouds.labs/jenkins!

ContextBuilder contextBuilder = ContextBuilder.newBuilder("jenkins"); RestContext localhost = contextBuilder.build(); Node master = localhost.getApi().getMaster(); localhost.getJobClient().createFromXML("newJob", xmlAsString);

This is especially helpful with the new jclouds-plugin, which uses jclouds to spin up new slaves and publish artifacts to BlobStore. Tons more in there, too.

The jclouds team have been working very hard lately, particularly on a few new apis.  We've decided to cage them no longer and cut jclouds 1.5.0-alpha.1:

Most notably, we've added the openstack-nova api, and three new providers, all of which discovered via OpenStack Keystone v2.0:

trystack-nova

hpcloud-compute

hpcloud-objectstorage

Here's how to boot up a new machine and add your login using the new hpcloud-compute provider in clojure:

(use 'org.jclouds.compute2) (import 'org.jclouds.scriptbuilder.statements.login.AdminAccess) (def compute (compute-service "hpcloud-compute" "tenantId:ACCESSKEY" "SECRETKEY" :slf4j :sshj)) (create-node compute "test" (build-template compute { :run-script (AdminAccess/standard) } ))

Here's an example of how to do the same on TryStack, authenticating w/ user & pass as opposed to key, via our java example:

java -jar target/compute-basics-jar-with-dependencies.jar \ trystack-nova tenantId:user password mygroup add

In the group org.jclouds.labs, you'll find two more new and notable members of the jclouds family:

vcloud-director: supports 250 user and admin operations defined in the vCloud Director 1.5 ReST API

virtualbox: start a group of vms on your laptop, provided an iso location for ubuntu

You can try out virtualbox like any other api.  For example, you can use the clojure above, only changing how you create the connection slightly:

(def compute (compute-service "virtualbox" "administrator" "12345" :sshj :slf4j))

The code in labs will certainly change before we release a beta, but feel free to check them out.  Meanwhile, you can try them out and give us feedback on #jclouds irc freenode or jclouds-dev google group!

Finally, many thanks to the dozen contributors who's work is in this alpha, and particularly HP and VMware for sponsoring substantial effort.

Oh, and don't forget to clean up your nodes :)

2 weeks ago, I started a bit of a rant on EC2's 32bit-only vms.

32bit vms make me angry

hey, #ec2! deprecate m1.small for a new 64bit t1.small #cloud

@jeffbar any chance of a future blog titled: m1.small is dead. long live t1.small!

This occurred on freenode and twitter, mainly due to pent-up frustration, set off by needing a separate if statement in our JDK installer just to accommodate the lame 32bit m1.small instance.

I'm not (quite) self-absorbed enough to think Jeff Barr's latest announcement was in response to this.  Even if I was, my mention never reached him! (my tweet went to the less interested jeffbar, yeah the only one 'r' one)  Nevertheless, our prayers were answered, and reported by the real @jeffbar:

EC2 Updates: New Medium Instance, 64-bit Ubiquity, SSH Client

Here's the jist: Instead of deprecating the old m1.small, they updated it to support 64bit images (and also the c1.medium).  To sweeten the deal, they also threw in a new m1.medium size, which is roughly a 2x m1.small.

Now, all users producing images don't have to make a 32bit option just cause they need more ram than t1.micro.  In fact, many will probably drop the maintenance entirely.

The 1.3 release of jclouds includes results of 3-months effort by our contributors. A total of 57 Issues were addressed between jclouds 1.2.0 and the current revision of jclouds 1.3 (1.3.1).

Notable updates include

support for Citrix CloudStack 2.2.13+

support for vCloud Director 1.5 endpoints

support for OpenStack Nova via our eucalyptus support

This release also supports more locations than ever including:

introduction of Ninefold compute in Sydney and HP Cloud Object Storage in SuperNAP (vegas)

new aws-ec2 regions in Oregon and Sao Paulo

new ElasticHosts zones in Toronto and Los Angeles

We also have a few new tricks for power users

image-id and login-user properties allow you to customize the default ComputeService template.

support for adding/removing nodes from Rackspace Cloud Load Balancers

solid state drive (SSD) support for Cloudsigma

support for aws-s3 Reduce Redundancy Storage

new property to automatically assign elastic IP addresses (handy for OpenStack Nova).

As always, we keep our examples site up to date so you can see how to work this stuff. Also, check out recent jclouds integrations including Abiquo 2.0, Apache Camel, ElasticInbox, and GigaSpaces Cloudify.

Please submit your own ideas and let us know if there are features you'd like to see, need help on, or are interested in contributing. Make sure you follow us on Twitter for updates. If you are interested in learning about jclouds 1.3 IRL, come to our training in Stockholm or our next meetup at Jfokus

Credits

Special thanks to Alcatel Lucent for sponsoring the majority of our CloudStack implementation, and the CloudStack community for answering hundreds of questions over the last year! Additional thanks to Jeremy Daggett from HP for contributing HP Cloud Object Storage support, and setting stage for further OpenStack improvements in future releases. Also thanks to Jesse Wilson from Gson, who took time with us to hone our use of Gson to the point where we no longer require patches.

Finally, thanks to everyone who contributed their time and effort in order to make this release happen. Check out who's been busy here.

More info?

For those of you in Stockholm, or coming in for Jfokus, don't miss out on a few nearby events.

8 Feb: jcloudsTraining@Citerus  - First comprehensive jclouds training, a 3-day dive into ComputeService led by Mattias Holmqvist and I

15 Feb: MultiPaaS@Jfokus - Multi-PaaS session by Andrew Phillips, including decision making in PaaS and experiences taking our Google App Engine demo to CloudBees

15 Feb: Meetup@Jfokus - Hear about what's new in jclouds 1.3, talk about cool jclouds core, integrations, and cloudy stuff in Sweden

The 1.2 release of jclouds includes results of almost 2 months effort by our community. A total of 55 Issues were addressed between jclouds 1.1 and 1.2, stabilizing the cloud so you don't have to!

We now support 33 cloud providers and reach 8 new data centers from CloudSigma, Go2Cloud, and SoftLayer.

We've made sysadminy tasks more programmable, and feel more like Java.  Using submitScriptOnNode, you can use java concurrent semantics for bash scripts!

future = client.submitScriptOnNode(node.getId(), AdminAccess.builder().adminUsername("foo").build(), nameTask("adminUpdate"));

As always, we keep our examples site up to date so you can see how to work this stuff.

Next release will be in approximately 1-months time. Look out for progress including vCloud 1.5, Voxel, and VirtualBox.

For full details on the jclouds 1.2.1 release, check out our release notes.

Catch up with us on twitter, irc, the mailing-list, or IRL at one of the many upcoming events.

Some of us hate checked exceptions, but still use them for one reason or another.  A common problem we have is unnecessary exception nesting, or runtime swallowing.  This often leads to the all to familiar and crufty code with a million catch blocks. jclouds has a slightly different approach that strikes a balance, allowing checked exceptions to be dealt with, but without the pain of so many lines of repetitious code.

try{ } catch (Exception e) { Utils.<ApplicationException>rethrowIfRuntimeOrSameType(e); throw new ApplicationException("Error applying stuff", e); }

This code does what it says, preventing us from unnecessarily nesting application exceptions or swallowing runtimes.

Now, this code shouldn't work, as current versions of java hava a generic type erasure problem.  The reason it does work is a somewhat hackish line in the rethrowIfRuntimeOrSameType method:

if (e instanceof RuntimeException) { throw (RuntimeException) e; } else { try { throw (E) e; } catch (ClassCastException throwAway) { // using cce as there's no way to do instanceof E in current java } }

The trick is that we try to force the exception we caught into the generic type. If that fails, we know it wasn't that type and that we should wrap, log, etc.

jclouds provides any-weight clouds tools for Java 5 and later: you choose the depth you want. We are pleased to announce our first beta of jclouds-s3. jclouds-s3 provides both Map<String,InputStream> and FutureCommand interfaces to Amazon S3.

It features a pluggable core, most notably supporting Apache HttpNio and Google App Engine for Java runtimes.