Code from the sideline

CouchbaseLite has been released for sometime now, so I thought it is time to give an update for using CouchbaseLite from RubyMotion.

When I ported ToDoLite-iOS to RubyMotion originally there where some bumps in the road, but it worked over all. There were for example some problems with RubyMotion not handling lambdas the way CouchbaseLite needs them, but this has been resolved since then. If you don't know what I mean by that, you can be happy and forget all about it, or read up on it.

So what do I need now to get going with CouchbaseLite and iOS?

By now it is possible to go 100% ruby for a RubyMotion project using CouchbaseLite, which is great. CouchbaseLite has been publicly released, is out of beta and has already received much love in terms of patches to make it work even more reliably cross all platforms. In case of RubyMotion this made things much easier, and the process is by now:

Add couchbase-lite via cocoapods

Tell rubymotion where to find the header files

Use it!

Installing CouchbaseLite via Cocoapods

Cocoapods is an awesome package manager for iOS and MacOS projects, and it integrates really well with RubyMotion. All there is todo is add cocoapods and motion-cocoapods to your gemfile

Now you can install any cocoapods by adding them to the Rakefile and running "bundle exec rake pod:install"

Important side not, make sure to include

as the headers are not going to be found otherwise.

Using Couchbase Lite from RubyMotion

You can now use CouchbaseLite like you would any other Obj-C library from rubymotion, for example to define a view which grabs all the "lists" in the database you can write this

I like to run JSHint to check my work when creating JavaScript for Couchbase views, but by default it shows a warning 'Missing name in function declaration'. Luckily since JSHint 1.0.0 you can ignore any warning by number so

Earlier today I ran across a segfault while trying something with cberl so I figured it's time to startup gdb and see what is going on, but how to actually do that? Luckily I found a post on the erlang programming list with the exact same problem. Since I'm on a Mac I wanted to use lldb instead so a quick modification to the erl startup script did the trick.

After having my Raspberry Pi sitting quietly in the corner for quite some time I recently heard Dave Starling talk about running Couchbase Lite on it and just had to try it out. After all having a small embedded database sounds perfect to the Pi and I can think of a couple fun projects with it, from dashboards to home automation, to the intelligent fridge ;). Also with DevoxxUK coming up, this sounded like a perfect talk subject for a quick 15min session.

So what is Couchbase Lite?

Couchbase Lite is a NoSQL database, for mobile and embedded devices. Similar to using SQLite you add a library to your project and now you have a database available to you, but unlike SQLite it provides additional syncing capabilities, and stores JSON documents instead of Rows. So what does that give you? Basically it means that you can use JSON APIs directly and store the results in the database, which makes building such apps much easier. Also having syncing done for you means that you can just work with your data locally like you always were, but have it available on every device connected, be it Android / iOS / Windows Phone and of course the Raspberry Pi, even if you are offline. This works via the so called sync_gateway which takes care of the nitty-gritty details of sync and managing conflicts as well as the connection to Couchbase to keep the data available on your severs.

Running Couchbase Lite + JRuby on MacOS from Philipp Fehre on Vimeo.

OK let's build something!

Since there is a full Linux, in my case Raspbian, running on the Pi you can develop in any language you like, but Couchbase lites portable version is written in Java, so the way to go for me was using JRuby. JRuby lends itself quite nicely to trying stuff quickly without the overhead of setting up a big project, and since it provides great interoperability with Java, using any of the Java libs is very easy.

So what are building?

As shown in the video we will create a small messaging app, which works cross platform and even if you are offline. The idea is to store all the messages in the local Couchbase Lite database and have it sync with a server somewhere on EC2, which in turn all clients subscribe to. The full final code is available on github. Just to quickly review what you need to follow along:

JRuby (http://jruby.org)

Raspberry Pi running with Raspbian or a Mac running OSX

A sync gateway instance, you can get one by signing up to CouchbaseCloud

Let's get started!

Loading in all the dependencies

Couchbase Lite has some dependencies, and we first of all need to make sure all the needed jars are loaded, for the most part those are plain java jars so there is no need to do anything platform specific, so simply loading the whole directory works fine.

There is one natively compiled dependency so which needs some special attention to be loaded depending which platform you are on. In this case detecting the OS as well as the CPU platform let's us get the right jar loaded. I compiled the couchbase-lite-java-native in for Raspbian and the linux-arm platform, as well as for MacOS so the application runs on either and load depending on which platform the app currently runs on.

Compiling for additional platforms is simple as well, just checkout the code and run $ gradle jar, in case of the Raspberry Pi you might need to adjust the build.gradle file according to the crosscompile-build.gradle and include the paths for the compiler as well as the path to jni.h. Take a look the compiler settings in this case

And make sure it finds the required JNI header

With this out of the way we can start using Couchbase Lite in our application.

Initialize the application

Using the database from within the application requires some setup during the initialization.

We need to get access to a JavaContext first, which is similar to an Android Application Context, and is needed to provide a basic information for the whole application. This contains elements like the directory for data storage and alike. We also need to make sure to get the java String object from a URL string used as the location of the Sync Gateway, which we will talk about shortly. Lastly the manager is used to acquire the actual database instances, and handle access to them. Getting access to the Database via the manager is done by simply getting the database with a given name, in case the database is not present already it will be created for you and stored on disk in the data directory, which defaults to the current application directory.

Setting up the sync

The ease of sync is one of the big advantages Couchbase Lite has, so let's set it up. First we need some backend to sync with. If you just want to try it out take a look at Couchbase Cloud which let's you setup a sync_gateway with just a few clicks, or just run it locally, by downloading form Couchbase Mobile, soon sync_gateway will also be available in homebrew for MacOS. After this is done just grab the URL and point the app to it.

Now all is left is to create two replications one to pull data from the remote whenever there is a change, and one to push our own changes up.

And finally we need to handle any changes which get passed along, this is done by implementing a so called ChangeListener Interface which just needs to implement one method void change(ChangeEvent event) which gets called for every change being replicated. I our case we just log the change and update the UI as needed. Since this class is implementing a Java Interface we also need to include some annotations for JRuby to handle the interfacing with Java correctly, mainly annotating the method with the type signature required.

And with this we are now ready to create and read the data.

Reading and writing data

When writing to couchbase lite we need to create a document first and than save the data to it.

In this case we create a document in the given database with the provided text. The type of the document is set to message. The type is no special property but it is common to set it because it allows us to filter more easily later as there are more different types of documents stored in the database. Now to read the data out we could either do so via the document id which is assigned on creation or by creating a query. Since we want to display all the documents it makes sense to create a query which reads all the documents and stores them in an array.

Couchbase Lite allows for more advanced queries using Map Reduce and also for live updating queries, but for this simple application this approach will be sufficient.

Displaying the results in the UI

Now that everything is in place we can create a UI using Swing to actually provide us with the needed interactions. This is rather simple so I'm not going to walk through all the code, the important part is to setup a listener to actually create new messages as the user presses send.

And adding rows to the created table as the new data rolls in

Thats it a quick and simple messaging app, runnable on a Raspberry Pi, MacOSX and basically anywhere there is Java.

Conclusion

Getting up and running with JRuby and Couchbase Lite on the Raspberry Pi is really simple, the JVM comes pre-installed and JRuby is just a download away, this opens up the Raspberry Pi to a nice set of applications with the ability to operate on the data locally without having to worry about the syncing yourself. I'm already looking forward to some home automation, dashboards and alike interfacing with my iPhone or Android Tablet, maybe I finally get my intelligent fridge? ;).

Resources

Calling Java From JRuby

Couchbase Lite

GPS Logger PI

API documentation

Recently Couchbase Lite 1.0 got released, and you can find anything about it at the Couchbase Mobile Developer Portal. As I've been quite interested in RubyMotion since some time, I decided to check how easy it would be to take CouchbaseLite for a spin on RubyMotion, and I have to say it is quite easy. Besides some small problems with RubyMotion handling of blocks everything worked as expected. The full project is up on github for you to checkout, but I'd like to walk through some of the parts worth noting here.

Getting the database ready

First things first, opening a database file for CouchbaseLite works as expected simply porting over what is presented in the examples.

This database object can now be passed down to wherever it is needed. So now is the time to get onto one of the big features of CouchbaseLite, which is sync. To get this working we authenticate against Facebook using ACAccountStore

and feed the credentials back to CouchbaseLite to authenticate any replications, which are setup.

And with those little pieces of code we actually have a working sync setup.

Data in...

A syncing database is nice, but without data in it also kind of useless ;), so let's get something in there. CouchbaseLite actually provides a nice layer on top here, letting you define models for your data which you can then read and write like you would with any ORM. And it's more than that, you can actually use those as well to directly bind to a view, but more about that later. First let's get some data in there, by first creating a model to represent this.

So all there is left to do is creating a new object of this kind and save it.

And that's it, we can now store items in our database fairly simple.

... and data out

With this we can take advantage of one of the other big features of CouchbasLite, at least to me, which is using CouchbaseLite as an TableViewDataSource. This cuts out huge amounts of boilerplate code, so

will get everything setup, and make sure that the display is always up to date.

One more (sad) thing...

Sadly right now, due to a bug in the handling of Procs in RubyMotion, it does not allow you to setup map and reduce blocks via Ruby. This will probably be resolved by Rubymotion soon, but for now it means you have to setup the blocks from Objective-C. But don't worry it's not hard, basically it's just the code for the block itself which is pretty easy to grasp.

In TodoLite-Motion I set it up to be a vendored Xcode project, so it can easily be compiled and changed when needed.

The end

Thanks to @trondn it is easier than ever to build couchbase from source. So I decided to take the new build system for a spin on my Macbook based on the documentation provided.

Getting all the dependencies

First of all if you are not yet using homebrew already, start doing so it will make your live a lot easier throughout the world of open source tool chains. Ok let's get started, by installing all the needed dependencies

Ok now let's see what all this is

repo is a tool which ties together multiple git repositories so we can use them as one big project. It it also reponsible for the whole code review process when used in conjuction with gerrit

v8 is googles JavaScript engine, know for being amazingly fast. It is used in Couchbase server to execute JavaScript.

snappy is a compression library optimized for speed. This is used internally to compress the data written to disk.

icu4c unicode library to handle all the unicode needs.

google-perftools provides TCMalloc a faster malloc implementation which is used whereever malloc is needed.

erlang a programming language build for highly concurrent systems. In Couchbase this is used in multiple places but especially the cluster managing.

libevent event library used internally in couchbase.

cmake cross platform build system to make it easy to build on multiple platforms native build systems. Couchbase uses this now to generate the build files needed on either Windows / Mac and Linux.

git Version control system.

This covers the basics now it's time to get the source and start building!

Getting the source Couchbase

As metioned before Couchbase relies on repo to manage all the interdepended repositories. So to get the source in the right versions you need to get the manifest for the version of Couchbase you want to build so repo can get all the right source for you. This is easy enough, simply create a folder for the source and setup repo in there in this case with the master branch.

And finally we get the source by telling repo to sync our local copy

That's it time to build

Building Couchbase

Building the software itself is now as simple as running make

Afterwards couchbase should be ready to run

and you can visit the webinterface for all your configuration needs.

The End

Getting up and running with PHP on Windows has always been quite easy due to XAMPP. It provides everything needed to get started quickly and build something using a default stack available everywhere. A lot of people get (or got) started this way, including me. But now that MySQL, PHP, and Apache are setup maybe it's time to try something new. For me it's been a while since I ran a setup like this and so I thought it's worth documenting the process.

Getting all the pieces

First of all you'll need:

XAMPP for all your PHP and Apache needs, so install at least PHP and Apache

Couchbase, remember that you can try out the Enterprise version for development for free.

Couchbase-PHP-SDK XAMPP uses thread safe PHP so make sure you download the correct version (either x86 or x64 Thread Safe) for your environment.

Getting everything installed

For Couchbase and XAMPP you basically follow the instructions on the screen to get it up and running, and make sure you configure Couchbase as well at the end by visiting the web interface. Now all there is left is moving the PHP stuff into place, which is probably the most complicated part of a really easy setup. The zip downloaded from the windows PECL page contains 2 dll files, libcouchbase.dll and php_couchbase.dll which need to be moved in the right place. First to make php find libcouchbase it's needs to be present c:\\apache\bin\libcouchbase.dll and to make sure you can also use php from the command line also copy it to c:\xampp\php\libcouchbase.dll and add the XAMPP PHP to your PATH by going to Control Panel -> System and Security -> Advanced System Settings -> Environment Variables and edit the PATH variable by appending c:\YOUR XAMPP DIRECTORY\php. And that's it you are now ready to use Couchbase in XAMPP, by adding the extension to php.ini, so just add

Try it out!

Create a new PHP file at c:\YOUR XAMPP DIRECTORY\htdocs\cb-test.php containing

and open it in your browser to make apache run it you should now see the result in your browser.

you can also use the command line of course.

The end!

OK so you are now ready to use Couchbase from PHP, and if you need any help and are in London this week stop by the Couchbase Developer Day as part of Big Data Week, or even if you don't still you should still stop by.

EDIT

If you are like me, you like your code and at best everything else about your app to be versioned. This makes setting up a new development environment quick, and even more important reproducible. So one thing which always bothered me was setting up external code outside of the application to generate Couchbase views, since those views are build using JavaScript, and by default live in the web interface, far away from my application code. If you want to know more about them watch the Couchbase 103 - Views and Map-Reduce by the way.

Since those JavaScript functions are code which is closely related to my app, I like to keep it inside as well, and provide a task to actually set them up for the given environments. The following quick gist provides a rake task to do this, adding all views in config/couchbase_views to Couchbase.

Homebrew is my favorite package manager on the Mac, sadly there is no package to setup and install the Couchbase sync_gateway which is used for interaction between a Couchbase Server and a Couchbase Lite mobile database. Since this is easily fixed, I created a formula for sync_gateway. Currently the Pull-Request is still open so if you like to have this in homebrew make sure to let the homebrew maintainers know. For the time being you can install it by referencing my fork of homebrew directly in the brew install.

Last friday I talked about schema migrations in NoSQL databases, and how to deal with ever changing requirements. The slides are online the video will be soon.

You got schema in my json from Philipp Fehre

Ok so I got a Kindle Fire and since there is no way to sync my Safari Readinglist over I decided it is time to move to something "portable". Luckily it is quite easy to get a list of all the current Safari Bookmarks, which includes everything on the Safari Readinglist, the basics are in a Gist by andphe

So all there is left to do is use the readit gem to get everything into Readability, using oauth to authenticate first. Overall this went pretty well and quickly, so for anybody interested you can find the scripts on github.

Yesterday I read about calling C libraries in Haskell, especially how the Haskell foreign function interface works. Never had to do this in Erlang I decided it would be a nice exercise, so here we go wrapping Peter Scotts Murmur3. The end result is available on github for anybody interested. Also I am not an Erlang so take everything with a grain of salt.

Looking for FFI in Erlang

There are actually multiple ways to call out from Erlang, you can use Ports and actually communicate with an external process via passing messages which is nice and save, or you can implement wrappers in C and use them as NIFs. NIF stands for native implemented function, so the library is actually dynamically linked to Erlang which can be dangerous since an error in the library can crash the Erlang VM, but it is also the fastest way to call out to C land. Since I was interested in the closest thing to FFI really I decided to create a NIF wrapper.

Exploring Murmur3

The C library for the Murmur3 hash function actually consists of 3 functions, 2 optimized for x86 and one for x64, but since they all work on any architecture it made sense to wrap all of them. All functions look really similar.

void MurmurHash3_x86_32 (const void *key, int len, uint32_t seed, void *out); void MurmurHash3_x86_128 (const void *key, int len, uint32_t seed, void *out); void MurmurHash3_x64_128 (const void *key, int len, uint32_t seed, void *out);

The first argument is the thing to be hashed, being of length len. The hash is also seeded so passing in the seed value is mandatory. After it is done the result will be placed in out, which is either 32 Bit or 128 Bit in size.

Setting up the project

Using rebar has proved to be the easiest way to setup a project.

$ mkdir murmerl $ cd murmerl $ wget http://cloud.github.com/downloads/basho/rebar/rebar && chmod u+x rebar $ mkdir src $ touch src/murmerl.app.src $ touch src/murmerl.erl

murmerl.app.src is really just a really basic app file containing only the following.

Now all we need is a place for storing the C code, and put the Sources in.

$ mkdir c_src $ cp <Download>/murmur3.c c_src/ $ cp <Download>/murmur3.h c_src/ $ touch c_src/murmur3_nif.c

After telling rebar about the C sources the project can be compiled and run.

$ rebar compile

Which will create priv/murmerl_drv.so as well as the beam and app files in ebin.

NIF in the Erlang world

NIFs behave like any other function from a callers perspective, so we need to define a module for them which is in charge of loading the library as well as provide some fallback if loading fails. Providing fallbacks is done via providing alternative function implementations.

To make it now work we need to load the library via the on_load for the Erlang module and make sure all the functions are exported as well.

Thats it for the Erlang side of things, now for C.

NIF in the C world

Everything needed to bind everything together is present in erl_nif.h which needs to be imported. Each of our functions looks really similar, they return an Erlang Term, and take the current environment, argument count and arguments array. Pulling the arguments apart needs to be done inside the function so this is the first thing.

Erlang provides a lot of functions here we expect something string like as the first argument so we pull the first argument inside the in struct and extract the second as an int. The actual return value will be a binary so we need to allocate one of the right size, 4 to get 4 Bytes = 32 Bit. Since this is just a piece of memory, it is best to zero it out before using. Now all there is to do is let MurmurHash3_x86_32 write to the allocated chunk and return as a Erlang Binary for everybody to use.

To make the functions available to Erlang we need to tell the VM about them, this is done here

Which exports the nif_funcs array containing a structure with

{ name_in_erlang, arity, c_function }

for each of the exported functions. And thats it the functions can now be called from Erlang like any normal Erlang Function in the module. For the full code see the github repository.

Where to go from here?

Currently all the module does is forward the calls as is to the C library, which does not feel very Erlang like, so building a nicer API is up next.

Resources

One of the plugins for vim I rely on heavily is YouCompleteMe. It really is amazing but there is one small problem as it generates completion error messages all the time while typing.

There is a fix for it which sadly has not landed in Vim yet, but as detailed in the issue on github it is easy to patch it yourself. Since I installed vim via Homebrew the way to go was to add the patch to the Homebrew formula and go from there. So this is what I did. First I don't like to work in the Homebrew master so I created a new branch

$ cd `brew --prefix` $ git checkout -b add-ycm-message-silence-patch $ vim Library/Formula/vim.rb

Afterwards it's time to put the new patch in the formula, at the bottom of the file after __END__. The method

def patches; DATA; end unless build.head?

will apply all patches preset in the DATA section, which is the part of the file after __END__. I created a version in my fork of homebrew.

And low and behold after a reinstall

$ brew uninstall vim $ brew install vim --override-system-vi \ --with-client-server --with-lua \ --with-mzscheme --with-perl --with-tcl

The annoying errors are gone.