A Day in the Code...

For years, I've wondered why the following fails:

def test_this(*args) case args.first.class when Array # do some Array stuff when Hash # do some Hash stuff else # do other stuff end end

Instead, I've always had to fall back to this:

def test_this_instead(*args) case args.first.class.name when 'Array' # do some Array stuff when 'Hash' # do some Hash stuff else # do other stuff end end

It made no sense to me. But finally I found this very useful post

The piece that caught my attention at the end of the post:

Every object in Ruby inherits a method called the case equality method, also known as the triple equals (===).... For built-in objects such as strings, the === operator/method is simply equivalent to the == method/operator.

After reading this, I jumped into irb and tested the following:

irb(main):001:0> [] === Array => false irb(main):002:0> [].class === Array => false irb(main):003:0> [] == Array => false irb(main):004:0> [].class == Array => true irb(main):005:0> Array === Array => false irb(main):006:0> Object === Object => true irb(main):007:0> Enumerable === Enumerable => false irb(main):001:0> Array.ancestors => [Array, Enumerable, Object, Kernel, BasicObject]

Interesting. Enumerable must overwrite === such that classes that extend this module are no longer case equal.

In my previous post, I mentioned that I was creating our schema.rb on a Linux version of MySql.

I originally tried to have my local version of MySql run with its data directory located on a Mac Disk image that was case sensitive. However, I found that to be too brittle. If my system needed to be forced powered-off for any reason, the image would break, and I would have to set a new one up from scratch. With VMWare's Fusion package, I've only had that happen once when my disk had become corrupted.

However there are a couple of things that have to be configured in order to make this work effectively.

VMWare Fusion

First make sure that your VMWare ubuntu image is using NAT (Share with my Mac) in order to access the network to download MySql.

Get your ubuntu IP address then ensure that you can ping it from your Mac. In an ubuntu window run:

ifconfig

And look for the IP address for the eth0 interface. While you have the window up, note the HWAddr of the same interface. We'll need it later.

Then in your Mac Terminal, ping that IP address:

ping 172.16.23.131

MySQL Configuration

First use the package manager to download and install MySql.

sudo apt-get mysql-server

You also have to enable the ubuntu version of MySql to accept external connections. Otherwise you can not connect to it from your Mac.

Edit your `/etc/mysql/my.cnf`` and comment out one line:

bind-address = 127.0.0.1

Restart MySql

services mysql restart

Run the mysql client from inside an Ubuntu terminal and create a user to have access from an IP address other than localhost:

mysql -u root -p <Enter your root password> mysql> GRANT ALL ON test_development.* TO 'rails_user'@'%' IDENTIFIED BY 'some_password'; mysql> FLUSH PRIVILEGES;

Verify that you can access the MySql server from your Mac Terminal:

mysql test_development -h 172.16.23.131 -u rails_user -p <Enter the rails_user password> mysql>

Just one more thing to do! This works as is, but quickly becomes VERY annoying because the VM will periodically refresh its IP address. Out of the box, a VMWare image uses DHCP to dynamically grab an available IP Address. However if this is going to be our development database, we would really prefer to use a FIXED IP address.

Setting a Fixed IP Address for Your Ubuntu Image

VMWare's site isn't very forthcoming for the Mac configuration, but I found a very useful post from Andrew Elkins.

Remember that HWAddr from running ifconfig inside an Ubuntu terminal? We are now going to use that here.

sudo vim /Library/Preferences/VMware\ Fusion/vmnet8/dhcpd.conf

Locate the following line:

####### VMNET DHCP Configuration. End of “DO NOT MODIFY SECTION” #######

In my file, it was at the very bottom. Note that inside the DO NOT MODIFY block that the subnet has a range defined. This is the range of IP addresses that DHCP draws from for dynamic IP assignment.

Select an IP address inside the subdomain, but outside the specified range. For example, 172.16.23.55. Below the DO NOT MODIFY LINE define a host using your HWAddr and your chosen IP Address.

host ubuntu64 { hardware ethernet 00:0c:29:78:56:05; fixed-address 172.16.23.55; }

Finally restart your Mac. When you open your Ubuntu Terminal and use ifconfig you should see the following:

Rails Configuration

Edit your config/database.yml file and set the following:

development: adapter: mysql2 reconnect: false pool: 5 host: 172.16.23.55 username: rails_user password: some_password database: test_development test: adapter: mysql2 reconnect: false pool: 5 host: 172.16.23.55 username: rails_user password: some_password database: test_test

Recently I ran into a perfect storm that caused many hours of work to understand, track, and fix in order for me to run one simple command:

rake db:test:prepare

Here's the setup:

I have a MacBook Pro

This is a legacy MySQL database originally built by Java developers ~for~ Java developers (So older tables are mixed-case)

We use foreign-key constraints

Since we still have multiple applications (written in multiple languages) accessing the same database, we keep the foreign-key constraints to maintain some minimal consistency across our application. One of engineers decided to use the schema_plus gem in order to generate foreign keys. (SchemaPlus provides ActiveRecord support for foreign keys, database defined validations and associations).

So here's where the cascade starts occurring:

The default filesystem (HFS+) on the Mac is actually not case-sensitive. Which does not make it the ideal filesystem to use mixed-case table names. You can read all about it here. Windows is even less case-sensitive, which might actually make it easier. For the Mac however, the behavior is just strange enough that when you create a mixed-case table, MySQL successfully creates a file that is mixed-case. HOWEVER, add an index after the initial create, and suddenly the table file is lower-cased.

During the building of the schema.rb file, the schema_plus gem polls the database for tables in order to start recording which foreign_key calls to make. It creates a hash of relationships keyed from the table name. As it find the pattern of <table_name>_id fields, it notes a potential FK relationship against the specified table.

Before this occurs, all of the migrations have run, which includes those migrations containing the stray indices any group misses after the initial creation.

So...

By the time schema_plus pulls the list of tables from MySQL, certain tables have been recast to lower case.

Because all of the foreign keys reference an upper case table name, schema_plus attempts to run .each on a table query that returns nil instead of minimally an empty array. So an error is thrown at the end of rake db:migrate.

Let's say that you decide to remove foreign key definitions that reference your mix-case tables. (A suboptimal solution for sure). You can now run rake db:migrate successfully, but you can not run rake db:test:prepare if you have any indices defined on mixed-case tables.

We have done a few things to mitigate the problem. First we replaced schema_plus with foreigner (https://github.com/matthuhiggins/foreigner). This allows us to explicitly set the table names in our foreign key definitions and has far less magic behind the scenes.

Second, we make sure that the schema.rb file that we check in is the result of running migrations on a Linux copy of mysql so that we preserve case in our table names.

Finally, we are moving to a service-oriented architecture so that only one codebase accesses the database directly. Everything else uses the services.

I've been meaning to start blogging for quite a while, but I just couldn't think of something to blog about. Now I've started a new job working in a new problem space (for me), and I thought I should at least capture some of the experience.

A New Problem-Space

After approximately seven years working in creating financial systems for startups using PHP (don't ask), Java, and finally Perl/Ruby, I am finally at a new job where my first goal is to design and implement a system handling activity streams. I am very excited to be working on something that is so different. This involves creating a real-time system that will scale and be fault-tolerant. It also takes me out of my comfort zone of using Ruby on Rails for the first time in over four years.

How did I approach the task? First I googled for any presentations about creating architectures for activity streams. It may be a new problem for me, but it certainly isn't new in the industry. Over the course of a couple of weeks I found the following presentations to be extremely helpful:

Social Networks and the Richness of Data

Etsy's Activity Feed Architecture

Riak and Scala at Yammer - offered at a Riak meetup, but still very informative

I especially liked the first presentation and must say that it informed a lot of my initial design. To work quickly in a distributed, decoupled environment, I realized I needed the following elements:

A method for generating unique ids across a distributed stack. I could use GUID, but I had seen ways to reduce the size of the id, so I investigated those.

A store that wrote quickly, could retrieve quickly and had all of the latest buzzwords: highly available, highly fault-tolerant, easy to scale.

A message queue that was fast but was also fault-tolerant, which meant some form of persistence.

And finally, a framework that was light-weight, scaleable, and reasonably fast.