#simpleworker

After using SimpleWorkr for a long time now, I always think to myself, "there's gotta be some way to make this more intuitive". I want it to feel more like it's just like running your normal code without even really realizing that the work is actually offloaded to the cloud. So with that in mind, the gem now has a much more intuitive usage pattern, more like ActiveRecord. The old way:

# Create a Service object @service = SimpleWorker::Service.new(@access_key, @secret_key) # Upload code @service.upload(worker_class_name, path_to_file) # Queue up jobs @service.queue(worker_class_name, data_for_job) # Or schedule jobs @service.schedule(worker_class_name, data_for_job, schedule) # Get status of jobs @service.status(task_id)

Now, here's the shiny new way of doing it: Some one time initialization:

SimpleWorker.configure do |config| config.access_key = @access_key config.secret_key = @secret_key end

Now throughout your app, you can do the following:

# Instantiate a new worker worker = MyWorker.new # Queue directly from the object worker.queue # Schedule directly from the object worker.schedule(my_schedule) # Check status directly from it as well worker.status

Previously, your worker file/class had to be self contained (ie: single file including all code for the worker). Now you can use the new "merge" class method for instance:

class TestWorker2 < SimpleWorker::Base merge "models/model_1.rb"

This will automatically upload the merged files to the SimpleWorker service and 'require' them in your Worker class. If you want to use the special methods like "log" in your merged classes, use the SimpleWorker::UsedInWorker mix-in, for instance:

class Model1 include SimpleWorker::UsedInWorker

That's about it, the rest will be taken care of for you. If you want to kick off other workers from within the first worker, use merge_worker instead:

class TestWorker2 < SimpleWorker::Base merge "models/model_1.rb" merge_worker "my_other_worker.rb", "MyOtherWorker" # second parameter is the class name

So I'm sure we've all had the need to want to run multiple threads at once to optimize a part of our applications. There are several options like spinning up new Ruby threads (example from):

pages = %w( www.rubycentral.com www.awl.com www.pragmaticprogrammer.com ) threads = [] pages.each do |page| threads << Thread.new(page) { |myPage| h = Net::HTTP.new(myPage, 80) log "Fetching: #{myPage}" resp, data = h.get('/', nil ) # DO SOME PAGE PROCESSING AND STORE STUFF IN DATABASE } end threads.each { |aThread| aThread.join }

Or you could use a library like delayed_job that works great after a bit of database configuration and starting up the job process from the command line.

But now there is another option that is actually easier and much more scalable and that is sending jobs off into the cloud using SimpleWorkr. Here is an example of the above using SimpleWorkr:

# First we create our Worker class WebPageWorker < SimpleWorker::Base attr_accessor :url def run h = Net::HTTP.new(myPage, 80) log "Fetching: #{myPage}" resp, data = h.get('/', nil ) # DO SOME PAGE PROCESSING AND STORE STUFF IN DATABASE end end # Now lets fire some of those workers up! pages = %w( www.rubycentral.com www.awl.com www.pragmaticprogrammer.com ) pages.each do |page| worker = WebPageWorker.new worker.page = page worker.queue end

SimpleWorker is a job queuing and scheduling system so while a lot of work comes in at random times usually based on some event in their system (ie: a user clicks a button), there are a lot of scheduled jobs too and those scheduled jobs have a huge affect on the capacity requirements for SimpleWorker. Check out this graph of CPU usage for the past 24 hours across the entire SimpleWorker server farm.

The big spikes start on the hour every hour and this is because most developers schedule their recurring jobs on the hour, not somewhere in the middle of the hour. It would be much better for us if they did so it could spread the load evenly and we could make better use of the unused time and perhaps we will provide incentive to do that at some point, but as it stands, we always have to have enough capacity for the big spikes.

SimpleWorker will now include all the models in your Rails app automatically so you don't have to merge them manually. Just use them almost as if they were to be run locally and you're good to go. eg: