#activejob

Sitepoint has a fantastic 3-part tutorial series on Ruby Background Processing Libraries, which gave me a bit of inspiration to write my own post on the topic. I’ll be using a few different resources, namely:

Sitepoint’s tutorial

Jumpstart Lab’s background jobs tutorial

Ryan Boland’s article on writing your first background worker

“Resque vs Sidekiq” Stackoverflow question

Ruby Background Processing Libraries

First, let’s discuss what background processing is:

Wikipedia gives the broad definition as: “a computer process that runs "behind the scenes" (i.e. in the background) and without user intervention.  Typical tasks for these processes include logging, system monitoring, scheduling, and user notification.”

But as this blog is centered around Ruby on Rails, I want to give a more RoR-specific answer. I’ll also be making a comparison to nightclubs to help dumb the explanations down a little bit.

Why would I need background processing?

To keep response times down on your application - which is done by delaying certain computations to be performed later.

A few examples may be:-

Sending an Email - this could take a long time and it’s definitely not something that you want to interrupt your user with.

Data Processing (image upload, video upload, etc) - because it takes a long time and you don’t want the user waiting!

Maintenance - Regular tasks such a clearing out any expired sessions

3rd Party APIs - If your app wants to interact with an outside service - because this process is risky and could fail (e.g. if the server of the outside service is offline).

Super quick summary on how background processing works:

Background processing consists of jobs in queues, with a first-in-first-out policy - exactly the same as when humans queue for something. But, like humans, jobs need someone to process them, like a nightclub bouncer - the nightclub bouncer in this scenario is called a “worker”. Workers process the jobs at a later time.

Types of Background Workers

A background worker that runs immediately after an action is performed. This is the bouncer that throws you out immediately after you start a fight in the nightclub.

A background worker that runs a certain amount of time after an action is performed. This is the bar staff that serve you ten minutes after you’ve been waiting at the bar.

A background worker that runs regularly at set intervals. This is that awful song at the nightclub that gets replayed every fifteen minutes.

So, how do you make sure this “queue”, with its jobs and its workers, is performing well?

INTRODUCING... SIDEKIQ + RESQUE (TO THE RESCUE!) (or ‘the nightclub manager’)

Sidekiq

Definition from their official Github page:

“Sidekiq uses threads to handle many jobs at the same time in the same process. It does not require Rails but will integrate tightly with Rails to make background processing dead simple... It is compatible with Resque and can run side-by-side with Resque.”

--> Link to their Github

Resque

Definition from their official Github page:

“Resque is a Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later. Background jobs can be any Ruby class or module that responds to perform... Resque comprises three parts: - a Ruby library for creating, querying, and processing jobs; - a Rake task for starting a worker which processes jobs; - a Sinatra app for monitoring queues, jobs, and workers.”

--> Link to their Github

To see a comparison of the Sidekiq VS Resque, I recommend this link to Stackoverflow.

Of course, there are alternatives to Sidekiq and Resque, such as Delayed job, and each have their pros and cons so it’s completely up to you and what is most suitable for your application.

So what is Ruby on Rails’ “ActiveJob” for?

Ruby on Rails edge guides has a great tutorial on this: http://edgeguides.rubyonrails.org/active_job_basics.html

But let me explain its purpose briefly (text in italics is from their edge guide):

“The main point is to ensure that all Rails apps will have a job infrastructure in place. We can then have framework features and other gems build on top of that, without having to worry about API differences between various job runners such as Delayed Job and Resque. Picking your queuing backend becomes more of an operational concern, then. And you'll be able to switch between them without having to rewrite your jobs.”

You use ActiveJob to:

Create a job

Enqueue a job

Obviously, you then need to execute the job:

“For enqueuing and executing jobs in production you need to set up a queuing backend, that is to say you need to decide for a 3rd-party queuing library that Rails should use... Active Job has built-in adapters for multiple queuing backends (Sidekiq, Resque, Delayed Job and others).”

Conclusion

So hopefully I’ve explained background processing well enough for you, but if you’ve got any questions then feel free to send me a message.

Today I was reading a random post on Stackoverflow about recurring jobs using the new ActiveJob framework.

Antarr Byrd was trying to reschedule a job using the after_perform callback

I read an answer from bcd who said:

But that's not a reliable way of doing it as if an exception occurs the chain breaks. Also you must have the initial job scheduled. If you use resque you can use activejob-scheduler

So I learned about ActiveJob Scheduler which claims to be "A background job scheduler for any queue backend".

ActiveJob::Scheduler

Unlike bcd said, it is actually possible to enqueue jobs with any queuing adapter, not only resque.

The project is really great, and it actually lets you schedule jobs even with ActiveJob's default inline adapter!

It uses rufus-scheduler under the hood, so you can even directly use rufus to delay a job with the inline adapter.

As bcd said, one great advantage of this solution is that it will reschedule the job even if the last run failed, which is not the case of the after_perform callback.

As I'm not writing an apology of ActiveJob-Scheduler, here come the downsides.

This is a lie.

Like its predecessors, ActiveJob::Scheduler is built with Rufus::Scheduler, an immensely powerful task scheduling library to make sure your jobs get kicked off at exactly the right time.

I've worked with a lot of different programming language including ruby but also some low level stuff like C and assembly. I don't know any solution to actually run something at exactly the specified time, do you?

On a single process operating system, you can maybe do it, but not on any common multi-processing OS we use everyday. That OS will let your process run once in a while, not when you request it.

This gem is great, but not magic, so please don't lie.

Scheduling in-memory? No, thanks

So ActiveJob-Scheduler uses rufus-scheduler, "an in-process, in-memory scheduler".

Great project too, but for scheduling jobs? Really? How about I want to schedule an email in 1 month but I update my app every week?

At every update, the application will be restarted, so a new process will be spawn with his own memory. As an in-memory scheduler, rufus will lose every information.

That's not a problem for a lot of cases, but when you want to schedule a job you may not want it to disappear at every update, and even less on unexpected restart!

What about my queue adapter?

When I started using ActiveJob, I took a look at the different adapters available and choose one according to its features.

If you use ActiveJob::Scheduler, you can pick any adapter is really does not matter is the end, because all the scheduling in done by rufus, not by your adapter!

So you may as well use the inline adapter, it will run your job directly at the desired time, instead of doing unnecessary work first (sending to the backend which will in most case run it directly).

Your adapter persists delayed jobs across restarts? No problem, ActiveJob-Scheduler will not use this feature and won't tell you unless you try to look how it works!

Initial scheduling

ActiveJob-Scheduler initial scheduling is great, you can simply define your jobs in a YAML file.

But then, you have to launch a rake task or an a binary to start it!

The schedule must run as a separate process, but it's very light...it delegates all the real processing to your queue workers, and simply enqueues jobs as the specified time rolls around.

This means you have one more worker just for the scheduling, this is insane! Okay, it won't take much ressources, but it'll still inscrease the worker number, and consequently the bill...

Useless in most cases

If you use a backend which supports delay (and you should), then ActiveJob-Scheduler is useless.

You can schedule initial jobs on worker start (check if it does not exists first maybe if you can).

after_perform is great, and great to reschedule job even if it does not get called on error. Why would you reschedule a job which just failed?

If you still want to do so, then rescue_from error to also reschedule or simply ensure that the job is rescheduled.

Whatever way does your backend works, it won't be restarted if you update your app, so even in-memory persistence is okay. Some backends also support file persistence.

It does not need an additional worker to run.