Schedule recurring jobs with Sidekiq 4

Something I keep getting back to is setting up scheduled recurring jobs with Sidekiq, even though some people say it's not the tool for the job, and you might be better of using Cron, I do like the simplicity of using tech that is already there!

With previous versions of Sidekiq, I used Sidetiq, which was super easy to setup. But it does rely on Celluloid, and that is one of the things that is removed in sidekiq 4. I had to start looking for an alternative.

As it turned out, there are a couple of alternatives. In this article, I want to talk about Sidekiq-cron, since this is the tool I'm currently using on some projects.

Sidekiq-cron

This tool has a bit of a confusing name. When I first saw the Github repo I believed it was using Cron, and closed the tab right away (Yeah I know, RTFM). It was when a friend of mine pointed out to me that it was not using Cron in any way but just the syntax; it got back my interest!

Pros

  • Getting it up and running is super simple
  • It has a web interface that hooks into the Sidekiq-web dashboard
  • You can use YAML for scheduling
  • It is compatible with ActiveJob

Cons

  • If you're using a web server like Unicorn you may run into problems with forking processes.

Getting it up and running

First you need to add the gem to your gemfile:

gem "sidekiq-cron", "~> 0.4.0"  

For this example, I'll be using the following job with the name ExampleJob

class ExampleJob < ActiveJob::Base  
  def perform
    # do scheduled expensive work
  end
end  

Then you need to create your first recurring job, which can be done in two different ways, lets first take a look at the "easiest" way. Which is manually scheduling them:

Sidekiq::Cron::Job.destroy_all!  
Sidekiq::Cron::Job.create(name: "Example Job - every 5 min", cron: "*/5 * * * *", class: "ExampleJob")  

You can, for example, create an initializer called config/initializers/scheduled_jobs.rb and add a few lines of job creation in there. The downside of this, you have to manually make sure old scheduled jobs are removed before adding new ones.

However, there is another way! You can also load all the scheduled jobs from a YAML file, and it will take care of removed and unique jobs!
You still need to create an initializer

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|  
  schedule_file = "config/sidekiq_schedule.yml"

  if File.exists?(schedule_file) && Sidekiq.server?
    Sidekiq::Cron::Job.load_from_hash! YAML.load_file(schedule_file)
  end
end  

Then we need to create the sidekiq_schedule.yml file:

schedule_example_job:  
  cron: "*/5 * * * *"
  class: "ExampleJob"
  queue: default
  active_job: true

When you now start your rails app, sidekiq-cron will check what jobs are currently scheduled, and if needed add jobs. Because we used load_from_hash! (With a bang) in the initializer, all jobs that are scheduled but not present in the YAML file will be removed.

Getting the web UI to work

If you're using Sidekiq's web UI, you can also leverage a nice web UI to see what jobs are being scheduled, you can enable this by adding require 'sidekiq/cron/web' after require 'sidekiq/web'

Questions?

If you want to know more about the ways I'm using this, or just general questions, you can ping me on Twitter: @jvanbaarsen or shoot me an email: jeroen@firmhouse.com