WatchCron

I did an audit on a client's server last year. Forty-seven cron jobs. Twelve of them hadn't run successfully in over a month. Three were pointing at scripts that no longer existed.

How does this happen?

Usually it's gradual. Someone deploys a new version and the script path changes. Or a dependency gets updated and breaks something. Or the disk fills up and writes start failing.

The job throws an error, cron sends an email to a mailbox nobody checks, and that's it. Weeks pass. Months, sometimes.

The backup job is the scary one. You think you have nightly backups. You haven't had a backup in six weeks. You find out when you actually need to restore something.

A quick audit you can do right now

Pull up every cron job on your servers:

# List all user crontabs for user in $(cut -f1 -d: /etc/passwd); do crontab -l -u $user 2>/dev/null | grep -v '^#' | grep -v '^$' | while read line; do echo "$user: $line" done done # Don't forget /etc/cron.d/ and /etc/cron.daily/ ls -la /etc/cron.d/ ls -la /etc/cron.daily/

For each job, ask two questions:

When did this last run successfully? (Check logs, output files, or database timestamps)

If it stopped right now, how long before someone noticed?

If the answer to #2 is "I don't know" — that job needs monitoring.

The dead man's switch pattern

Instead of monitoring whether a job fails (hard to do reliably), monitor whether it succeeds. Add a ping at the end of the script:

#!/bin/bash # do the actual work rsync -a /data/ /backup/data/ # signal success curl -fsS --retry 3 https://monitor.example.com/ping/job-id

If the ping doesn't arrive within the expected window, something's wrong. You get notified. Simple.

Took me about an afternoon to add monitoring to every cron job on three servers. Caught two broken jobs within the first week. One was a log rotation that had been silently failing for two months — the disk was at 94% and climbing.

Quick story. A team I worked with had a 40-item production readiness checklist. Beautifully formatted. Color-coded priority levels. Lived in Notion. Got filled out during every launch review.

Their payment webhook processor stopped running on a Tuesday. They found out Friday afternoon — from the finance team.

The Checklist Paradox

Writing a checklist feels productive. You had an outage, you do a postmortem, you add "monitor X" to the list. Good. Except the list grows and nobody re-reads the whole thing. Items 1-5 get done because they're obvious. Items 15-40 are aspirational at best.

And the stuff that actually breaks in production? It's never on the list. It's the thing you didn't think of.

What Monitoring Should Actually Cover

Most setups nail the visible layer. HTTP endpoints, response times, error rates for your main API. That's table stakes. If you don't have that, stop reading and go set it up.

The gap is always in the invisible layer.

Scheduled tasks. Queue workers. Batch processes. The stuff that runs in the background, has no UI, and doesn't respond to HTTP requests. This is where outages hide for days. I've personally missed a failing cron job for over a week because nothing in our monitoring stack was watching it. The job would start, hit a config error, exit with code 0 (yeah, fun), and everyone assumed it was fine.

A Better Mental Model

Instead of "what should we monitor," ask "what would hurt if it stopped working and we didn't know for 48 hours?"

Run that exercise with your team. You'll probably get a list like:

Nightly database backups

Email sending (transactional, not marketing)

Scheduled data imports/exports

SSL cert renewals

The one Lambda function that processes invoices

Now check how many of those have active monitoring with clear alerting. In my experience? Maybe one or two.

The Dead Man's Switch Pattern

For anything that runs on a schedule, this pattern is probably the most reliable approach I've found.

The job runs. When it finishes successfully, it pings an external endpoint. If the ping doesn't arrive within the expected time window, you get an alert. Simple. No log parsing, no checking dashboards, no "did the cron even fire?"

You can build this yourself with a webhook and a timer. Or use an external service — there are several. Point is, the monitoring lives outside your infrastructure. If your entire server goes down, the alert still fires because the expected ping never arrives.

Alerts Need Owners, Not Channels

"Send all alerts to #ops-alerts" is how you get 500 unread messages and zero action.

Every alert should go to a person. A specific, on-rotation person who is responsible for acknowledging and acting on it. If nobody owns the alert, delete it. Seriously. An unowned alert is worse than no alert because it creates a false sense of coverage.

Btw, if your alert doesn't have a runbook attached — even a two-sentence one — it's just a notification that something is wrong with no guidance on what to do about it. That's noise, not signal.

Monitoring Doesn't Age Well

This one cost me a few hours at a previous job. We had monitoring set up for a cron job that ran every 6 hours. Someone changed it to hourly. The monitoring still expected a ping every 6 hours. So for 5 hours between pings, we had zero coverage. The job failed for 4 hours straight and we had no idea.

Config drift is real. Your monitoring setup is accurate on the day you configure it. After that, it's a snapshot of history. Unless you treat it like code — version it, review changes, update it alongside the systems it watches — it'll quietly become useless.

What cron actually is

Cron is a daemon — a background process that checks a schedule file (the crontab) every minute and runs whatever is due. Each user can have their own crontab.

The schedule format

Five fields, separated by spaces, followed by the command:

minute(0-59) hour(0-23) day(1-31) month(1-12) weekday(0-7)

Stars mean "every." So * * * * * runs every single minute. 0 6 * * * runs at 6 AM daily. 0 0 * * 0 runs at midnight on Sundays.

Useful shortcuts some cron implementations support:

@hourly — same as 0 * * * *

@daily — same as 0 0 * * *

@weekly — same as 0 0 * * 0

@reboot — runs once after system boot

Debugging cron

When a cron job does not work, it is almost always one of these:

The script works in your shell but not in cron because PATH is different. Fix: use absolute paths for everything.

You cannot see errors because output is not redirected. Fix: >> /var/log/myjob.log 2>&1

Permissions. The crontab runs as your user. If the script needs root, put it in root's crontab.

The script has Windows line endings (\r\n). Cron chokes on them. Convert with dos2unix.

Cron's blind spot

Cron starts your job. It does not verify your job finished. A backup script that errors out on line 3 still "ran" as far as cron is concerned. The next night, it will try again, fail again, and keep failing silently until someone manually checks.

This blog will mostly be about that — cron monitoring, alerting, on-call pain points, and the unglamorous side of keeping systems running.

I'll also write about whatever else I'm working on or thinking about. Sometimes it'll be DevOps. Sometimes it'll be a random Python script that saved me two hours. No promises on consistency.