Every production server has them. Cron jobs that stopped working at some point and nobody noticed. They sit there in crontab like abandoned furniture — technically present, functionally useless.
I did an audit on a client's server last year. Forty-seven cron jobs. Twelve of them hadn't run successfully in over a month. Three were pointing at scripts that no longer existed.
How does this happen?
Usually it's gradual. Someone deploys a new version and the script path changes. Or a dependency gets updated and breaks something. Or the disk fills up and writes start failing.
The job throws an error, cron sends an email to a mailbox nobody checks, and that's it. Weeks pass. Months, sometimes.
The backup job is the scary one. You think you have nightly backups. You haven't had a backup in six weeks. You find out when you actually need to restore something.
A quick audit you can do right now
Pull up every cron job on your servers:
# List all user crontabs for user in $(cut -f1 -d: /etc/passwd); do crontab -l -u $user 2>/dev/null | grep -v '^#' | grep -v '^$' | while read line; do echo "$user: $line" done done # Don't forget /etc/cron.d/ and /etc/cron.daily/ ls -la /etc/cron.d/ ls -la /etc/cron.daily/
For each job, ask two questions:
When did this last run successfully? (Check logs, output files, or database timestamps)
If it stopped right now, how long before someone noticed?
If the answer to #2 is "I don't know" — that job needs monitoring.
The dead man's switch pattern
Instead of monitoring whether a job fails (hard to do reliably), monitor whether it succeeds. Add a ping at the end of the script:
#!/bin/bash # do the actual work rsync -a /data/ /backup/data/ # signal success curl -fsS --retry 3 https://monitor.example.com/ping/job-id
If the ping doesn't arrive within the expected window, something's wrong. You get notified. Simple.
Took me about an afternoon to add monitoring to every cron job on three servers. Caught two broken jobs within the first week. One was a log rotation that had been silently failing for two months — the disk was at 94% and climbing.
Probably worth an afternoon of your time too.

















