Most of my automation is a Python script under cron. The scripts are boring. The failures were not, until I settled on four rules that have kept things quiet for about a year.
The overlapping-run failure is the classic one: the job takes longer than its interval, two copies run, and they fight over the same rows. A lock file with an exclusive flock is enough, and it releases itself if the process dies.
import fcntl, sys
lock = open("/tmp/sync.lock", "w")
try:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
sys.exit(0) # предыдущий запуск ещё идёт — молча выходим
Cron mails on non-zero exit and stays silent on zero. That is a usable alerting system if the script is honest about failure. Catching every exception and exiting zero turns a monitored job into an unmonitored one.
On Windows the default encoding is still cp1251 and it will corrupt anything non-Latin the moment you write a log line with a customer name in it. Be explicit everywhere, even in scripts that "will only ever run on the server".
logging.basicConfig(
filename="/var/log/sync.log",
encoding="utf-8",
level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
)
Every job I write can be run again immediately with no harm. That means upserts instead of inserts, and a processed-at column instead of a queue that gets drained. It sounds like extra work and it is, once. It pays back the first time something fails halfway through and the fix is simply to run it again.
*/10 * * * * /opt/venv/bin/python /opt/jobs/sync.py >> /var/log/sync.out 2>&1
Absolute paths for everything. Cron has almost no environment, and a script that
works in your shell and not under cron is nearly always a PATH problem.