Cron Failures (Silent Drops)

When your nightly billing script fails silently and you lose $10,000.

The idea

A Cron Job is a scheduled task (e.g., "run this script every night at midnight"). Because cron jobs run in the background without a user waiting for an HTTP response, they are notorious for failing silently. If a server is rebooting exactly at midnight, the cron job just doesn't run. If the script throws an error halfway through, it crashes, and nobody knows until customers complain.

Step 1: Midnight approaches. The Cron daemon prepares to run the 'charge_users' script.

How it works (Dead-Man's Switches)

To fix this, professional systems never rely on raw OS cron. They use Cron Monitors (like Sentry Cron or Healthchecks.io). This is a "Dead-Man's Switch." The system expects your script to ping it every 24 hours. If your script crashes, or the server was off, the ping never arrives. The monitor realizes the script is missing, and sends an alert to PagerDuty.

# A Safe Cron Job Pattern

import requests

def run_nightly_billing():
    try:
        # 1. Ping the monitor: "I am starting"
        requests.get("https://hc-ping.com/YOUR_UUID/start")
        
        # 2. Do the actual work (Iterate over users)
        charge_all_users()
        
        # 3. Ping the monitor: "I succeeded"
        requests.get("https://hc-ping.com/YOUR_UUID")
        
    except Exception as e:
        # 4. Ping the monitor: "I failed!"
        requests.get(f"https://hc-ping.com/YOUR_UUID/fail?msg={e}")
        raise e

Cost

Adding observability to background jobs requires an external service to hold state (the expected schedule). Without it, your system is blind to the absence of a success log. The overhead is minimal (a few HTTP requests per job run).

Watch out for