A stuck Kubernetes CronJob can quietly skip every run after it

concurrencyPolicy: Forbid sounds safe: never run two copies of a job at once. But the Kubernetes docs note that if a scheduled time arrives while the previous Job is still running, that run is skipped and counted as missed. (Kubernetes CronJob docs)

So if one run hangs on a stuck network call, the next runs never start. Nothing crashes and no new Job fails, so a failure alert has nothing to catch.

Two fixes:

1. Set activeDeadlineSeconds on the Job template so a hung run gets stopped. (Kubernetes Job docs)
2. Send a heartbeat as the last step of the job, after the real work succeeds. Skipped and hung runs never check in, so you get an alert.

Set up a heartbeat for a hung job, free for 3 watches.