Two capabilities decide whether an incident is a bad hour or a bad week:
Backfills — because runs are idempotent per partition, reloading history is a loop, not a project. Design the partition parameter in from day one.
Alerts on absence — the worst failure isn't a crash, it's silence: the job that didn't run. Alert when expected data doesn't arrive, not just when code throws.
Important
Freshness checks ("newest row is < 2 h old") catch entire categories of failure that per-task error handling never sees.