A watchdog becomes trustworthy when healthy work is quiet, retries are finite, and uncertainty has somewhere to go.
Recurring automation is easy to start and strangely hard to finish. A timer fires, a check runs, and a result sends the process around again. When the happy path is healthy, this looks almost trivial. The real design appears when the check fails at the edge of a maintenance window, produces incomplete evidence, or repeats a repair attempt that isn't going to work.
I model those cases as states rather than piling conditionals around a loop. A watchdog should know why it's running, what changed on the last attempt, and which result ends its authority. Otherwise I've built a very punctual way to repeat a mistake.
Healthy should become quiet
The normal cycle begins with an eligible check and ends without fanfare when evidence is healthy. Quiet doesn't mean invisible. The run can retain a timestamp and compact result for later inspection. It means the system doesn't manufacture an alert, message, or work item just to prove the timer fired.
This keeps attention attached to changes that need it. A stream of success notices trains me to scan past the channel, and eventually the one important transition looks like background noise.
The healthy state also resets failure history when appropriate. A prior transient failure shouldn't leave the next clean run carrying an escalation counter forever. At the same time, clearing history shouldn't erase the evidence needed to understand intermittent behavior. Current state and retained history have different jobs.
Maintenance inhibits action
A check that runs during planned maintenance may observe exactly what the maintenance intended: an unavailable endpoint, a stopped process, or delayed work. Treating that observation as a new failure wastes attention and can trigger unsafe interference.
Maintenance has to be an active state the watchdog can evaluate, not a sentence in an operations calendar the automation never reads. When inhibition applies, the system records that the check or action was suppressed, with enough timing context to resume later.
Suppression isn't success. The watchdog didn't establish health, so it shouldn't publish a fresh healthy result. When the maintenance condition expires, the next eligible run gathers new evidence. If the inhibition has no reliable end, the automation needs to surface that ambiguity rather than wait forever and call it patience.
Retry only when evidence can change
Retries are useful for transient conditions: a dependency may be late, a request may time out, or an observation may be temporarily incomplete. Repeating the same operation without a reason to expect different evidence is just a loop consuming time. I've watched enough loops do this with great confidence.
I bind retries to both a count and a condition. The automation records the attempt, the observed failure class, and any backoff. Another attempt is permitted only if policy still considers the work eligible. New maintenance inhibition, an operator hold, or a changed target can stop the sequence.
The transition set can remain compact:
due -> checkingwhen the schedule is eligiblechecking -> healthywhen current evidence passeschecking -> inhibitedwhen maintenance policy applieschecking -> retry_waitfor a retryable result with budget remainingretry_wait -> checkingafter backoff and renewed eligibilitychecking -> needs_attentionfor a terminal result or exhausted budget
This list is more useful than a generic “retry three times” rule because it says what the attempts mean. The exact budget can vary by task. A terminal transition can't be optional.
Evidence capture happens before another attempt. Otherwise the loop overwrites the detail that explains why it kept running. Each retry should add context, not merely update an integer and hope nobody asks.
Escalation is a completed automation state
When the budget is exhausted or the failure isn't safe to retry, the watchdog transitions to needs_attention. That state includes the last evidence, the attempt history, the maintenance evaluation, and the reason automation stopped. It should identify the operating owner or queue without guessing at a repair.
Classification at this boundary should stay modest. The watchdog can report that a route stayed unavailable across eligible attempts or that a check couldn't obtain evidence. It shouldn't convert correlation into a confident root cause. A concise packet of observations, timing, and actions already taken gives the operator a better starting point than a dramatic diagnosis assembled because the alert template had an empty field.
This is where many recurring jobs accidentally remain active. They send an alert and keep retrying on the next tick, creating duplicate messages while a human is already investigating. Escalation has to inhibit further attempts until the state is acknowledged, reset by policy, or replaced by new evidence through an explicit action.
An escalation can be quiet in its own way. One actionable notice beats a fresh notice every interval. The job is done when it has gathered current evidence, used its bounded retry authority, and transferred the unresolved condition with enough context for the next decision.
Recovery also needs a defined route. An operator may clear the attention state after correcting the issue, but clearing it should lead to a new check, not directly to healthy. Human action changes the conditions; current evidence establishes the result.
With the state machine, every run has an end I can inspect. Healthy evidence closes quietly. Maintenance explains inhibition. Retry waits for a plausible change. Exhaustion produces one durable escalation and stops. The timer will fire again. This failure won't keep following it around.