Watchdog

Sometimes something hangs: a server never PXE-boots, a BMC stops answering in the middle of a call, the worker process dies while a job runs. The watchdog finds anything that runs far too long, stops it and says why.

Jobs

The job worker checks every running and queued job once a minute (with several workers, only one checks at a time). A job is stopped when:

Check	Default	Measured from
A step runs longer than its limit	see the table below	the step’s first start (a resumed step keeps its clock)
The whole job runs longer than the limit for its type	see the table below	the job’s start
No worker runs the job any more	10 min without a heartbeat	the runner writes one every 30 s, also while a step waits
The job keeps being resumed without finishing	6 starts	worker restarts or crashes during the job
The job never starts	30 min in the queue while nothing else runs on the server (6 h in any case)	when it was created

A stopped job takes the normal cancel path: provisioning jobs lose their PXE entry at once (like Stop installation), the step ends at its next check, and the power state is left as it is — the watchdog never powers a server off or on. The job then ends as failed with the reason. A job whose worker is gone is marked failed right away; a job that does not end within 3 minutes after the stop is marked failed as well.

The reason says which step, how long it ran, the limit and the last thing that happened — the PXE diagnosis, the BMC call the job was waiting on, or the last warning in the job log. With wait:installer-started set to 40 minutes, for example:

Stopped by the watchdog: step wait:installer-started ran 47 min (limit 40 min).
Last: No DHCP request from 52:54:00:ab:00:01 47 min after the power cycle — the server probably did not PXE boot

The job page, the job lists, the dashboard and the server’s jobs show stopped by watchdog with this reason.

Limits

Job types (whole job):

Job type	Limit
power, bmc_command, switch_port, outlet_power	5 min
firmware_update	90 min
rescue	90 min
hardware_collect, password_reset	2 h
hardware_discover	2 h 15 min
reinstall	3 h
disk_wipe	5 h (a secure wipe: its own time budget, see disk wipe)
any other type	6 h

Steps:

Step	Limit	Step	Limit
prepare (boot files, site agent)	40 min	wait:installer-started	60 min
boot:mode, boot:pxe-once, disarm	5 min	wait:finished	4 h 15 min
power:cycle	10 min	boot:disk	25 min
other power:… steps	5 min	finalize	10 min
check, flash	10 / 15 min	wait:flash	60 min
verify	15 min	wait:report (discover)	2 h 15 min
bmc:execute, switch:port-admin, pdu:outlet	5 min	any other step	30 min

The limits sit above the deadlines the steps have themselves (the PXE wait gives up after 30 minutes plus 20 after the UEFI retry, the installer after 4 hours), so a normal timeout still ends a job with its own message; the watchdog catches what hangs.

Changing the limits

Set overrides in minutes (1 minute to 24 hours) with the setting watchdog:

curl -X PUT https://panel.example.com/api/v1/settings \
  -H "authorization: Bearer $TOKEN" -H 'content-type: application/json' \
  -d '{"watchdog": {"jobs": {"reinstall": 240}, "steps": {"wait:installer-started": 90}, "tasks": {"power": 5}}}'

or in the environment with TRIXOS_WATCHDOG_LIMITS (same JSON; the setting wins). power:* sets every power step at once. Values outside the bounds are refused. TRIXOS_WATCHDOG=false (or "enabled": false in the setting) turns the stops off. GET /api/v1/system/watchdog shows the limits in effect.

BMC, Redfish and SNMP calls

Every call to a BMC or a switch / PDU has a hard timeout: ipmitool 30 s (60 s for custom commands and hardware reads), Redfish 30 s, SNMP 15 s per get / set and 2 min per walk. A call that times out fails with the command and the host, and that is what the job log shows:

Job failed: ipmitool chassis power cycle at 10.0.5.66 timed out after 30 s (no answer from the BMC)

Polls and other loops

Each poller task (power, bandwidth, sensors, PDUs, prune, firmware) has a budget of three intervals (at least 1 minute, at most 1 hour; tasks in the setting changes it). A run over its budget is reported right away with what it waits on — the poller turns Warning on the dashboard with the reason, for example:

Poller: power poll over its budget: poller task power ran 3 min, longer than its budget of 3 min,
waiting on ipmitool chassis power status at 10.0.5.66 (pending for 25 s) and 11 more calls

The task is not started again until that run has ended (the call timeouts make sure it does); the warning stays until a run finishes within its budget. The license check (50 s) and the health samples (30 s) of the API are watched the same way.

Consoles

A live console session (serial, KVM, BMC web interface) ends after 8 hours; the window says “Stopped by the watchdog”. Viewer containers for Java and vendor consoles are removed after 4 hours, with the reason in the viewer launcher’s log.

Where you see it

• Dashboard → Control server → Watchdog · last 24 h: every stop and warning with its reason.
• The job drawer and job page: a red Stopped by the watchdog box with the reason; the job lists show a badge.
• Webhook / e-mail event job.watchdog (see Notifications) with jobId, type, serverId, step, kind, ranMs, limitMs and reason; the job’s end sends job.failed as usual.
• The audit log: job.watchdog, by system.