System health
Admins see two cards on the Dashboard: the control server (the host the panel runs on) with its usage, services and the watchdog’s last 24 hours, and below it Site agents, one compact row per site with its status, version and CPU, memory and disk. Each has a status: Healthy, Warning or Problem. A row opens with a click (or Expand all) to the reasons (for example “disk 92 % full (/)” or “offline, last seen 14 min ago”), host, OS, public IP, uptime, WireGuard handshake and PXE state; agents with problems are listed first, and with many sites the list has a search and pages. The page refreshes every 30 seconds. Manage opens System → Agents, which shows each agent in full (24-hour graphs, forwarding targets) with its actions. Jobs are on the Jobs page.
Control server
Section titled “Control server”| Shown | Where it comes from |
|---|---|
| CPU, load, memory, uptime | /proc of the host. Inside the API container /proc/stat, /proc/meminfo, /proc/loadavg and /proc/uptime show the host’s values. |
| Disk | The filesystem of TRIXOS_HEALTH_DISK_PATH (default /: inside the container, the disk behind Docker’s storage, which also holds the database volume on a standard install). |
| 24-hour graphs | One sample per minute, kept in Redis for 24 hours. A fresh panel starts with “collecting history…”. |
| API, job worker, poller, PXE / boot files | Each service writes a heartbeat every 30 seconds. A missing heartbeat for 2 minutes counts as stopped. The poller also reports its last run of every task (bandwidth, sensors, power, ping, …). |
| Postgres, Redis | Latency of a test query, database size, Redis memory. |
| Job queue | Running, waiting and delayed jobs and the number of connected workers. |
| License | State, edition and servers used. |
Warnings: disk from 85 % (problem from 95 %), memory from 90 % (problem from 97 %), CPU from 90 %, load above twice the number of CPUs, more than 50 jobs waiting. A stopped worker or poller, an unreachable Postgres or Redis and a locked license are problems.
Site agents
Section titled “Site agents”Every site with an agent shows when the agent was last seen, its version, uptime, the age of the WireGuard handshake and its PXE state (idle, serving jobs, or the problem that stops it). Agents from version 1.3.0 also send their CPU, load, memory and disk usage with every heartbeat (once a minute): bars in the dashboard’s agent rows, the same 24-hour graphs as the control server under System → Agents.
| Status | Meaning |
|---|---|
| Last seen more than 3 min ago | Warning |
| Last seen more than 10 min ago | Problem: the agent is offline (host down, panel unreachable, service stopped) |
| WireGuard handshake older than 5 / 15 min | Warning / problem: the tunnel to the panel is down, BMCs at this site are not reachable |
| Older agent version | Warning with the version that is available. Use System → Agents → Update. |
An agent older than 1.3.0 shows “update it … to see CPU, memory and disk” instead of the bars; everything else works as before. Sites without an agent are listed as No agent.
Watchdog
Section titled “Watchdog”Below the services, Watchdog · last 24 h lists what the watchdog stopped or warned about: jobs over their limits, poller tasks over their budget, console sessions open longer than 8 hours, with the reason. A poller task over its budget also turns the poller Warning with what the poll waits on; a slow license check or health sample of the API is a warning for 15 minutes.
GET /api/v1/system/health (admins) returns the same data (the watchdog’s actions under watchdog), see the
API reference. Usage history points are [unix seconds, CPU %, memory %, disk %, load].