ELXSoftware

ELX-HealthMonitor

Documentation

Installing the agent, choosing what it watches and making the alerts worth reading.

Installation

Install
sudo install -m 755 elx-healthmonitor-linux-amd64 /usr/local/bin/elx-healthmonitor
sudo /usr/local/bin/elx-healthmonitor --install-service
sudo systemctl enable --now elx-healthmonitor
# web interface: http://<server>:8555
Recommended, not required
sudo apt-get install -y smartmontools
PathWhat it is
/usr/local/bin/elx-healthmonitorThe executable
/etc/elx-healthmonitor/config.jsonSettings, mode 0600
/var/lib/elx-healthmonitor/history.dbMetric history and the event log
/etc/systemd/system/elx-healthmonitor.serviceThe unit
Command-line flags
--config <path>       configuration file
--listen <address>    web interface address, e.g. :8555
--log-level <level>   debug | info | warn | error
--version             version and build time

Choosing what to collect

The Settings section decides what is polled, how often and what is excluded; the Alerts section decides when you hear about it. Changes apply immediately, without a restart. Editing config.json and restarting the service has the same effect.

The first thing to excludeLoop and virtual devices in the filesystem section. Left in, they fill the table with entries nobody has ever needed to look at.

Alerts

A rule is a condition, a threshold and a hold-down. The hold-down is the part that matters: without it, a monitoring system reports every momentary spike, and within a week nobody reads its mail.

Watch forA threshold that works
Filesystem filling up85 per cent — enough warning to act without hurrying
Processor loadAbove 90 per cent held for ten minutes, not ten seconds
Disk temperatureAbove 50 °C sustained; a brief peak during a backup is normal
SMARTAny change in reallocated or pending sectors — no hold-down, report at once
RAIDArray degraded, or a non-zero mismatch count — always
ServiceA unit that should be running and is not

Mail supports three encryption modes: SMTPS, where TLS starts with the connection, usually on port 465; STARTTLS, where it is negotiated, usually on 587; and unencrypted, which belongs only on a mail relay inside your own network.

History

  • raw_retention_days — how long readings are kept exactly as measured; three days by default.
  • rollup_interval_sec — the bucket size after that; five minutes, storing the average, minimum and maximum of each bucket.
  • retention_days — when everything is deleted; a month by default.

Thinning runs in six-hour windows so that it never holds the database long enough to be noticed.

Why minimum and maximum are keptAn average alone hides the event you were looking for: a disk that reached 60 °C for one minute inside a five-minute bucket averages to something unremarkable.

When something is wrong

SymptomWhere to look
The interface does not opensystemctl status elx-healthmonitor, and whether the port is already taken
The SMART section is emptysmartmontools is not installed, or the disks are behind a controller that does not pass SMART through
No temperatures at allA virtual machine — there are no physical sensors to read
Mail is not deliveredThe wrong encryption mode for the port: 465 is SMTPS, 587 is STARTTLS
Alerts arrive constantlyThe hold-down is too short, or the threshold is inside the machine's normal range
History stops after a few daysRetention is set lower than you think, or the disk holding the database is full

Frequently asked questions

Where are the agent's own logs?
journalctl -u elx-healthmonitor -f.
Should the interface be exposed to the internet?
No. It can restart services and reboot the machine. Reach it over a private network, or through a tunnel.
How do I remove the power buttons?
Clear server.allow_power_control in the configuration, and reboot and shutdown disappear from the header.