Scrutiny

Scrutiny is a utility that provides storage drive S.M.A.R.T monitoring, historic trends, and real-world failure thresholds.

Scrutiny is deployed as hub-and-spoke format, where the web gui is hosted on k8s, a collector runs as a replicaset on nodes where rook-ceph drives are hosted, and a collector also runs on TrueNAS

See supported OSes

Application

Scrutiny is deployed as 2 HelmReleases:

scrutiny runs the web application front end / data collection endpoint and its InfluxDB data store as two containers in a single release
scrutiny-collector deploys a daemonset so that each k8s node can log SMART data to Scrutiny

TrueNAS setup

Add’l references:

docker-compose

Monitor the Health of Multiple SMART Disks with Scrutiny | Stefandroid Blog

NOTE: Scrutiny collector may need to be installed after every TrueNAS update!

Scrutiny needs Smartmontools version 7+. Check on the TrueNAS terminal that version 7 is installed.
```
sudo su  # must be run as root!
smartctl -V
```

Download the Collector agent binary (below link is for version 0.3.13 - the latest as of January 2022). Then copy it to /usr/local and make it executable. In short, execute the following as root:

mkdir -p /mnt/ssdpool/Users/admin/scrutiny
wget "https://github.com/AnalogJ/scrutiny/releases/latest/download/scrutiny-collector-metrics-$(uname | tr "[:upper:]" "[:lower:]")-amd64" -P /mnt/ssdpool/Users/admin/scrutiny/bin
chmod +x /mnt/ssdpool/Users/admin/scrutiny/bin/scrutiny-collector-metrics-$(uname | tr "[:upper:]" "[:lower:]")-amd64

Create the configuration file by downloading the sample collector.yaml from the GitHub repo.

mkdir -p /mnt/ssdpool/Users/admin/scrutiny/config
wget https://raw.githubusercontent.com/AnalogJ/scrutiny/master/example.collector.yaml -O /mnt/ssdpool/Users/admin/scrutiny/config/collector.yaml

Edit the configuration and change the following parameters:
- host.id should be an identifier for your TrueNAS server
- api.endpoint should be the HTTP endpoint of the Scrutiny Web server that we deployed
```
version: 1
host:
  id: truenas
api:
  endpoint: https://scrutiny.${SECRET_DOMAIN}
```

Create a TrueNAS cron job (System Settings → Advanced → Cron Jobs):

Key	Value
Description	Scrutiny
Command	. /etc/profile; /mnt/ssdpool/Users/admin/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 run –config /mnt/ssdpool/Users/admin/scrutiny/config/collector.yaml
Run As User	root
Schedule	Hourly

Source: kubernetes/apps/observability/scrutiny/README.md