Start with a question
Monitoring without a question is just a wall of graphs. For each service I wrote down what "working" means in one sentence, then turned that sentence into a measurement.
For the homelab the questions are modest:
- Can I reach the service from outside the network?
- Is it responding in under a second?
- Is the disk going to run out this month?
- Did the last backup finish?
Everything on the dashboard maps to one of those.
Symptoms, not causes
The first version of my alerting paged on CPU, memory, and container restarts. It was noisy and, worse, it was not actionable. High CPU is a cause. A page that takes three seconds is a symptom.
The alerts I keep now are the ones where the answer to "what do I do next?" is obvious enough to put in the alert description.
- alert: EndpointSlow
expr: probe_duration_seconds{job="blackbox"} > 1.5
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} is responding slowly"
runbook: "https://example.com/runbooks/slow-endpoint"Make the dashboard boring
A good dashboard is read in ten seconds. I keep one row per service with four panels: up, latency, error rate, and saturation. If a panel has never changed my mind about anything, it gets removed.
The unglamorous part
The most valuable monitor is a dead man's switch. If Prometheus itself stops scraping, nobody gets an alert, because the thing that sends alerts is gone. A tiny external check watches the watcher and emails me when the metrics stop arriving.
That check has fired exactly once. It was worth building.