Alerts
Alert rules you write as JavaScript expressions, checked every minute, sent to Slack or an HTTP webhook, and logged for 30 days.
CPU stays above 80% for 5 minutes, and no one gets a message
Wave Alerts docs (the default rule for workload metrics); prometheus.io, Alerting overview; Prometheus and Alertmanager configuration docs.
Kubernetes has no built-in way to send a message when a workload stays hot. The usual setup is Prometheus with Alertmanager: "Alerting rules in Prometheus servers send alerts to an Alertmanager" (prometheus.io).
That is two more systems to run and configure. By default, Prometheus scrapes and evaluates rules every 1m, and Alertmanager waits 30s (group_wait) before the first notification of a group.
The rule in the title is the Wave default for workload metrics: send a message when, over at least 5 minutes, the maximum CPU or the average memory reaches 80%.
Default rule, 1-minute check, and message template: Wave Alerts docs. The CPU series, the workload name, and the channel are illustrative; 88% is the max of the window, as in the docs example.
How Wave does it
- Prometheusscrapes metrics every 1m by default
- rulesAlerting rulea PromQL expression, evaluated every 1m
- firesAlertmanagergroups alerts, waits 30s first
- routesReceiverchat, webhook, and more
- Eventsworkload metrics, schedule phases, Autopilot logs
- matchRulea JavaScript expression, checked every minute
- renderMessageyour template, with the workload and its CPU
- sendChannelSlack Webhook, Slack Web API, or HTTP
Add a channel
Send to a Slack Webhook, the Slack Web API, or an HTTP webhook. One channel can serve many rules, and each channel can go through an HTTP proxy.
Write the rule
A rule is a JavaScript expression over one of three event types: workload CPU and memory, Deployment scheduling phase START and END, or missing Autopilot logs. Wave checks rules every minute.
Check every delivery
Each send is logged with its response and status, and logs are kept for 30 days. Wave does not retry a failed send, so a failure does not turn into a flood of messages.