Alert CRDs

These kinds configure Alerts in CRD Mode. See the Alerts overview for an introduction to channels and rules. WaveAlertChannel defines where notifications go; WaveAlert defines what to monitor and when to fire.

WaveAlertChannel

WaveAlertChannel defines a notification destination — an HTTP endpoint, a Slack Webhook, or a Slack Web API channel. Alert rules reference channels by name. All credentials, URLs, headers, and proxy settings are stored in Kubernetes Secrets and never written into the CR itself.

Scope: cluster-scoped — no metadata.namespace required.

First, create the Secrets in the wave-autoscale namespace:

apiVersion: v1
kind: Secret
metadata:
  name: http-endpoint
  namespace: wave-autoscale
type: Opaque
stringData:
  url: "https://hooks.example.com/webhook"
---
apiVersion: v1
kind: Secret
metadata:
  name: http-headers
  namespace: wave-autoscale
type: Opaque
stringData:
  # Must be a JSON object string — key/value pairs of HTTP headers.
  headers: '{"Authorization":"Bearer <token>","Content-Type":"application/json"}'

Then create the channel CR:

apiVersion: wavek8s.com/v1alpha1
kind: WaveAlertChannel
metadata:
  name: oncall-http
spec:
  http:
    urlSecretRef:
      name: http-endpoint
      key: url
    method: POST
    headersSecretRef:
      name: http-headers
      key: headers

Spec fields

Exactly one of http, slackWebhook, or slackWebApi must be present. Setting zero or more than one reports ValidationFailed in .status.conditions.

http fields

slackWebhook fields

Example:

apiVersion: v1
kind: Secret
metadata:
  name: slack-webhook-url
  namespace: wave-autoscale
type: Opaque
stringData:
  url: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXX"
---
apiVersion: wavek8s.com/v1alpha1
kind: WaveAlertChannel
metadata:
  name: oncall-slack-webhook
spec:
  slackWebhook:
    webhookUrlSecretRef:
      name: slack-webhook-url
      key: url

slackWebApi fields

Example:

apiVersion: v1
kind: Secret
metadata:
  name: slack-bot-token
  namespace: wave-autoscale
type: Opaque
stringData:
  token: "xoxb-your-bot-token-here"
---
apiVersion: wavek8s.com/v1alpha1
kind: WaveAlertChannel
metadata:
  name: oncall-slack-webapi
spec:
  slackWebApi:
    tokenSecretRef:
      name: slack-bot-token
      key: token
    channel: "#alerts"

Notes

• Cluster-scoped — no metadata.namespace required.
• Exactly one variant (http / slackWebhook / slackWebApi) must be set. This is validated by Wave at reconcile time and surfaces in .status.conditions as ValidationFailed — not rejected at kubectl apply.
• All credentials, URLs, headers, and proxy settings are Secret references. Wave reads Secret values from the wave-autoscale namespace at reconcile time and never writes them into the CR. A missing Secret or key reports SecretNotFound in .status.conditions and self-heals when the Secret appears.
• The headersSecretRef value must be a JSON object string. A malformed value is currently treated as SecretNotFound.
• Channels are referenced by name from WaveAlert.spec.alertMessages[].channelRef. Deleting a WaveAlertChannel CR is never blocked by Kubernetes; while existing WaveAlerts still reference it, Wave keeps the channel configuration active and reports RefInUse in .status.conditions — it is cleaned up once the referencing alerts are gone.

WaveAlert

WaveAlert defines an alert rule: which event type to evaluate, what expression determines whether the alert fires, which targets to scope it to, and which channels to deliver notifications to.

Scope: cluster-scoped — no metadata.namespace required.

apiVersion: wavek8s.com/v1alpha1
kind: WaveAlert
metadata:
  name: high-cpu-prod-api
spec:
  title: High CPU/memory on prod/api
  eventType: deployment_workload_metrics
  eventTargets:
    specific:
      - resourceType: Deployment
        namespace: prod
        name: api
  eventRuleExpression: "evaluation_period_minutes >= 1 && ( max(cpu_utilization_arr) >= 80 || avg(memory_utilization_arr) >= 80 )"
  eventRuleCheckIntervalMin: 5
  alertMessages:
    - channelRef: oncall-http
      message: "${namespace}/${workload_name} CPU/memory exceeded 80% over the evaluation window"

Spec fields

eventTargets shape

eventTargets takes exactly one of two forms:

# Target all resources of a given type:
eventTargets:
  all:
    resourceType: Deployment
 
# Target specific named resources:
eventTargets:
  specific:
    - resourceType: Deployment
      namespace: prod
      name: api
 
# Targetless event type (cluster-wide) — use an empty specific list:
eventTargets:
  specific: []

Supported target types by eventType

Rule expression variables by eventType

Each event type exposes a different set of variables. eventRuleExpression uses bare variable names — no ${} wrapper. alertMessages[].message uses ${varName} JavaScript template-literal syntax.

Event type reference

pending_pod_duration — a Pod has been stuck in Pending longer than the threshold. reason_category groups the scheduler's reason (for example insufficient resources versus an unbound volume), so you can route capacity problems separately from storage problems.

apiVersion: wavek8s.com/v1alpha1
kind: WaveAlert
metadata:
  name: pods-stuck-pending
spec:
  title: Pods pending over 5 minutes
  eventType: pending_pod_duration
  eventTargets:
    specific: []
  eventRuleExpression: "pending_duration_minutes >= 5"
  alertMessages:
    - channelRef: oncall-slack-webhook
      message: "Pod ${namespace}/${pod_name} has been pending for ${pending_duration_minutes} minutes"

pod_container_failure — a container entered a failure state. transition_type names the transition (for example CrashLoopBackOff, OOMKilled, ImagePullBackOff) and workload_kind is Deployment, StatefulSet, or DaemonSet. The default expression is true — it fires on every failure incident, and you narrow it with a transition_type condition. Workloads listed in WaveDiagnosisConfig.spec.podContainerFailures.excludedWorkloads never produce events here.

apiVersion: wavek8s.com/v1alpha1
kind: WaveAlert
metadata:
  name: crashloop-only
spec:
  title: CrashLoopBackOff
  eventType: pod_container_failure
  eventTargets:
    specific: []
  eventRuleExpression: "transition_type == 'CrashLoopBackOff'"
  alertMessages:
    - channelRef: oncall-slack-webhook
      message: "Container ${container_name} in pod ${namespace}/${pod_name} failed: ${transition_type}"

pv_usage — a PersistentVolumeClaim crossed a usage threshold. Detection runs every minute, so pair the threshold with a minutes_since_last_alert clause to act as a re-notify cooldown — without it a PVC that stays over threshold re-alerts every minute.

apiVersion: wavek8s.com/v1alpha1
kind: WaveAlert
metadata:
  name: pvc-nearly-full
spec:
  title: PVC over 80%
  eventType: pv_usage
  eventTargets:
    all:
      resourceType: Pvc
  eventRuleExpression: "usage_percent >= 80 && minutes_since_last_alert >= 30"
  alertMessages:
    - channelRef: oncall-slack-webhook
      message: "PVC ${namespace}/${pvc_name} usage: ${usage_percent}%"

smart_sizing_recommendation — Smart Sizing produced a recommendation that differs materially from the container's current request. The *_delta_abs_percent variables are absolute percentage gaps, so a single threshold catches both over- and under-provisioning. CPU values are in cores, memory values in MiB.

apiVersion: wavek8s.com/v1alpha1
kind: WaveAlert
metadata:
  name: sizing-drift
spec:
  title: Smart Sizing drift over 20%
  eventType: smart_sizing_recommendation
  eventTargets:
    all:
      resourceType: Deployment
  eventRuleExpression: "cpu_request_delta_abs_percent >= 20 || memory_request_delta_abs_percent >= 20"
  alertMessages:
    - channelRef: oncall-slack-webhook
      message: "Smart Sizing: ${namespace}/${workload_name}/${container_name} CPU ${current_cpu_request} -> ${recommended_cpu_request} cores, Memory ${current_memory_request} -> ${recommended_memory_request} MiB"

oom_kill_risk — a container's memory utilization reached the OOM-kill threshold, in real time, before the kill happens. The threshold and the per-container cooldown are set on WaveDiagnosisConfig.spec.memoryAnomaly (oomkillUtilization, default 95%; oomkillCooldownMinutes, default 10). Because that cooldown already suppresses repeats at the source, the rule expression usually only needs a minutes_since_last_alert guard.

apiVersion: wavek8s.com/v1alpha1
kind: WaveAlert
metadata:
  name: oom-kill-risk
spec:
  title: OOM-kill risk
  eventType: oom_kill_risk
  eventTargets:
    specific: []
  eventRuleExpression: "minutes_since_last_alert >= 10"
  alertMessages:
    - channelRef: oncall-slack-webhook
      message: "Container ${container_name} in pod ${namespace}/${pod_name} at ${utilization}% memory — OOM-kill risk"

Notes

• Cluster-scoped — no metadata.namespace required.
• eventType is validated by Wave at reconcile time. An unrecognized value reports ValidationFailed in .status.conditions and the alert is not activated.
• eventRuleExpression is validated by Wave. An expression referencing variables not available for the configured eventType (or that is otherwise malformed) reports ValidationFailed and the alert is not activated. The rule self-heals when the expression is corrected.
• alertMessages is optional. A WaveAlert with no messages is valid and its rule is still evaluated — it just delivers to no channel. This is useful for testing expressions before wiring up channels.
• Message templates use JavaScript template-literal interpolation (${varName}). Go-style {{.var}} is not interpolated and emits verbatim.
• Each channelRef must match the metadata.name of an existing WaveAlertChannel CR. A missing channel reports RefNotFound in .status.conditions and converges automatically when the channel CR appears. The entire alert is skipped (not just the missing entry) until all channel references resolve.
• eventTargets is a free-form object at the CRD layer (x-kubernetes-preserve-unknown-fields: true). Structural errors such as typos in field names are not caught at kubectl apply — they surface as ValidationFailed via Wave's reconciler.
• Deleting a WaveAlertChannel that is still referenced by a WaveAlert is never blocked by Kubernetes — Wave keeps the channel configuration active and reports RefInUse in .status.conditions until the referencing alert CRs are removed, at which point it is cleaned up.