Alerts Overview

Wave Alerts enables proactive monitoring of your Kubernetes cluster by automatically sending notifications when specific conditions are met. Instead of manually checking dashboards or setting up complex monitoring pipelines, Alerts provides a flexible, event-driven notification system that integrates seamlessly with your existing communication tools.

What are Alerts?

Alerts is a comprehensive notification system that monitors cluster events and sends real-time notifications to your configured channels. It consists of two core components:

1. Alert Channels: Define where notifications are sent (Slack, HTTP webhooks, etc.)
2. Alert Rules: Define what to monitor and when to send notifications

This separation allows you to:

• Reuse channels across multiple alert rules
• Configure different notification destinations for different teams or severity levels
• Maintain audit trails of all alert executions

Why Use Alerts?

Proactive Problem Detection

Instead of discovering issues when users report them, Alerts notifies you immediately when:

• Workload CPU or memory usage exceeds thresholds
• Deployments start or stop
• Autopilot logs go missing for extended periods
• Custom conditions defined by your team are triggered

Flexible Notification Routing

Send alerts to the right people through the right channels:

• Send critical production alerts to PagerDuty via HTTP webhooks
• Route development cluster notifications to Slack channels
• Deliver weekly reports to email via custom integrations
• Integrate with any system that accepts HTTP requests

Customizable Alert Logic

Define alert conditions using JavaScript expressions with access to:

• Real-time workload metrics (CPU, memory utilization)
• Deployment lifecycle events (scheduling phases, rollout status)
• Historical data arrays for trend analysis
• Custom evaluation periods and thresholds

Key Features

Multiple Channel Types

Wave supports three notification channel types out of the box:

All channels support optional HTTP proxy configuration for enterprise network environments.

Event-Driven Architecture

Alerts monitors specific event types:

Workload Metrics Events

• Aggregated CPU and memory utilization over time windows
• Supports max(), avg() functions for array analysis
• Configurable evaluation periods (e.g., "alert if CPU > 80% for 5+ minutes")

Deployment Scheduling Events

• START/END lifecycle transitions
• Scheduling phase monitoring
• Deployment count aggregation

Autopilot Events

• Missing log detection
• Log gap duration tracking
• Per-workload monitoring

Flexible Targeting

Control which resources each alert monitors:

• None: Disable the alert without deleting it
• All: Monitor all resources of a type (e.g., all Deployments cluster-wide)
• Specific: Target individual resources by namespace and name

Rule Expressions with JavaScript

Write alert conditions using JavaScript expressions:

// Alert when CPU exceeds 80% for 5+ minutes
${evaluation_period_minutes} >= 5 && ${max(cpu_utilization_arr)} >= 80
 
// Alert on deployment lifecycle events
${phase} == "START" || ${phase} == "END"
 
// Alert when Autopilot logs missing for 3+ minutes
${missing_duration_minutes} >= 3

Message Templates

Customize notification content with variable interpolation:

⚠️ ${alert_title}
Workload: ${namespace}/${workload_name}
CPU Usage: ${max(cpu_utilization_arr)}%
Memory Usage: ${avg(memory_utilization_arr)}%
Time: ${alert_time}

Alert Logs and Audit Trail

Every alert execution is logged with:

• Timestamp and alert details
• Channel delivery status
• Response from notification endpoints
• Error messages and debugging information
• 30-day retention period

How Alerts Fits Into Wave

Alerts complements other Wave features:

Common Use Cases

Production Incident Response

• Alert: CPU > 90% for 3+ minutes on production deployments
• Channel: PagerDuty via HTTP webhook
• Result: On-call engineer receives immediate page

Development Team Notifications

• Alert: Deployment START/END events in staging namespace
• Channel: Slack channel #deployments-staging
• Result: Team visibility into deployment activity

Capacity Planning

• Alert: Average memory utilization > 80% across all deployments
• Channel: Slack DM to infrastructure team lead
• Result: Early warning to add cluster capacity

SLA Monitoring

• Alert: Autopilot logs missing for 5+ minutes
• Channel: HTTP webhook to custom dashboard
• Result: Automated SLA breach tracking

Next Steps

• Getting Started - Set up your first alert channel and rule
• How it Works - Technical deep-dive into alert evaluation and delivery