Pod Scheduling Delay Detection
Records every Pod that waited in Pending, how long it waited, and each scheduler message behind the wait, and keeps them after Kubernetes deletes the events.
A Pod waits in Pending, and its FailedScheduling event is gone after 1 hour
kubernetes.io, Debug Pods, Resource Management for Pods and Containers, and the kube-apiserver reference (--event-ttl, default 1h0m0s).
"If a Pod is stuck in Pending it means that it can not be scheduled onto a node" (kubernetes.io). The reason is in a FailedScheduling event, with a message such as "0/42 nodes available: insufficient cpu".
kube-apiserver keeps events for 1 hour by default (--event-ttl). A delay during the night leaves no reason behind by morning.
Events kept 1 hour by default: kube-apiserver --event-ttl (kubernetes.io). The row and the conditions history: Wave Pod Scheduling Delay docs. The 12-minute wait, the times, and the 3-hour check are illustrative.
How Wave does it
- Podcreated, phase Pending
- schedulekube-schedulerfinds no node that fits
- no fitEventFailedScheduling, with the reason
- 1h TTLkube-apiserverdeletes events after 1 hour by default
- Wave Agentreports Pod state from every node
- gRPCWave Coremeasures the time in Pending per Pod
- per PodConditions historyevery scheduler message, with its time
- keptWave Consoledelays table, the last 7 days by default
Measure the wait
Every delayed Pod gets a row with its scheduling delay, workload, namespace, and status, Resolved or Pending. KPI cards show total, average, resolved, and unresolved delays for the last 24 hours.
Keep the reason
The conditions history keeps every scheduler condition in order, with its time and message, such as "0/5 nodes are available: 2 Insufficient cpu". For Pods with a PVC, it also shows why the volume did not bind.
See when delays happen
An hourly chart of average pending time shows whether delays cluster around deploys or peak hours. The default range is the last 7 days.