Node Warmup
Starts a node before the next Pod needs one. In one Wave benchmark run, the first new Pod was ready in 3 seconds instead of 31.
The new Pod sits in Pending for 31 seconds while a node starts
Wave 3.1.0 release post (one benchmark run, first new Pod); Wave Node Warmup docs; karpenter.sh v1.14, NodeClaims and CapacityBuffers.
By default, Karpenter launches a node only after a Pod is Pending and no node has room for it. The instance has to launch, register as a Node, and initialize before the Pod can start (karpenter.sh).
In one Wave benchmark run on EKS, the first new Pod waited 31 seconds. The Node Warmup docs estimate the split: about 15s to launch the node, 5s for kubelet registration, 10s to pull the image, and 1s to schedule.
Karpenter can keep spare room too, with CapacityBuffers (an alpha feature) or a static NodePool. A CapacityBuffer holds room with placeholder Pods that are never created as real Pods, so the workload images are not pulled in advance (karpenter.sh).
31s and 3s: the first new Pod in one Wave benchmark run (Wave 3.1.0 release post; EKS, c5n.xlarge and c6i.xlarge Spot, ap-northeast-1). The split of the 31s is an estimate from the Wave Node Warmup docs.
How Wave does it
- PodPending: no node has room for it
- launchesNodeClaimLaunched: the instance is created
- kubelet joinsNodeRegistered and initialized as a Node
- placeskubeletpulls the image, then starts the container
- Wave Corefinds no node that fits the largest Pod plus a 10% buffer
- no roomWarmup Podmakes Karpenter launch a node now
- node joinsWarm nodeconsolidation paused 10 min; images pulled when pre-pull is on
- placesNew Podstarts on a node that has already joined
Find the gap
Wave Core tracks the largest Pod in each NodePool. When no node can fit it plus a 10% buffer (the default), Node Warmup starts a node.
Start a node early
A warmup Pod makes Karpenter launch a node now. Max Pod mode runs always or on a cron schedule. Trigger mode starts a set number of nodes of a given size on a cron schedule.
Keep the node ready
While warmup runs, Karpenter consolidation is paused for the NodePool, 10 minutes by default, and warmed nodes carry the do-not-disrupt annotation. When image pre-pull is on, Wave pulls the workload images onto the node in advance. Pre-pull is off by default.
Results
until the first new Pod was ready, in one Wave benchmark run
Wave 3.1.0 release post: one benchmark run on EKS, first new Pod only