Wave Karpenter

Spot Workload Placement

Keeps the first replicas of each workload on On-Demand and places the rest on Spot, by replica number, with no change to your YAML.

The challenge

AWS takes back a Spot node with 2 minutes of notice

AWS, Amazon EC2 User Guide, Spot Instance interruption notices; AWS, Amazon EC2 Spot Instances FAQ; karpenter.sh v1.14 docs, NodePools and Scheduling.

AWS issues a Spot interruption notice two minutes before EC2 stops or terminates the instance (AWS EC2 User Guide). In exchange, Spot Instances can save up to 90% off On-Demand prices (AWS Spot FAQ).

By default, when a NodePool allows both, Karpenter launches Spot before On-Demand (karpenter.sh). Nothing keeps a part of the replicas on On-Demand, so every replica can be on nodes that get the notice.

Karpenter can split a workload with one NodePool per capacity type and a topology spread, and its docs call this a crude method (karpenter.sh).

Without Wave

By default, when a NodePool allows Spot and On-Demand, Karpenter launches Spot first, so all 10 replicas can land on Spot. A 2-minute interruption notice drains one node, and its 4 replicas must start somewhere else.

With Wave

With threshold 5, Wave places Pods by replica number: 1 to 5 on On-Demand, 6 to 10 on Spot. A 2-minute notice still drains a Spot node, and replicas 1 to 5 keep running.

2-minute notice: AWS EC2 User Guide. Spot before On-Demand when a NodePool allows both: karpenter.sh v1.14 docs. Placement by replica number, blue for On-Demand and orange for Spot: Wave Spot Workload Placement docs. The node split is illustrative.

How Wave does it

From a new Pod to its node

Karpenter by defaultNo On-Demand base

NodePool
allows Spot and On-Demand
used by
Karpenter
launches Spot first when both are allowed
places
Deployment
every replica can land on Spot
reclaims
Spot interruption
2-minute notice, then the node drains

Wave Spot Workload PlacementBase kept on On-Demand

Wave Console
threshold per workload, or a CR
Pod create
Mutating webhook
intercepts each new Pod
injects
nodeAffinity
set by the replica number
places
Placement
1 to threshold On-Demand, the rest Spot

Spot before On-Demand: karpenter.sh v1.14 docs, NodePools. 2-minute notice: AWS EC2 User Guide. Threshold, webhook, and nodeAffinity by replica number: Wave Spot Workload Placement docs.

Set a threshold

Pick a Deployment, an Argo Rollout, or an OpenShift DeploymentConfig in Wave Console, or apply a WaveKarpenterSpotPlacement CR, and set the Pod Spot Threshold.

Wave sets the affinity

A MutatingWebhookConfiguration adds nodeAffinity to each new Pod by its replica number. With threshold 5, Pods 1 to 5 go to On-Demand and Pods 6 and up go to Spot. Your YAML does not change.

Choose Prefer or Require

Prefer uses Spot and falls back to On-Demand when Spot is short. Require keeps the Pod Pending until Spot capacity exists. The webhook needs a TLS certificate, from cert-manager or your own.

Related in Wave Karpenter

Karpenter Dashboard →Node Warmup →Read the docs →

Book a demo Start free