Kubernetes

Idle GPU node capacity on nimbus-data-use2

Why: 47% of the cluster's paid node capacity is unrequested by any pod. GPU (g5/p4d) nodes sit idle between training jobs — the most expensive idle in the fleet.

Recommended action: Enable Karpenter consolidation (or Cluster Autoscaler scale-down) and set a lower GPU node floor. Schedule training on Spot capacity pools so idle GPUs deprovision.

Compute Savings Plan on the steady-state node baseline

Why: Node fleet is only ~47% committed while $22,505/mo runs on on-demand. Your always-on floor is stable enough to commit without losing burst flexibility.

Recommended action: Purchase a 1-year No-Upfront Compute Savings Plan sized to the on-demand baseline (keep burst headroom on-demand). Projected ~30% cut on the committed portion.

Idle node capacity on nimbus-prod-use1

Why: Bin-packing slack: nodes are provisioned well above aggregate pod requests, so you pay for capacity no workload asked for.

Recommended action: Tune Karpenter consolidation and right-size the node group instance shapes to fit actual pod requests more tightly.

Idle capacity on nimbus-staging-use1 (over-provisioned)

Why: Staging runs at only 55% allocation efficiency — nearly half the node spend is idle. Non-prod clusters rarely need 24/7 capacity.

Recommended action: Scale staging node groups down off-hours (scheduled scale-to-zero) and move to Spot; staging workloads tolerate interruption.

Graviton migration for prod node groups

Why: Production node groups run x86 (m5/c5). Your workloads are containerized and multi-arch friendly — Graviton (m7g/c7g) delivers ~20% better price-performance.

Recommended action: Build multi-arch images and migrate the m5/c5 managed node groups to m7g/c7g. Roll namespace by namespace behind node affinity.

Spot for fault-tolerant data & ML workloads

Why: Training jobs and batch data pipelines are interruption-tolerant but run largely on on-demand nodes today.

Recommended action: Move these namespaces onto a Spot node pool with on-demand fallback (Karpenter capacity-type spot). Checkpoint long training runs to resume after reclaims.