ADR 0015 — VM-Level Burst on Semi-Static Metal, Not Node-Level Autoscaling

Status: Accepted (Architecture §5.3).

Context

Burst capacity has to come from somewhere, and there are two granularities to scale at: the physical/bare-metal node the virtualization stack runs on, or the VM running on top of already-provisioned metal.

Node-level (bare-metal) autoscaling does not fit a burst-grade elasticity requirement:

  • On the ROKS burst site, bare-metal workers bill monthly (OCP license; infra billing hourly-vs-monthly ambiguity is unresolved and needs confirming with IBM), and take tens of minutes to hours to actually join a cluster — raw VPC metal itself provisions "in 10 minutes or less" per IBM, but a field report puts the full ROKS+ODF+virt stack at roughly 2-3 hours end to end (Architecture §5.3).

  • This matches the broader industry pattern: bare-metal node provisioning generally runs 5-30+ minutes even under favorable conditions, versus VM boot/join in low single-digit minutes on metal that is already paid for and running.

Against that, the build measured VM-level elasticity directly during Phase 2’s golden-image and join work: KubeVirt VirtualMachine clone-to-Running timings on real reclones were consistently around 2-3 minutes from delete (T0) to VMI phase Running, with AgentConnected=True (qemu-guest-agent up) by roughly T0+5 minutes, and CSI smart-clone of both root disks from the DataSource snapshot completing within ~3 minutes. That is roughly two orders of magnitude faster than the metal join path, on infrastructure that is already standing.

Host Factory’s own demand model is built around exactly this granularity: its five-script provider interface (Host Factory KubeVirt provider) treats individual machines as the unit of request/return, and its lazy return policy (billing_interval, default 60 minutes) is designed to tolerate frequent VM churn without penalty when the underlying billing granularity is coarse — which describes the ROKS metal-is-monthly situation precisely (Architecture §5.3).

Decision

Treat the bare-metal worker pool as semi-static, sized to the burst ceiling on a weekly/monthly planning horizon, not scaled reactively. Treat the VM as the burst unit: Windows compute VMs clone from the golden image and join the grid in minutes on already-provisioned, already-paid-for metal, so the marginal infrastructure cost of an individual burst VM is effectively zero once the metal floor exists. This is the granularity the custom Host Factory KubeVirt provider (ADR 0005) is built to drive, and its lazy-return tuning is the mechanism that absorbs the metal pool’s coarse billing.

Consequences

Pros

  • VM clone-to-join is measured in minutes in this build, not the tens-of-minutes-to-hours a fresh bare-metal node join takes on ROKS — the elasticity claim is grounded in real timing data, not projection.

  • Matches Host Factory’s native demand model (per-machine request/return, tunable lazy return) without fighting it — no adapter is needed to make the scheduler’s own semantics fit a monthly-billed metal floor.

  • Decouples burst responsiveness from metal provisioning time entirely: once the metal pool exists, burst latency is bounded by VM boot/join, not by cluster infrastructure changes.

Cons

  • The metal pool is a standing cost floor, sized to the burst ceiling rather than to current demand — capacity that goes unused most of the time is still being paid for (.metal-class instances run ~$4/hr+ per the project’s cost-awareness rule), which is the direct trade for VM-level responsiveness.

  • Capacity planning moves to a weekly/monthly horizon for the metal pool itself, separate from the minutes-level VM burst loop — under-provisioning the metal ceiling caps burst headroom regardless of how fast VMs can join, and re-sizing the metal pool inherits all of node-level autoscaling’s slow, coarse-billed characteristics that this design otherwise avoids.

  • ROKS’s monthly/hourly billing ambiguity for bare-metal workers is explicitly unresolved at time of writing (Architecture §5.3) — the "metal floor is a sunk cost, VM churn is free" argument depends on confirming that billing model with IBM, not just on the measured VM timings.

Continue to Glossary.