ADR 0013 — Dedicated-CPU d1 Instance Types + CPU Manager Static Policy

Status: Accepted (build-discovered, commit 8a8a79f; not logged in the phase log despite the commit message describing it as a fix — see Architecture §4.3 for the grid-workload/licensing rationale that motivated choosing d1 in the first place).

Context

Windows compute VMs use KubeVirt’s d1 dedicated-CPU instance-type family (Architecture §4.3), chosen for two independent reasons: Symphony compute hosts are a CPU-bound grid-scheduling workload that benefits from real CPU isolation rather than shared/overcommitted vCPUs, and Red Hat’s own guidance ties AWS Windows License Included compliance to dedicatedCpuPlacement — non-compliant VMs may fail KMS activation, suffer degraded CPU pinning, and risk licensing audits.

What was not obvious until it broke in the build: d1 bakes in dedicatedCPUPlacement: true at the instancetype level. This is a hard scheduling prerequisite, not a preference — it requires the target node’s kubelet to run with cpuManagerPolicy: static. Without that KubeletConfig in place, every VM using a d1.* instancetype is permanently ErrorUnschedulable ("didn’t match Pod’s node affinity/selector"), even though the Argo CD Application itself shows Synced/Healthy — the failure is invisible at the GitOps layer and only visible as a stuck VM.

Two further constraints shaped how the fix was rolled out:

  • The KubeletConfig had to be scoped to a dedicated virt-metal MachineConfigPool (selecting only the node-role.symphonygrid/virt-labeled metal node) rather than the shared worker pool, specifically to avoid forcing a rolling reboot of the three non-metal workers that don’t run any VMs at all.

  • The metal node itself still reboots when the Machine Config Operator rolls this out. Because sym-master1/sym-master2 (the RHEL management-host VMs) are not live-migratable — their disks are EBS/RWO, not RWX — the MCO’s node drain would hang indefinitely on KubeVirt’s eviction-blocking PodDisruptionBudget unless those masters are gracefully stopped first, before the pool rollout, and restarted after.

Decision

Ship a scoped virt-metal MachineConfigPool + KubeletConfig (gitops/clusters/rosa/virt-metal-cpumanager.yaml) that:

  • Selects only the metal virtualization node (not the shared worker pool).

  • Sets cpuManagerPolicy: static, cpuManagerReconcilePeriod: 5s, reservedSystemCPUs: "0-3".

  • Is applied with sym-master1/sym-master2 gracefully stopped first, to avoid an MCO drain hang on non-migratable VM PDBs.

Treat this KubeletConfig as a hard platform prerequisite for the d1 instance-type recommendation, not an optional tuning knob — document it alongside the d1 recommendation itself (Architecture §4.3), since at the time of the build it was only discoverable from a manifest comment and a commit message, not from the phase log or the reference architecture text.

Consequences

Pros

  • d1.* VMs schedule correctly and satisfy the AWS Windows LI compliance requirement for dedicatedCpuPlacement, with real CPU isolation appropriate to a CPU-bound grid workload.

  • Scoping the MachineConfigPool to the metal node avoids an unnecessary rolling reboot of the three non-metal workers — a real cost/disruption saving on a shared PoC cluster.

  • The graceful-stop-masters-first sequencing is now a documented, repeatable procedure rather than a rediscovered trap on every future MCO rollout against this pool.

Cons

  • reservedSystemCPUs: "0-3" (4 cores) on the metal node is capacity permanently lost to the system and unavailable for VM scheduling — this must be subtracted from every capacity-planning calculation for the virt-metal pool, not treated as free headroom.

  • Any future MCO-driven change to the virt-metal pool (KubeletConfig update, kernel arg, etc.) reboots the metal node and therefore requires the same non-migratable-master stop-first choreography every time — this is now a standing operational procedure, not a one-off migration step.

  • Because the failure mode (ErrorUnschedulable on a Synced/Healthy Application) gives no GitOps-layer signal, any other unmet dedicated-CPU prerequisite in the future would fail the same silent way unless it, too, is checked explicitly.

Continue to ADR 0014: Layer2 UDN + MTU.