ADR 0008 — Symphony Management Hosts as RHEL VMs On-Cluster

Status: Accepted (D8)

Context

Symphony’s master and master-candidate hosts (EGO/SOAM control plane, Host Factory) need to be reachable from the same grid network as the Windows compute hosts, on the fixed base-port range, with no NAT — Symphony’s EGO registration is hostname/IP-anchored (see ADR 0001 and Network model). Two placements were considered: RHEL VMs on the same OpenShift Virtualization cluster as the Windows compute fleet, or native EC2 instances outside OpenShift in the same VPC.

Putting the management hosts on the same layer2 User Defined Network (UDN) as the compute VMs makes the choice straightforward inside a single site: management and compute hosts see each other’s real, stable IPs with no NAT and no exposure problem, because both sides live on the same cluster network. The alternative — native EC2 management hosts, outside OpenShift, in the same VPC — trades that simplicity for VPC-routable IPs by construction, at the cost of managing instances outside the GitOps-reconciled cluster and outside OpenShift Virtualization’s VM lifecycle entirely. Native EC2 removes the NLB listener-count constraint against SSM’s pinned port range, at the cost of a separate management surface — this design defers that trade to Phase 3, when the burst site (and therefore the master-exposure question) becomes real.

A build-time constraint has since sharpened the trade. Rolling out CPU Manager’s static policy on the virt-metal node (required for KubeVirt’s d1 dedicated-CPU instance types) forces a reboot of that metal node. Because the Symphony master and master-candidate VMs (sym-master1, sym-master2) sit on RWO (EBS) storage, they are not live-migratable — the only safe way to apply that node-level change was to gracefully stop the master VMs first, rather than let the MachineConfigOperator drain hang on KubeVirt’s eviction-blocking PodDisruptionBudget (git commit 8a8a79f). This is a concrete, build-proven instance of the operational cost this ADR accepts below.

Decision

Run the Symphony master and master-candidate hosts as RHEL VMs on the same OpenShift Virtualization cluster as the Windows compute fleet, on the same layer2 UDN, reconciled via gitops/apps/symphony-mgmt/ (per ADR 0007). Defer the native-EC2 alternative to a Phase 3 revisit if NLB-based master exposure to the burst site proves awkward. See Symphony management for the hosts as deployed.

Consequences

Pros

  • No NAT, no exposure problem between management and compute hosts within one site — both sides of the grid’s most latency- and reachability-sensitive traffic (vemkd <→ PEM, SSM <→ SIM) share the same real, stable-IP network by construction.

  • One GitOps-reconciled surface for the whole in-site grid (masters and compute together), rather than splitting the management plane onto infrastructure outside OpenShift’s lifecycle and outside Argo’s reach.

  • Keeps the design symmetric across sites: the same pattern (management + compute VMs on one cluster’s UDN) is expected to repeat on the burst site, rather than inventing a site-specific management-host placement.

Cons

  • Masters on RWO storage are not live-migratable — node maintenance on the shared metal requires gracefully stopping the master VMs first, not a transparent live migration. This is not theoretical: it is exactly what the CPU Manager KubeletConfig rollout required in the build (git commit 8a8a79f). Any future node-level change to the virt-metal pool carries the same operational cost until the masters move to RWX storage (FSxN) or off-cluster.

  • Master exposure to the burst site is still an open design point. The NLB path requires pinning SSM’s port range small enough to fit typical NLB listener limits (~50); the native-EC2 path removes that constraint but reintroduces a management surface outside GitOps. This ADR does not resolve that trade — it is explicitly deferred to Phase 3.

  • Co-locating masters and compute on the same cluster means a metal-node-level incident (capacity, node failure) can affect both planes at once — there is no infrastructure isolation between "the grid’s control plane" and "the grid’s compute capacity" the way native-EC2 masters would provide.

Continue to Reference for the port matrix and config fragments these decisions depend on.