ADR 0002 — Staged Minimal Footprint: AWS Before IBM Cloud

Status: Accepted (D1)

Context

The target architecture is a stretched grid spanning AWS (primary) and IBM Cloud (burst), joined by an interconnect (ADR 0004). Building both sides and the link at once multiplies the variables in flight: platform install, Symphony config, Windows golden image, network invariants, and cross-cloud connectivity, all changing simultaneously. IBM Cloud burst capacity (bx2d.metal.* bare-metal workers, ROKS OpenShift Virtualization, VNI-routable VMs) also carries real, standing cost and a longer lead time that isn’t needed until the burst mechanism itself is being tested.

The build to date bears this out. Phase 1 (self-hosted OpenShift IPI, OpenShift Virtualization, layer2 UDN) completed in 44m35s for the cluster and ~5 min for HyperConverged convergence. Phase 2 (Symphony management hosts, Windows golden image, compute-host join) consumed by far the largest share of engineering time and defect surface — a WAN-tuning idempotency-guard false positive, a CDI StorageProfile gap, an eval-license <ProductKey/> trap, roughly ten failed attempts at a silent Windows Symphony install before the pre-extracted-MSI pivot, and firewall/service-naming/MTU root causes in the Windows compute-host join. None of this required IBM Cloud to be provisioned at all — proving that the AWS side alone has enough unknowns to justify staging.

Running cost is also a live constraint: the baseline AWS footprint (3 masters + 3 workers, m6i.xlarge class, plus one m5zn.metal node) runs at roughly $5.2/hr, of which the single metal node is ~$4/hr and can scale to zero when paused, re-provisioning in ~20 min. Standing up a second, IBM-side metal pool before the AWS side is even join-verified would double that exposure for unvalidated work.

Decision

Build and de-kink the entire AWS-side stack first — OpenShift Virtualization, RHEL management hosts, the Windows golden image, and the Windows compute-host join — using a minimal single-metal-node footprint, before provisioning any IBM Cloud infrastructure. IBM Cloud spend (VPC, ROKS, bare-metal workers, interconnect) starts only once the AWS-side grid is functionally proven against Phase 1/2 exit criteria.

Consequences

Pros

  • Isolates defect surface: every Windows-join, Symphony-config, and network-invariant bug found so far was findable and fixable entirely within AWS, with no cross-cloud variables to rule in or out.

  • Minimizes standing cost during the highest-defect-density phase: one m5zn.metal node (~$4/hr) instead of two metal pools running in parallel while the harder half of the build (the Windows image pipeline and join) was still being solved.

  • Matches the reference architecture’s own phased build plan and gives each later phase (interconnect, manual burst proof, custom provider) a working, already-verified AWS side to join against, rather than debugging both ends of a link at once.

Cons

  • Cross-cloud burst is unproven until Phase 3: nothing in Phases 1-2 exercises the interconnect, cross-cloud DNS, or a real Windows VM joining over WAN latency — the stretched-grid WAN sensitivities (MTU, keepalives, EGO_DYNAMIC_HOST_WAIT_TIME) remain design-time reasoning, not measured fact, until then.

  • A single metal node is a SPOF by construction — acceptable only because this is a PoC build being staged for correctness, not a production-sized or HA deployment.

  • Staging defers discovery of any IBM-Cloud-specific blocker (the VNI allowlist gate, ROKS bare-metal billing-model ambiguity) until later in the timeline than a fully parallel build would.

Continue to ADR 0003: Windows Server 2019.