ADR 0009 — Self-Hosted OpenShift IPI over ROSA HCP

Status: Accepted (D9, docs/deployment-plan.md, 2026-07-09). Build-discovered — this decision was not planned; it replaced a locked earlier choice mid-Phase-1.

Context

The original plan (decision D1) targeted ROSA HCP as the primary-site platform, following Red Hat’s own Cloud Experts reference deployment for OpenShift Virtualization and the fact that AWS Windows License Included for OpenShift Virtualization on ROSA targets HCP specifically (Architecture §4.1, §4.4). Two Terraform apply attempts against this sandbox account changed that plan.

First attempt failed with CLUSTERS-MGMT-400: Invalid number of compute nodes: 2. Hosted clusters require that the compute nodes be a multiple of the private subnets 3 — a mundane replicas-vs-AZ mismatch (2 replicas against a 3-AZ VPC). This result was itself useful evidence: it proved ROSA HCP enablement (OCM auth, account linking, IAM) was not the problem — the cluster create got far enough to validate real HCP-specific constraints before failing on configuration, not access.

Second attempt, after fixing the replica count, hit a harder wall: CLUSTERS-MGMT-400 billing account not linked to organization at the aws marketplace. This is an AWS Marketplace billing-link requirement that is console-only — verified no CLI/API path exists (the aws CLI has no ROSA service; the rosa CLI has no billing-accounts subcommand). This particular sandbox account had that link unset, and there was no scripted or automatable way to set it.

The pivot. Drop ROSA, install self-hosted OpenShift via IPI, which needs only AWS credentials and a pull secret (mintable from an existing OCM refresh token) — sidestepping the Marketplace billing wall entirely, since self-hosted IPI has no Marketplace-billing dependency at all. The metal worker for OpenShift Virtualization became a plain post-install MachineSet rather than a ROSA machine pool — simpler to add after IPI than as an HCP machine pool.

Result, measured. IPI installed end-to-end (3 masters + 3 workers Ready, console up) in 44m35s; the metal Machine took ~20 min separately from provision to node-Ready; the CNV operator installed in ~2 min; HyperConverged converged in ~5 min; the first test VM booted (create-to-Running) in 18s.

The general lesson. The first failed attempt (replicas-vs-AZ) proved enablement was fine; only the second attempt exposed the real platform-level gap (Marketplace billing linkage). Concluding "ROSA HCP is blocked" from the first error alone would have been wrong — it took a second, cheap terraform apply to find the actual blocker. The reusable practice: verify Marketplace/billing linkage — and AZ-vs-replica math — with a cheap apply before assuming a platform enablement gap.

Decision

Install the primary-site cluster as self-hosted OpenShift 4.20 via IPI (sym-aws.sandbox5301.opentlc.com, using openshift-install create cluster with an installer-provisioned VPC), replacing the planned ROSA HCP deployment. Add the OpenShift Virtualization metal worker as a plain post-install MachineSet (virt-metal). The infra/aws ROSA-HCP Terraform is superseded; the pattern generalizes as guidance for future infra choices in this repo: prefer a cheap apply to confirm a platform blocker before designing around it.

Consequences

Pros

  • Sidesteps the Marketplace billing wall entirely — IPI’s credential model (AWS creds + pull secret only) has no dependency on the AWS Marketplace billing-link chain that blocked ROSA HCP in this account, and that link has no CLI/API remediation path today.

  • Simpler credential/dependency chain for a PoC/reference architecture: AWS creds
    pull secret is strictly less fragile than ROSA HCP’s OCM-account + Marketplace-billing dependency chain — fewer moving parts that can independently block a from-scratch environment.

  • Measured and fast: a 44m35s full cluster install is well within the "cluster create ~40 min" expectation the project already budgets for background/loop polling; the metal worker and OpenShift Virtualization convergence add well-bounded, separately measured time on top.

  • Metal capacity is a plain MachineSet, simpler to reason about and to scale (including to zero for cost control) than a ROSA HCP machine pool.

Cons

  • Self-hosted means the project now owns cluster upgrades and SRE — control-plane lifecycle, patching, and platform operational burden that ROSA’s managed control plane exists specifically to remove. This is ROSA’s actual value proposition, and this decision gives it up for the primary site.

  • AWS Windows License Included is a ROSA-only feature — it was confirmed GA (~March 2026) specifically for OpenShift Virtualization on ROSA, bundled per-vCPU into the bare-metal EC2 cost. Self-hosted IPI forfeits this: the AWS-side Windows licensing burden reverts to BYOL, the same problem Architecture §4.4 already had to solve on the alternative (Dedicated Hosts, pre-Oct-2019 perpetual license) path.

  • This decision should be revisited if/when the AWS Marketplace billing link is established for this account — nothing about the IPI choice is permanent, and ROSA HCP’s benefits (managed control plane, native LI Windows) remain the better fit for this design once that blocker is gone.

Continue to ADR 0010: stretched grid, multicluster fallback.