Install phases

Imperative bootstrap, then GitOps. You run openshift-install, one MachineSet, and two bootstrap scripts by hand; everything after that reconciles from gitops/ via Argo CD. This is exactly how the reference cluster sym-aws was built — not an idealized rewrite.

Honesty up front

Phase 0 (prereqs) and Phase 1 (OpenShift + OpenShift Virtualization) are proven — IPI install, HyperConverged, the metal MachineSet, and the CPU Manager KubeletConfig are all measured on the live cluster. Phase 2’s Linux management plane (RHEL masters, EGO/SOAM cluster symgrid) is proven — egosh resource list shows both masters ok. Phase 2’s Windows golden image (ws2019-symphony DataSource: WS2019 + virtio
qemu-guest-agent + Symphony compute host, sysprep-generalized) is proven — built clean, boots, clones.

What is NOT yet proven: the Windows compute VMs joining the EGO grid. Four compounding root causes were found and fixed (see Build traps) and the join was demonstrated live against running clones, but as of the last logged handoff a WinRM password-desync regression was blocking re-verification. Phases 3-5 (IBM Cloud burst, the KEDA demo, the custom Host Factory provider) are designed, not built.

Reference cluster: sym-aws (OpenShift 4.20 IPI, us-east-1, 3 control-plane + 3 worker m6i.xlarge + 1 m5zn.metal for OpenShift Virtualization). Install artifacts (kubeconfig, kubeadmin password, pull secret) live outside the repo, in ~/clusters/sym-aws/ — never in git.

At a glance — the order that works

§1 phase0-checks.sh
  -> §2 openshift-install create cluster (IPI)
  -> §3 virt-metal MachineSet + CPU Manager KubeletConfig
  -> §4 bootstrap-gitops.sh
  -> §5 create-secrets.sh
  -> §6 upload-installers.sh
  -> §7 Linux management hosts (masters)
  -> §8 golden image pipeline (2 stages + 1 one-time MSI extract)
  -> §9 traps (read before you hit them -- xref:41-traps.adoc[full catalog])
  -> §10 Windows compute pool + join (open item)

1. Prerequisites

Run until green:

scripts/phase0-checks.sh

It checks local tooling (aws, rosa, oc, virtctl, terraform, kustomize, ibmcloud), AWS credentials and EC2 standard-vCPU quota >= 48 (an m5zn.metal alone is 48 vCPU — request an increase before Phase 1 if this fails), ROSA/OCM login (harmless — see the D9 note below), IBM Cloud login and bx2d.metal visibility (a Phase 3 lead item — open the VNI-allowlist ticket with IBM now, it is the longest lead-time item in the whole plan even though the burst cluster comes later), and local install media in installers/ (the Symphony eval installers, entitlement file, and the WS2019 Evaluation ISO, all gitignored, never uploaded anywhere).

Secrets that must exist out-of-band before you start, never in git: OCM token, AWS credentials, IBM API key, Windows local-admin password, Symphony admin password, the entitlement file, and (Phase 3) a VPN PSK.

D9 platform note

The plan originally targeted ROSA HCP. rosa/OCM tooling is still worth having, but the actual Phase 1 path is self-hosted OpenShift IPI — ROSA HCP cluster creation hard-blocked on an AWS Marketplace billing link with no CLI/API workaround in this account. IPI needs only AWS credentials and a pull secret. The ROSA-HCP Terraform in infra/aws is kept in tree for reference but superseded — don’t run it. See ADR 0010 — Stretched Single Grid, Multicluster Federation as the Fallback and the D9 decision in the reference architecture for the full rationale.

2. OpenShift on AWS (IPI)

mkdir -p ~/clusters/sym-aws
openshift-install create cluster --dir ~/clusters/sym-aws
# stable-4.20, cluster name sym-aws, base domain a public Route53 zone,
# us-east-1, installer defaults (3 control-plane + 3 worker), OVN-Kubernetes CNI.

Measured: 44m35s end to end (3 masters + 3 workers Ready, console up). Install artifacts stay in ~/clusters/sym-aws/ — this directory is outside the repo and must never be committed.

export KUBECONFIG=~/clusters/sym-aws/auth/kubeconfig
oc whoami   # confirm cluster-admin (kubeadmin)

3. Bare-metal MachineSet + CPU Manager

OpenShift Virtualization requires bare metal. The metal MachineSet is committed manifest, not templated Terraform — it was hand-derived from the installer’s own worker MachineSet after IPI created the cluster:

oc apply -f gitops/clusters/rosa/virt-metal-machineset.yaml
oc get machineset -n openshift-machine-api -w    # wait for 1/1 Ready

m5zn.metal, replicas: 1 by design (roughly $4/hr — don’t scale without a cost decision). Measured Machine-provision-to-node-Ready: ~20 min. The template labels the node node-role.symphonygrid/virt: "" — everything downstream selects on that label, not on instance type directly.

Once the operators land (via GitOps, §4), also apply the CPU Manager KubeletConfig — this is a hard scheduling prerequisite, not an optimization:

oc apply -f gitops/clusters/rosa/virt-metal-cpumanager.yaml

KubeVirt’s d1. instance types (used for every Windows compute VM, §10) bake in dedicatedCPUPlacement: true, which hard-requires the node’s kubelet to run cpuManagerPolicy: static. Without it, every VM using a d1. instancetype is permanently ErrorUnschedulable — while the Argo Application still shows Synced/Healthy, because the VMs silently never schedule (see Build traps). It is scoped to a dedicated virt-metal MachineConfigPool (selecting only the metal-labeled node), specifically so it does not force a rolling reboot of the three non-metal workers.

4. Bootstrap GitOps

export REPO_PAT_PATH=~/clusters/git-pat   # a file containing a PAT for the repo, chmod 600
scripts/bootstrap-gitops.sh rosa

(The argument is rosa — a naming legacy from before the D9 pivot; it now means "the AWS-side cluster," not the ROSA platform.) The script installs the OpenShift GitOps operator, grants the Argo application controller cluster-admin (needed to manage Operators/HyperConverged/VMs), creates the git repository credential over HTTPS+PAT (Argo’s SSH access hit a source-IP restriction on this project — HTTPS+PAT was the durable fix), and applies the rosa app-of-apps.

oc get applications -n openshift-gitops -w

Argo then syncs gitops/clusters/rosa/: the OpenShift Virtualization operator
HyperConverged, the Pipelines operator, the symphony namespace, the layer2 UDN (no NAT, base port 7869), the StorageProfile override (see Build traps), and the CPU Manager KubeletConfig/MachineConfigPool from §3.

Exit: HyperConverged Available on the metal node; virtctl can start a test VM (measured boot time: 18s create-to-Running).

5. Secrets

Nothing in §4/§6/§7 works without secrets, and none of them are in git. scripts/create-secrets.sh renders and creates every Secret/ConfigMap that must not live in the repo, from environment variables — names only, values never committed, never printed in this doc:

Env var Required Purpose

SYM_ADMIN_PASSWORD

yes

Symphony OS-level admin account password

WIN_ADMIN_PASSWORD

yes

Windows local Administrator password (baked into sysprep unattend.xml)

SYM_MASTER_NAME

no (defaults sym-master1)

primary master hostname

SYM_MASTER_IP

yes

primary master’s grid-UDN IP

SYM_MASTER2_NAME / SYM_MASTER2_IP

no

candidate master, if present

EGO_ADMIN_PASSWORD

no (auto-generated if unset)

EGO’s built-in web/REST Admin user — alphanumeric only, a different password policy than SYM_ADMIN_PASSWORD

 
export SYM_ADMIN_PASSWORD=... WIN_ADMIN_PASSWORD=... SYM_MASTER_IP=10.10.0.31 \
       SYM_MASTER2_NAME=sym-master2 SYM_MASTER2_IP=10.10.0.34
scripts/create-secrets.sh

This creates: symphony-entitlement (the eval entitlement file), symphony-admin / win-admin (passwords), sym-master1-cloudinit / sym-master2-cloudinit (rendered cloud-init), and win-sysprep-join (the per-VM sysprep ConfigMap — rendered unattend.xml + firstboot-symphony.ps1).

Regeneration hazard

If EGO_ADMIN_PASSWORD is unset, the script mints a fresh random one every run — re-running the script to fix one unrelated ConfigMap silently rotates the already-joined masters' EGO admin credential. Always pull the existing password back out of the live symphony-admin Secret and pass it through explicitly when re-rendering anything after the masters are up. See Build traps.

6. Upload installer media

scripts/upload-installers.sh

Copies the Symphony eval installers and the WS2019 eval ISO from the gitignored local installers/ directory onto the installer-repo PVC (an in-cluster nginx pod) — the media server every VM pulls install bits from over the grid UDN, since none of these hosts have outside internet or a Red Hat subscription. installer-repo must already be synced by Argo (§4) before this script has a pod to copy into. A RHEL9 net-tools RPM is also served this way — not an IBM artifact, required because the Symphony 7.3.2 Linux installer hard-requires netstat, absent by default on RHEL9.

7. Linux management hosts (masters)

gitops/apps/symphony-mgmt/ (synced automatically once create-secrets.sh has produced the two *-cloudinit Secrets it depends on) creates 2x RHEL 9 VMs on the grid UDN: sym-master1 (primary) and sym-master2 (candidate). Their cloud-init:

  1. Pulls the Symphony Linux installer from installer-repo.

  2. Silent-installs with the license accepted and the eval entitlement, under a dedicated egoadmin OS account.

  3. Applies symphony/config/ fragments (keepalives + dynamic-host settings, pinned SSM port ranges) via apply-config.sh.

  4. sym-master2 joins sym-master1.

Exit: egosh resource list shows both sym-master1 ok and sym-master2 ok with live load metrics, not dashes. Port matrix verified live via ss: sym-master1 listens the full 7-port block TCP 7869-7875 + UDP 7869; sym-master2 (non-primary) listens TCP 7869 + 7871 + UDP 7869 only — expected, not drift. If `sym-master2’s resource row shows all dashes, check master-to-master DNS first — see Build traps.

8. Windows golden image pipeline

The contract: the golden image is fully baked except identity — Windows Server 2019, virtio, qemu-guest-agent, Symphony 7.3.2 compute host installed but not joined, sysprep-generalized. Per-VM identity (hostname, master addresses, join) happens at clone time in the sysprep specialize pass, never baked into the image.

WS2019 eval ISO ---+
virtio-win ISO -----+  Stage 1: windows-efi-installer Tekton pipeline (upstream
autounattend.xml ---+      KubeVirt community example, resolved via the Tekton hub
                            resolver -- NOT a Red Hat-productized artifact)
                           -> vanilla golden: WS2019 + virtio + qemu-guest-agent
                              (measured 15m56s)
                           output: DataSource ws2019-base

ws2019-base --------+  Stage 2: customize task (authored in this repo)
egocore/soamcore msi     boot temp VM from ws2019-base
(pre-extracted,           -> install-symphony.ps1 (msiexec /qn against
served by installer-repo)    pre-extracted MSIs)
                          -> sysprep /generalize /oobe /shutdown
                          output: DataSource ws2019-symphony
# render + serve the sysprep ConfigMaps the pipelines point at (never an external URL):
scripts/render-image-pipeline-media.sh

# stage 1 (deliberately NOT in the app's kustomization -- Argo must never
# auto-submit a pipeline run; each PipelineRun documents its own prereqs):
oc create -f gitops/apps/image-pipeline/pipelinerun-stage1.yaml
oc get pipelinerun -n symphony -w

# stage 2, once ws2019-base is Ready:
oc create -f gitops/apps/image-pipeline/pipelinerun-stage2.yaml
oc get pipelinerun -n symphony -w

One-time MSI extract

symeval-7.3.2.0.exe --extract is a real 3-click InstallShield wizard, not a headless extraction — automating it inside every stage-2 run proved too flaky (see Build traps). The durable fix, executed once: extract the two compute-host MSI packages on a throwaway scratch VM and publish them to installer-repo:

scripts/extract-symphony-msi.sh

Publishes egocore-4.0.0.0.msi (117 MiB) and soamcore-7.3.2.0.msi (221 MiB) to installer-repo via a WebDAV PUT endpoint. install-symphony.ps1 then just msiexec /qn`s the two pre-published files directly — zero self-extraction, zero UI automation, in the standing pipeline from here on. Re-run only if `symeval-7.3.2.0.exe itself changes (e.g. a fix pack). Measured: extract-VM run ~13 min wall clock (launch to clean shutdown, one manual click required); egocore upload ~63s, soamcore upload ~118s.

Verify

oc get datasource ws2019-base -n symphony
oc get datasource ws2019-symphony -n symphony
virtctl console <stage-2-vm>   # install-symphony.ps1's progress lines live off COM1
Not a Red Hat-productized artifact

The windows-efi-installer pipeline is an upstream KubeVirt community example pipeline (kubevirt/kubevirt-tekton-tasks), run on the fully-supported OpenShift Pipelines operator — it is not Red Hat-productized, and its README documents neither guest-agent injection nor sysprep/generalization as part of stage 1 itself, which is why this repo authors stage 2 as its own Pipeline. Say "supported platform, community-example pipeline, adapted in-repo," not "Red Hat productized."

9. Traps

Read Build traps before you run any of the above for the first time — every row in that catalog cost real hours on the reference build.

10. Windows compute pool + join (open item)

gitops/apps/symphony-compute-win/ is a VirtualMachinePool cloning the ws2019-symphony golden DataSource, using the d1 instancetype family (dedicated CPU — required both for AWS Windows License-Included compliance and to satisfy §3’s CPU Manager KubeletConfig prerequisite). Each replica’s sysprep specialize pass runs firstboot-symphony.ps1: enable WinRM first, open inbound EGO firewall, rewrite EGO_MASTER_LIST in ego.conf directly, start the LIM service.

Measured on real reclones: VM Running phase reached ~2-3 min from delete; AgentConnected=True (qemu-guest-agent up) by ~5 min. Full clone-to-joined timing is the actual Phase-2 exit measurement and is genuinely open — four compounding root causes were fixed and the join was demonstrated live against running clones, but a WinRM password-desync regression was blocking re-verification as of the last logged handoff. This is the one unproven step in an otherwise fully-verified pipeline.

Timings and costs measured on the live build

Step Measured Source

IPI cluster install (§2)

44m35s

phase-log "Phase 1 workflow run 1"

Metal Machine provision → node Ready (§3)

~20 min

phase-log "Phase 1 EXIT CRITERIA MET"

HyperConverged converge

~5 min

same

Test VM boot (create → Running)

18s

same

Golden image stage 1 (§8)

15m56s

phase-log "stage-1 GREEN"

MSI-extract scratch-VM run (§8, one-time)

~13 min

phase-log "MSI-carve pivot EXECUTED"

Compute VM clone → Running

~2-3 min

phase-log "compute VMs recloned…​"

Compute VM Running → AgentConnected

~5 min

same

Running cluster cost (3+3 m6i.xlarge + 1 m5zn.metal)

~$5.2/hr

phase-log, HANDOFF-windows-join.md

Scale virt-metal MachineSet to 0 (park)

saves ~$4/hr, re-provisions in ~20 min

same

Continue to Build traps — the catalog every gotcha above links back to.