IBM Spectrum Symphony grid with Windows Server compute VMs on OpenShift Virtualization — primary on self-hosted OpenShift on AWS (IPI), bursting into ROKS (IBM Cloud). Fully automated via GitOps; built and operated by Claude agents with human decision gates.
Everything that has lead time, before any money is spent.
IPI cluster + 1× m5zn.metal MachineSet; bootstrap Argo; operators, layer2 UDN — all reconciled from git.
RHEL master VMs silent-install Symphony from the in-cluster media repo. Two-stage image pipeline bakes WS2019 + virtio + Symphony (unjoined, sysprep-generalized); clones join at first boot.
ROKS 4.20+ (OVN chosen at create), 2–3 metal workers, OCP Virt manual install. IPsec VPN, DNS forwarding, MTU verified. One WS2019 VM on ROKS joins the AWS master over the tunnel — the single most important milestone in the plan.
Metrics shim polls Symphony REST for demand; KEDA scales the ROKS Windows pool via /scale. Accepted gap: scale-in doesn't drain tasks — documented live in the demo.
Custom Host Factory KubeVirt provider replaces KEDA scale-in; Direct Link decision; Advanced Edition BYOL before eval expiry; licensing sign-offs; reference architecture v1.0 with everything measured.
Prove the whole stack on AWS with 1 metal node before any IBM spend — metal is the dominant cost.
Works on the eval entitlement (WS2022 needs a Fix Central fix the eval can't obtain); certified OCP Virt guest.
Hours to stand up, near-zero cost, and deliberately stresses Symphony's MTU/keepalive sensitivity early.
Scale-out works via /scale; scale-in is not grid-aware (no task drain) — accepted for demo, Host Factory provider is the production path.
Full product (Host Factory, Windows, multicluster) free until 2026-12-31; BYOL procurement before production.
App-of-apps per cluster; no hand-applied cluster state outside bootstrap and break-glass.
Symphony needs full Windows Server; nodes stay Linux — golden image, sysprep-cloned, identity injected at first boot.
Same layer2 UDN as compute (no NAT). Revisit native-EC2 placement if NLB exposure gets awkward in Phase 3.
ROSA HCP cluster create hard-blocked on the AWS-Marketplace billing link (console-only). IPI needs only AWS creds + a pull secret mintable from the OCM token; metal worker becomes a plain MachineSet.
Implemented the Skupper-over-Submariner choice from the previous day's connectivity survey: the Red Hat Operators skupper-operator (stable-2 channel, "Red Hat Service Interconnect") installed and a Site/Connector/AccessGrant applied live in ns symphony-metrics on sym-aws — router Running, AccessGrant minting real tokens. Since sym-burst is currently torn down, proved the Connector→Listener→backend chain with a scratch namespace standing in for it, confirming (via endpoint inspection, not just a matching response) that traffic genuinely transited the router mesh rather than coincidentally resolving direct. Repeating that proof inside a throwaway UDN-labeled namespace caught a real bug in the plan before it shipped: a Skupper router Pod placed in the grid's UDN namespace is not reachable from a plain-network consumer like KEDA — direct pod-IP curls timed out even though the pod carries a secondary default-network address — the same ingress-island restriction this repo already knew about for VMs, just not previously confirmed for plain Pods. Fixed by giving sym-burst's Skupper Site its own dedicated plain namespace instead of reusing the grid namespace. Re-pointed the KEDA ScaledObject at the new private Listener URL and retired the public metric Route (no live consumer left); wrote the one-time token-redemption handshake script and the Track B rebuild runbook, including exactly how the eventual internal-scheme AWS NLB gets activated (a skupper.io/label-template ConfigMap — there's no CRD field for Service annotations, confirmed by reading the controller source) without being created speculatively while sym-burst doesn't exist. Reconciled with, rather than silently overriding, the parallel session's Submariner proposal.
The shim/server split built for KEDA's metrics-api trigger (Track A3) already had real pending/running/free-slot/done-task numbers flowing from symrest into a re-servable ConfigMap; this closed the loop into OpenShift's own observability stack. Enabled enableUserWorkload: true cluster-wide and added a ServiceMonitor scraping a new, additive metrics ServicePort on symphony-metrics-server (the existing http port and the live ScaledObject are untouched). Extended /metrics with symphony_done_tasks and three symphony_resources_* gauges — all were already sitting unused in the same symrest payload the pusher was already fetching. Verified live, not assumed: UWM's Prometheus/Thanos-Ruler pods Running, a token-authed thanos-querier query returning the exact same live pending-task count as the JSON API, and the scrape target itself reporting up == 1. Documented a KEDA prometheus-trigger as a described option for later, not switched into the live ScaledObject. Zero new billables.
MetalLB is disqualified on AWS by its own docs (emulated ARP breaks L2 mode, BGP has nothing useful to peer with) and by LB semantics (a VIP is DNAT; backends never source from it; a shared VIP can't receive per-host dial-back). The valuable part came from adversarial verification: current OpenShift docs contradicted our July-11 "UDN VMs are ingress-islands" finding, and a live NodePort test against a running master proved Service ingress into primary-UDN VMs now works — the earlier result was stale (the by-design ClusterIP isolation and the no-relay port-security findings still hold). That revives the parked NLB master-exposure design and opens two bounded spikes: moving the WireGuard hub into the cluster as a GitOps-deployed pod behind a UDP NodePort (deleting the EC2 instance and its Terraform), and — courtesy of another refuted assumption — AWS VPC Route Server (GA 2025) as a BGP peer for OVN-K RouteAdvertisements, the first coherent tunnel-less no-NAT fabric candidate on AWS. Neither changes the D14 recommendation; both sharpen it.
Follow-up to the Submariner verdict: a second adversarially-verified research pass evaluated Red Hat Connectivity Link (Kuadrant), OpenShift Service Mesh 3 (including ambient), Skupper/Red Hat Service Interconnect, and Cilium ClusterMesh against the grid's invariants. Every product in the family shares the same disqualifying architecture — a connection-terminating proxy that re-originates traffic from its own address — and none carries the UDP half of EGO's base port or models the master dialing back to a freshly registered host. Cilium ClusterMesh, the one genuinely L3-routed option, would require replacing OVN-Kubernetes and with it the primary-UDN construct the whole VM fleet stands on. The productive output: a ranked service-plane menu for the one thing that should change (the KEDA metric's unauthenticated public Route) — harden-in-place, Submariner ServiceExport if the ACM hub proceeds, or Skupper/RHSI as the lightest VPN-less private path (dial-out mTLS links, no CIDR planning, home-lab-capable). Also disambiguated: Red Hat Connectivity Link has nothing to do with IBM Direct Link, which remains the Phase-5 WAN upgrade.
A 31-agent research pass (with 23 adversarial verifications, 24/27 claims confirmed) answered the "swap WireGuard for Red Hat-supported Submariner?" question definitively: Submariner's OVN-Kubernetes integration attaches only to the default pod network's router, so the primary-UDN networks every Symphony VM lives on are architecturally invisible to it — and moving VMs off UDNs would reintroduce NAT (masquerade), which breaks EGO host identity. Globalnet is NAT (disqualified), and OVN-K's own BGP RouteAdvertisements (which does cover layer2 UDNs in 4.20) is bare-metal-only in practice — AWS VPCs don't BGP-peer with instances. So the proven per-VM WireGuard construction keeps the grid. The productive half of the answer: the home-lab virt cluster is a viable ACM hub (all ACM flows are spoke-initiated, and the hub API already answers from inside sym-aws), unlocking Hive ClusterDeployments — cluster create, destroy, and hibernate as git commits — Crossplane for the break-glass AWS plumbing, and a fully-GitOps Windows image flow: the Tekton disk-uploader task pushes the sysprepped golden image to oci.arsalan.io once, and a git-declared DataImportCron on every cluster keeps its ws2019-symphony DataSource auto-updated. Proposed as D14–D16, awaiting the call.
With the cross-cluster burst proven, the second cluster was destroyed cleanly and the burn is back to the single-cluster baseline (~$5.2/hr). Then an 8-agent research pass on the operator-hybrid question (Track C) reached a decisive, primary-sourced verdict: IBM's archived Symphony operator can't be resurrected on OpenShift 4.20 — its public repo contains no operator at all, and the certified build is credential-walled and missing from the cluster's catalogs — and its container image hardcodes the wrong base port with no override. The hybrid idea (a containerized Linux master with a Windows VM as compute) isn't dead, but it needs a hand-built master, not the operator; a single untested port-remap command now gates whether even that works. Awaiting a go/park call on that bounded test.
The headline milestone: a Windows compute VM on a second, separate OpenShift cluster joins the AWS Symphony grid over a per-VM WireGuard tunnel and runs work at local-host speed, and KEDA autoscales that remote pool 0→2 on real task demand — booting, tunneling, and joining with zero human steps in ~8 min, then draining to zero. Getting there this session cleared seven distinct bugs, each a keeper: a template that sed rewrote into disabling its own feature; a metal-node OVN flow that silently blackholed every in-cluster service IP for UDN pods (and wedged EGO's control plane with it) until an ovnkube restart; a stale-ConfigMap image build that looked like a bake failure; a wg.exe path off by one directory; the SSM advertising the master's unroutable DNS name to remote workers; and an EGO identity-cache race, now closed by a self-healing delayed re-registration so the burst is genuinely hands-off. The whole cross-cluster interconnect — EC2 WireGuard hub, per-VM tunnels, and a reconciler that completes each clone's key handshake through the guest agent — is committed and reproducible. Next: tear down the burst cluster to halve the burn.
ndkh9's mystery closed: it started before the bake-block render landed on installer-repo, so it honestly built a WG-less image — the "source is right" check was made against a later render. Rerun 47fpw carries a FATAL wg.exe assertion, so green = baked. Meanwhile yesterday's session shipped the wg-peer reconciler (remote guest-exec pubkey handshake → hub peer over SSH → masters' hosts sync, GC-guarded), root-caused the sym-aws UDN outage to a broken OVS masquerade flow on the metal node (kapi + CoreDNS ClusterIPs blackholed; mitigations: VPC-resolver DNS + external API endpoint for every UDN-namespace client), found the same incident wedged EGO's vemkd/SD/SYMREST on master1 (grid runs; symping/KEDA wait on an authorized bounce), and sed-proofed the firstboot WG guard the render had been eating.
Stage-2 rebuilt on sym-burst; firstboot's impossible UDN download replaced by a golden-image bake. In-guest forensics (qemu guest-exec) drove the diagnosis. Last mile: get WireGuard to actually persist into the image, then the sym-burst clone joins the sym-aws masters over the tunnel.
The production burst path (replacing demo-KEDA) will be built as an open-source ORB backend — the upstream-leverage route the original research recommended. A Track C experiment brief also landed: containerized Linux master + KubeVirt Windows compute hybrid.
Standing up the cross-cluster interconnect from the redesigned B3 plan (gateway-VM was ruled out — OVN port_security blocks any UDN VM from relaying traffic). New Terraform (infra/interconnect) brings up a t3.small hub via fully declarative cloud-init — installs WireGuard, generates its own keypair, pre-seeds both masters as peers, self-tags its pubkey back via IMDSv2 ec2:CreateTags, no SSH round-trip. Converged and self-tagged ready in ~30s; terraform plan clean. Masters (no subscription/repos) got wireguard-tools via SFTP + rpm -i --nodeps (the dbg-shell/m.py pattern), generated their own keys, and now hold a live handshake with the hub — proven by master1 pinging master2's tunnel IP through the hub (0% loss) at a comfortable auto-negotiated MTU. Backported to the cloud-init templates for future master rebuilds, with the key-durability caveat documented. Also caught and flagged (not fixed): in-cluster CoreDNS silently stopped resolving anything mid-session.
Real Symphony demand (symrest → shim → K8s-API relay → KEDA metrics-api) scaled Windows VMs out in 26 s, the clones joined hands-off and executed the backlog tail, and cooldown scaled everything back to zero — VMs, PVCs, hosts entries all cleaned. The road there surfaced five real bug classes now fixed in git, including a PVC ordinal collision that paused the entire Windows fleet (with the full wedged-VMI recovery runbook: force-delete races, ghost domains, virt-handler restart) and total primary-UDN network isolation, mapped empirically and worked around via the API-as-transport pattern.
The coordinator's own live testing confirmed the isolation was worse than first found: ClusterIP, NodePort, node-IP, and even the apiserver's service-proxy are ALL blocked between a plain-network pod and anything inside the grid's primary-UDN namespace, in both directions — the K8s API and outbound egress are the only channels out. Re-architected around that: symphony-metrics-shim (inside the UDN) now PATCHes real symrest numbers into a ConfigMap every ~10s over the K8s API; a new plain-network app symphony-metrics-server re-serves it. ScaledObject sym-compute-win-burst is now Ready=True, verified reachable from a genuine cross-namespace scratch pod, staying at 0 replicas with pending=0. All 4 Track-A3 Argo apps Synced/Healthy; base pool, masters, and reconciler untouched throughout. Flagged as a decision input for Track B's cross-cluster design too. Not yet exercised: an actual backlog-driven scale-out.
The last mile fell tonight — six stacked root causes (no egoconfig join on Windows / self-referencing MASTERHOST / WMI dead in sysprep-specialize / LIM≠EGO* service name / masters at MTU 8901 vs Windows 1500 (ibm16164145) / master LIM reverse-DNS identity + 24h cache), then the workload layer (.\egoadmin execution user, IBM's mixed-cluster form). Result: clones join hands-off in ~9.5 min (firstboot 12 s), symping runs 48/48 tasks on the NTX64 hosts at 10.8 ms avg, and the pool scales 2→1→2 through pure git pushes — scale-down in 31 s, fresh clone schedulable in 8m42s. qemu-guest-agent guest-exec replaced WinRM as the auth-free debug channel that cracked it open.
PRD (thesis + 7-option comparison matrix) → architecture (as-built D9 topology) → 15 ADRs with pros/cons → 8 component docs → SWOT + sign-off gate → install guide with the ~20-trap build catalog, plus the full Antora Showroom site (39 pages, builds clean with rendered diagrams). Research ran as a 46-agent fan-out with adversarial verification — 9 of 40 web claims corrected before landing. The v0.2 RA remains as the cited research backbone.
WinRM channel built for in-guest forensics; three join blockers root-caused and fixed (ConfigMap staleness, EGO firewall, interactive prompt). Master hit context limit and handed off cleanly with a mandate for Workflow orchestration.
The full gauntlet: cloud-init byte-0, UDN DNS, netstat dep, EGO password policy, no-BOM em-dash, XML double-hyphen, watchdog meta-bugs, the 3-click extract wizard (killed via one-time MSI extraction), Start-Process ExitCode, egoconfig path. Every fix is in git; rebuild is one deterministic pipeline run.
Five real install bugs found and fixed by live debugging overnight — all backported to git. Management plane on the layer2 UDN at stable IPs.
Installer-repo populated (PVC resized after a measured 1.8MB-free near-miss), grid secrets on-cluster, sym-master1 booting, two-stage Windows golden-image pipeline committed with explicit unverified-items list.
Self-hosted 4.20 IPI after the D9 pivot; 7 nodes Ready, console up, GitOps bootstrap underway. First sustained cloud burn begins (~$1.5/hr base cluster).
User cut the knot — no marketplace billing link needed for IPI. Pull secret minted from the existing OCM token, install-config on defaults, 4.20.27 provisioning now. Cluster: sym-aws.sandbox5301.opentlc.com.
Attempt 1: replicas=2 vs 3 subnets (HCP wants a multiple) — fixed. Attempt 2: proved the actual gate is the AWS Marketplace billing link ("Enable ROSA with HCP", console-only). Both runs fully destroyed (63 resources each); config is otherwise apply-clean end to end.
Overnight: Argo → HTTPS+PAT and converged; ~10 GB installers uploaded + checksummed; two RHEL master VMs silent-installed Symphony 7.3.2 — EGO cluster "symgrid" with 9 services. Bonus root-cause: config script silently skipped the WAN/dynamic-host fragment (stock ego.conf false-positive) — fixed in git and live. Golden Windows image build now running.
User call: prove everything on AWS first (per D1), defer IBM key + VNI ticket until Phase 3 nears. Only the ROSA-HCP console toggle stands between here and Phase 1.
Zero-cost init/validate sweep found infra/aws could never init: aws provider pinned to 5.x, rosa-hcp module resolving to 1.7.x demanding ≥6.44. Reconciled pins (aws ~>6.44, modules ~>1.7, auto_repair now required → true); validate green. infra/aws is apply-ready for Phase 1.
CLIs installed (no sudo), AWS creds + quota verified (1152 vCPU — no increase needed), rosa revived an existing OCM token, WS2019 ISO pulled straight from Microsoft. Found ROSA-HCP was never enabled on the account (console-only fix). Remaining reds are IBM-side, all user-gated.
Handoff protocol + model economy in place; execution agents pinned to sonnet.
Deployment plan D1–D8, Terraform (ROSA HCP + metal pool), full Argo tree, Windows imaging workflow, EGO config fragments, secret-safe scripts. Caught AWS credentials before they reached git.
4.5 GB of installers + entitlement (sym_advanced_edition 7.3.2, expires 2026-12-31) from IBM's eval portal — the whole PoC now runs at $0 license cost.
Runnable Community Edition burst PoC (containers + Linux VMs); review pass re-verified VNI routability, MTU, dynamic-host claims; KEDA trade-off documented.
Topology, port matrix, WAN tuning, licensing, burst mechanics, risk register — repo initialized and pushed.
Two adversarial research workflows across Symphony, ROSA/ROKS OCP Virt, Windows licensing, cross-cloud networking. Key negatives: no VM-aware Host Factory provider exists; containerized Symphony is a dead end.