1 |
Stage-1 DataVolume admission fails: Spec is missing accessMode |
The AWS EBS CSI driver (ebs.csi.aws.com) only auto-infers {ReadWriteOnce, Block}
claim property sets by probing CSI capabilities — never Filesystem, which the
Windows ISO/root disks need. Fix: gitops/clusters/rosa/storageprofile-gp3-csi.yaml
explicitly adds {ReadWriteOnce, Filesystem} alongside the auto-inferred Block
entry (commit 2d5242f, 2026-07-10). |
2 |
Setup fails license-terms resolution: "Windows cannot find the Microsoft Software
License Terms" |
A present-but-empty <ProductKey><Key/></ProductKey> node in autounattend.xml is
treated as a submitted invalid key, not "no key" — it blocks eval-edition
resolution via sources/EI.CFG’s `[Channel]=eval. Fix: removed the empty <Key/>
node entirely (commit 831c98a, 2026-07-10). |
3 |
Stage-1 hangs at the same license/edition dialog even with the key removed, or fails
later at the offline-servicing pass |
A <servicing><package> assemblyIdentity block must exact-version-match the real
install.wim servicing package (17763.3650, not WS2019 RTM 17763.1) — and even
once version-matched, the block does not resolve to a real configurable package on
this media at all: a red herring twice over. Fix: block removed entirely; edition
selection is handled by ImageInstall/OSImage/InstallFrom plus the absent
ProductKey (commit 6ff7643, 2026-07-10). |
4 |
symeval-7.3.2.0.exe -i silent -DVAR=val hangs forever, no error, no log output
|
It is an InstallShield self-extracting bundle of native MSIs, not an
InstallAnywhere installer like its Linux twin — InstallAnywhere-style silent flags
fall through to the full interactive GUI wizard, which then waits on unanswerable
input. Fix: never run the self-extractor in the pipeline. Extract the two
compute-host MSIs once, on a throwaway scratch VM (scripts/extract-symphony-msi.sh),
and serve the pre-extracted .msi files from installer-repo from then on — zero
self-extraction, zero UI automation, in the standing pipeline (2026-07-10, "MSI-carve
pivot EXECUTED"; headless wizard-button automation for the one-time extract itself,
commit b845496). |
5 |
msiexec genuinely succeeds (exit 0, verbose log confirms) but PowerShell reports a
blank .ExitCode
|
Start-Process -PassThru -NoNewWindow reliably fails to populate .ExitCode for
msiexec.exe on WS2019 PowerShell 5.1 — not an exception, not fixed by .Refresh().
Fix: use raw System.Diagnostics.Process/ProcessStartInfo with
UseShellExecute=false instead of Start-Process (phase-log 2026-07-10, "stage-2
attempt 1 root-caused").
|
6 |
A .ps1 fails to parse at all (no output, try/catch never runs), or unattended Setup
hangs with an XML parse error at line 2 column 1 |
Two linked traps from one well-intentioned fix. A BOM-less PowerShell file containing
a UTF-8 em-dash is read via legacy cp1252 on WS2019 PS 5.1 — the em-dash’s UTF-8 bytes
decode to a smart quote, which the tokenizer treats as a string delimiter and cascades
into unterminated-string errors. The blanket em-dash-to--- fix, applied to
.xml.tmpl files too, then hits a second bug: XML 1.0 forbids the literal
two-character sequence -- anywhere inside a <!-- -→ comment body. Fix: one lint
gate covers both — ASCII-only in .ps1 scripts (no em-dashes, no smart quotes) and
no literal -- inside XML comments (phase-log 2026-07-10, "stage-2 attempt 4/5
root-caused"; the ---in-comment bug class recurred and was caught pre-launch a second
time by a local XML parse check, commit c6fee81). |
7 |
egoconfig.exe / install-symphony.ps1 can’t find the binary at
$InstallDir\kernel\bin\
|
Real EGO binaries live under an MSI-property-versioned subdirectory,
$InstallDir\4.0\bin\egoconfig.exe — confirmed by reading the MSI’s own
Directory/File tables with msitools, not by guessing from install logs. Fix:
install-symphony.ps1/firstboot-symphony.ps1 reference the versioned path
(phase-log 2026-07-10, "phase2d join verification"). |
8 |
egoconfig setbaseport 7869 (no -f) hangs the entire specialize pass at "Getting
ready," freezing every later firstboot step
|
It waits on an interactive Y/N confirmation that Tee-Object (piping its output to a
log) swallows — never rendered, never answerable, blocks on stdin forever. Fix:
always call with -f (commit 43a289b, 2026-07-10). |
9 |
Windows egoconfig.exe given a join argument prints a paginated usage screen ending
"Press Enter to continue" and hangs identically to #8 |
Windows egoconfig.exe has no join subcommand at all, unlike the Linux CLI.
Fix: skip egoconfig for the join entirely — directly rewrite
EGO_MASTER_LIST="<master1> <master2>" in kernel\conf\ego.conf on firstboot. Ports
are already correct from the MSI’s BASEPORT=7869 install-time property (commit
5158077, "Windows join root-cause fixes," 2026-07-10). |
10 |
Every *-NetFirewallRule / WMI-CIM firewall cmdlet dies with 0x8004100a during the
sysprep specialize pass |
The WMI/CIM firewall provider is not functional that early in boot. Fix: use
netsh advfirewall firewall add rule instead — works reliably in the same context
(commit 5158077, 2026-07-10). |
11 |
Get-Service -Name 'EGO*' matches nothing (no error); LIM auto-starts on every clone
before firstboot can fix EGO_MASTER_LIST, then exits fatally on the entitlement
check because a host heading its own master list acts as a master
|
The egocore/soamcore MSIs register exactly one Windows service, named LIM, not
any EGO*-prefixed name. Fix: scripts reference the service as LIM; golden image
ships it Manual/inert until firstboot has corrected ego.conf (commit 5158077,
2026-07-10). |
12 |
The compute host registers with the master but sits unavail forever, guest-side LIM
log empty/silent |
Inbound firewall on 7869-7875/TCP + 7869/UDP is missing on the compute host — the
master’s LIM dials back to the compute host after registration, and Windows blocks
the unsolicited inbound connection by default (all firewall profiles enabled, zero EGO
port rules). Fix: firstboot-symphony.ps1 opens inbound EGO ports via netsh before
starting LIM (commit 97ea356, 2026-07-10). |
13 |
Same symptom as #12, but firewall + service name + EGO_MASTER_LIST are all confirmed
correct |
MTU mismatch: the layer2 UDN’s DHCP hands out the pod-network jumbo MTU (8901 on
the AWS underlay) via DHCP option 26; Linux masters honor it, Windows ignores option 26
and stays at 1500. The master’s ~3.8KB UDP LIM join payload leaves as a single jumbo
frame the 1500-MTU Windows NIC silently drops — matches IBM KB ibm16164145
exactly. Fix: pin masters' eth0 to MTU 1500 (nmcli persist + runtime ip link
set) before Symphony install, so both sides match at 1500 end-to-end. Do not set
spec.layer2.mtu on the live immutable UDN object — only if the UDN is recreated from
scratch (commit 5158077, 2026-07-10). |
14 |
WinRM auth=basic fails even after explicitly enabling Basic in
WSMan:...\Auth\Basic |
The WinRM server on this image only ever offers Negotiate/NTLM, regardless of the
Basic config key. Fix: use auth="ntlm" (pypsrp) from the start, not Basic — and
test WinRM from an in-cluster pod on the grid UDN, never the operator’s own shell,
which has no route to the private subnet and produces an identical false-negative
timeout (commits 0c7df15, 6408277, 2026-07-10). |
15 |
A fix is committed to git but every reclone still shows the old bug |
The win-sysprep-join ConfigMap is rendered by create-secrets.sh, not auto-synced
by Argo — it is deliberately outside kustomize per the pipeline’s own
autounattendXMLConfigMapsURL contract (its tasks curl+oc apply/oc delete that
URL every run; a kustomize-managed ConfigMap would race a mid-run pipeline). A code fix
with no re-render is invisible on the next clone. Fix: verify the live ConfigMap
actually changed (oc get cm win-sysprep-join -o yaml | grep <expected-string>)
before every reclone — the stale-ConfigMap trap cost hours more than once (commit
aac0bc7, 2026-07-10). |
16 |
Re-running create-secrets.sh to fix one unrelated ConfigMap silently locks you out
of already-joined masters |
EGO_ADMIN_PASSWORD auto-generates fresh if unset on every run. Fix: always derive
EGO_ADMIN_PASSWORD (and SYM_ADMIN_PASSWORD/WIN_ADMIN_PASSWORD) from the live
Secrets when re-rendering anything after the masters are up, never let the script
regenerate them (install-guide.md §5; phase-log 2026-07-09 "Phase 2 secrets
bootstrap").
|
17 |
apply-config.sh’s idempotency guard reports the WAN-tuning fragment already applied,
but `EGO_ENABLE_CHAN_KEEPALIVE etc. are missing from the live ego.conf
|
The guard checked grep -q EGO_DYNAMIC_HOST_WAIT_TIME ego.conf — but stock
Symphony 7.3.2 ego.conf already ships that key, defined at 60, so the guard
false-positives on a fresh install and the fragment is silently never appended. Fix:
guard on a fragment-unique key (EGO_ENABLE_CHAN_KEEPALIVE) instead, and explicitly
sed-delete the stock EGO_DYNAMIC_HOST_WAIT_TIME=60 line before appending, to avoid
a shadowing duplicate (phase-log 2026-07-10, "EGO master verified alive; WAN-fragment
drift found and fixed"). |
18 |
qemu-guest-agent reports AgentConnected=True and the join is assumed done
|
Guest-agent connectivity only proves the VM booted and the agent is running — it is
not evidence the EGO join succeeded. Fix: always confirm the join independently
via egosh resource list, never infer it from guest-agent state (install-guide.md §9
item 23; the distinction that shaped every join-verification pass in phase-log,
2026-07-10). |
19 |
Cancelling a stuck stage-2 PipelineRun orphans the customize VM / ConfigMaps every
time |
Neither oc delete pipelinerun nor status=Cancelled runs the pipeline’s
finally-block cleanup tasks in this cluster — only status=CancelledRunFinally
does. Fix: set status=CancelledRunFinally to cancel a stage-2 run cleanly;
otherwise force-cleanup (oc delete vm, force-delete the virt-launcher pod, delete the
DataVolume/PVC) by hand (phase-log 2026-07-10, "MSI-carve pivot EXECUTED"). |
20 |
(Design discipline, not yet hit as a bug) A tempting "shut down on failure" fast-fail
in the customize script |
The pipeline’s wait-for-status task only checks status.phase, which can’t
distinguish "shut down because sysprep finished cleanly" from "shut down because the
script gave up mid-install" — a failure-triggered shutdown would let a half-installed
VM publish itself as the golden DataSource (silent corruption). Fix: never add
shutdown-on-failure to a customize script. Leaving it running until the task timeout
fails the pipeline closed is slower but safe (install-guide.md §9 item 25). |