ADR 0014 — Layer2 UDN Grid Network: No NAT, Same Base Port, MTU Pinned

Status: Accepted (Architecture §4.2/§6/§7; the project’s hard rule 3; MTU root cause build-discovered in commit 5158077, not logged in the phase log).

Context

Every Symphony host — Linux management VMs and Windows compute VMs alike — must see every other host’s real, stable IP with no NAT, because EGO registration is hostname/IP-anchored: a compute host’s LIM registers with the master by address, and masquerade/pod-network NAT is a documented breakage risk. Red Hat’s own docs state the constraint explicitly for ROSA-class clouds: "Connecting VMs directly to the underlay network is not supported on Red Hat OpenShift Service on AWS, Azure, OpenShift Dedicated, Google Cloud, or OCI" — AWS authenticates every MAC against EC2’s own instance records, so KubeVirt-generated MACs can never hold a VPC-routable IP, and Red Hat’s recommended topology on public clouds is a layer2 User Defined Network (UDN) (Architecture §4.2).

This gives all Symphony VMs a shared, NAT-free L2 network — but two build-discovered problems show the invariant is easy to violate even while nominally following the design.

The MTU story. The layer2 UDN’s DHCP hands out the pod-network jumbo MTU (8901 on this AWS underlay) via DHCP option 26. Linux masters honored it and ran at 8901; Windows guests ignore option 26 and stay at 1500 regardless. The mismatch didn’t fail loudly — the compute host still registered with the master, then sat unavail with an empty, silent guest-side LIM log, because the master’s ~3.8KB UDP LIM join payload left as a single jumbo frame that the 1500-MTU Windows NIC silently dropped. This is exactly the failure IBM KB ibm16164145 documents for mismatched MTU. The fix applied: pin masters' eth0 to MTU 1500 (via nmcli persist + runtime ip link set, before the Symphony install) so both sides match at 1500 end-to-end — and explicitly do not set spec.layer2.mtu on the live, immutable UDN object; that field is only settable at UDN re-creation.

The bidirectional-port lesson. The join also depends on the master dialing back to the compute host after the compute host registers — not a one-way, compute-initiates-only conversation. The first join attempt failed because compute hosts had no inbound firewall rule for ports 7869-7875/TCP + 7869/UDP: outbound compute→master 7869 was reachable, but Windows blocked the master’s unsolicited inbound connection by default (all firewall profiles enabled, zero EGO-port rules), so the host silently stayed unavail even though the "obvious" direction of the connection worked fine. This trap, and the MTU one above, were both cracked using the debug channel from ADR 0012 — see Build traps #12 and #13.

Decision

Encode the Symphony network invariants as non-negotiable platform rules, already stated as project hard rule 3 and enforced by this ADR:

  • Same base port (7869) on every host.

  • No NAT between grid hosts — layer2 UDN, real routable IPs.

  • MTU identical across the grid — pinned to 1500 end-to-end (masters explicitly pinned down from the UDN’s jumbo default; Windows guests already default to 1500 and cannot be made to honor DHCP option 26).

  • SSM ports pinned per symphony/config/sd-ssm-ports.xml, not left at OS-random defaults.

  • EGO_DYNAMIC_HOST_WAIT_TIME must be explicitly set, or dynamic hosts silently fail to join.

  • Firewall rules must allow bidirectional establishment on the 7869-7875 range: IBM’s docs do not state which side initiates every channel, and the master dialing back to a newly registered compute host is a real, required direction, not an edge case.

Consequences

Pros

  • A NAT-free, same-MTU, same-base-port layer2 UDN is the platform Red Hat itself documents as the supported topology for VM networking on ROSA-class clouds — this is not a workaround, it is the recommended shape.

  • The MTU fix is durable and cheap once identified (a one-line pin on the master’s interface) and requires no change to the live UDN object, avoiding a disruptive UDN re-creation.

  • The bidirectional-firewall and pinned-SSM-port rules, once encoded as explicit invariants (project hard rule 3), turn a class of symptom that looks like "the host just isn’t joining" into a checklist rather than a fresh multi-hour investigation each time.

Cons

  • Pinning to MTU 1500 leaves jumbo-frame performance on the table — the UDN’s native 8901 MTU is not usable end-to-end while Windows guests are on the grid, so the grid gets no benefit from the pod network’s jumbo capability.

  • UDN MTU is only settable at UDN re-creation, not as a live edit — if a future change to the MTU strategy is needed (e.g. all-Linux compute, or a Windows fix that respects option 26), it requires recreating the UDN object, not a config tweak.

  • The invariants are easy to violate silently: a missing inbound firewall rule, an unpinned SSM port, or an undefined EGO_DYNAMIC_HOST_WAIT_TIME all produce the same symptom (host stuck unavail or never joining) with little differentiating signal, so diagnosis still depends on checking each invariant explicitly rather than any single error message pointing at the cause.

Continue to ADR 0015: VM-level burst.