ADR 0010 — Stretched Single Grid, Multicluster Federation as the Fallback

Status: Accepted (Architecture §1, §8).

Context

Once the burst site exists, Symphony compute hosts on the burst site need to participate in the same grid as the AWS-side masters. Symphony offers two topology models for this:

  • Stretched single cluster — burst compute hosts join the primary masters directly across the interconnect via egoconfig join, so the grid stays one logical EGO/SOAM cluster spanning both sites. This is the topology Architecture §1 recommends as the starting point.

  • Multicluster federation (Advanced-Edition-only) — independent clusters at each site, federated through a small dedicated primary cluster plus one SMCP proxy per member cluster, with session redirection/placement and cross-cluster task dispatch handling the cross-site work instead of a single stretched EGO cluster (Architecture §8).

Why stretch first. egoconfig join keeps single-grid semantics — one scheduler, one view of resources, no session-redirection layer to reason about — which is the simplest thing that could work, and matches how this design already treats the grid within one site (ADR 0008).

The documented caveat that motivates the fallback. IBM and AWS jointly warn that stretch mode puts all daemon traffic over the WAN, explicitly calling this "a source of extra cost or lowered reliability," and position multicluster as the alternative (Architecture §6, closing note before §7). Two concrete costs back that warning up:

  1. Latency is unmeasured and undocumented. No vendor publishes a Symphony daemon-to-daemon WAN latency tolerance. The closest data point IBM publishes is a client-side tuning study at ~28 ms simulated WAN RTT, where task turnaround degraded from ~27-28s to baseline-plus-~35% and only recovered to ~18.5s after enabling data compression and overlapped/async send (Architecture §6) — useful for setting expectations about chatty sessions under RTT, but it is client-path tuning evidence, not a daemon-tolerance spec, and it says nothing about the SD<→SSM<→SIM control-plane traffic a stretched grid puts on the WAN.

  2. Egress is billed on both ends, and can dominate at scale. In the stretched topology, every burst task’s input and output crosses clouds and is billed twice: AWS data-transfer-out on the AWS→IBM return leg, and IBM Direct Link’s metered billing plan charges per-GB egress on the IBM side too. Architecture §9 states the economics plainly: tasks/day x (input+output payload) x egress rate can exceed the compute cost of the burst itself — and that this, not tunnel bandwidth, is often what forces the multicluster topology, because federation keeps data local to each site instead of shipping it across the interconnect on every task.

Decision

Run the PoC and initial production topology as a stretched single Symphony grid (egoconfig join across the interconnect), and measure RTT, jitter, and per-task egress volume during Phase 3 of the install plan (Install phases). Treat Symphony multicluster federation as the documented fallback, to be adopted if the Phase-3 measurements show the stretch topology is fragile on latency or uneconomic on egress — not as a default, and not speculatively ahead of data.

Consequences

Pros

  • Simplest topology that could work: one logical grid, one scheduler view, no session-redirection/placement layer to build or operate on top of what ADR 0008 already assumes within a site.

  • Defers the multicluster management-plane cost (a second, smaller Symphony deployment plus SMCP proxies) until the data says it is actually needed, rather than paying it upfront.

  • The fallback is not speculative — it is a documented, Advanced-Edition-licensed IBM feature (already covered by ADR 0006's licensing decision), not new build scope if it is triggered.

Cons

  • No published Symphony latency tolerance exists to validate against — the only public data point (the ~28ms client-tuning study) measures client-session behavior, not the daemon-to-daemon control traffic a stretched grid actually depends on. The stretch/fallback decision has to be made from Phase-3 measurement, not from vendor guidance, because the vendor guidance doesn’t cover this case (risk register #1, Architecture §11).

  • The fallback is not free: multicluster federation costs a small IBM-side management plane (a dedicated primary cluster plus one SMCP proxy per member cluster) and accepts weaker single-grid semantics — task placement and failover now cross a session-redirection boundary instead of being handled by one scheduler (Architecture §8).

  • Choosing to stay on the stretched topology past the point where egress economics say otherwise is a real risk: the §9 analysis shows egress cost can exceed the burst’s compute cost at scale, so this decision is explicitly time-bounded to "until Phase 3 measurement says otherwise," not a permanent architectural commitment.

Continue to ADR 0011: pre-extracted MSIs.