ADR 0004 — IPsec VPN First, Direct Link as the Production Upgrade

Status: Accepted (D3)

Context

The stretched grid needs a private, non-NAT’d path between the AWS-side masters and the IBM Cloud-side burst compute hosts — the same base port, no-NAT, matched-MTU invariants that hold within one cluster (ADR 0001) apply across this link too. Two options are documented:

  • Production path — IBM Direct Link Connect through a partner virtual router (Megaport MCR, Equinix Fabric, PacketFabric, all present at IBM’s Washington DC 2 PoP) to an AWS Direct Connect hosted VIF, with BGP on each leg and a Transit Gateway hub on the IBM side. Higher-bandwidth and jointly documented, but requires a physical colocation cross-connect — real lead time and a procurement step neither side of this project controls unilaterally.

  • PoC/fallback path — a plain IPsec site-to-site VPN: IBM VPC VPN gateway to AWS VGW. IBM’s documented recipe for this pairing is static routing only; route-based mode needs 2 AWS connections / 4 tunnels for ~2 Gbps aggregate on the IBM side, and AWS’s Phase-2 PFS setting forces a custom IBM-side IPsec policy.

The VPN path has real MTU/MSS consequences: effective MTU drops to roughly 1446 on the AWS side with an MSS clamp of 1360 on the IBM side, well below the 1500 this project standardizes on elsewhere. The compute-host join defect chain already demonstrated how unforgivingly a matched-but-wrong MTU breaks LIM (IBM KB ibm16164145) — a VPN-first choice means this project exercises that sensitivity early, rather than discovering it for the first time on the higher-bandwidth production path.

Decision

Stand up the interconnect as an IPsec site-to-site VPN (IBM VPC VPN gateway to AWS VGW, static routing) for Phase 3. Treat Direct Link Connect to Direct Connect, stitched through a partner (Megaport/Equinix), as the defined production upgrade path, not a hypothetical.

Consequences

Pros

  • Hours to stand up and effectively free, versus a partner cross-connect’s procurement lead time and per-port/per-Gbps billing — matches ADR 0002's cost-conscious, stage-before-you-spend posture.

  • Deliberately exercises the grid’s most WAN-sensitive settings early and cheaply: MTU matching, EGO_ENABLE_CHAN_KEEPALIVE / EGO_CHAN_KEEPALIVE_TIME, EGO_DYNAMIC_HOST_WAIT_TIME, EGO_DYNAMIC_HOST_TIMEOUT — surfacing tuning bugs on the cheap path before they’d be harder to isolate on a higher-bandwidth production link.

  • No physical colocation dependency: both VPN endpoints are software-defined, so this path is reachable entirely from Terraform/CLI, consistent with this project’s IaC-first approach (infra/interconnect).

Cons

  • Effective MTU (~1446) and MSS clamp (~1360) are below the 1500 standardized elsewhere in the grid — every VPN-carried Symphony connection runs under a tighter frame budget than the AWS-only or IBM-only legs, and this discrepancy has to be actively managed, not assumed away.

  • IBM’s documented recipe for this AWS/IBM VPN pairing is static routing only — no dynamic route propagation; route-based mode’s higher throughput ceiling comes at real connection/tunnel-count cost.

  • Bandwidth ceiling (single-digit Gbps at best under route-based mode) is well below what Direct Link/Direct Connect can carry, and published latency numbers do not exist for either path — Ashburn-metro adjacency (IBM WDC vs AWS us-east-1) implies low single-digit ms RTT, but that is an inference to be measured in the PoC, not a vendor-published figure.

  • Latency jitter on a VPN path is expected to be higher and less predictable than a dedicated cross-connect — acceptable for a "does the grid join and hold across a WAN" functional PoC (Phase 3 exit criteria: stable join across >=24 h), not representative of production task-turnaround numbers.

Continue to ADR 0005: KEDA demo, Host Factory production.