Interconnect

The private, non-NAT’d network path between the AWS-side Symphony masters (sym-aws) and the IBM Cloud-side burst compute hosts (sym-roks) that lets the grid stay a single stretched cluster instead of two independent ones. Everything else this project builds on the IBM side — VNI-attached Windows VMs, the custom Host Factory KubeVirt provider, the direct EGO_MASTER_LIST join — is reachable only if this component holds: matched MTU, bounded RTT, DNS that resolves both ways, and every port in the Symphony matrix open bidirectionally between the two sites. It also carries the project’s most expensive open question — cross-cloud egress economics — because every burst task’s input and output pays this link twice.

Purpose

CREATE. Nothing under infra/interconnect/ exists yet — the explicit decision to defer IBM-side work ("ignore IBM for now…​ AWS first") was recorded 2026-07-09; infra/ibm and infra/interconnect have no Terraform. This is Phase 3, gated behind two prerequisites owed by the user since Phase 0 and still outstanding: an IBM Cloud API key and the VNI-allowlist ticket for OVN localnet UDN on ROKS — the longest lead-time item in the whole plan, requested at project start specifically so it isn’t the Phase-3 bottleneck. Until both land, this component is designed but not buildable.

Two paths are designed (ADR 0004):

  • PoC path (Phase 3, build first): IPsec site-to-site VPN, IBM VPC VPN gateway to AWS VGW, static routing.

  • Production path (Phase 5 upgrade): IBM Direct Link Connect to a partner virtual router to an AWS Direct Connect hosted VIF, BGP both legs, Transit Gateway hub on the IBM side.

Placement

AWS (sym-aws, us-east-1, 10.0.0.0/16)              IBM Cloud (sym-roks, us-east/WDC, 10.64.0.0/16)
┌──────────────────────────────┐                   ┌──────────────────────────────┐
│ Symphony masters (RHEL VMs)  │                   │ Windows compute VMs          │
│ on layer2 UDN                │                   │ VNI-attached, localnet UDN   │
└──────────────┬────────────────┘                   └──────────────┬────────────────┘
               │                                                    │
        AWS VGW / Direct Connect                     IBM VPC VPN gateway / Direct Link
        hosted VIF                                   Connect + Transit Gateway
               │                                                    │
               └──────── PoC: IPsec site-to-site VPN ───────────────┘
                    (static routing, ~1446 MTU AWS / MSS clamp 1360 IBM)
               └──── Prod: Direct Link ⇄ partner MCR/Fabric ⇄ Direct Connect ────┘
                    (BGP both legs, Washington DC 2 PoP / Equinix DC2 Ashburn)

        DNS: Route53 Resolver endpoints ⇄ IBM Cloud DNS Custom Resolver
             + $EGO_CONFDIR/hosts on every Symphony host (belt-and-braces)

CIDRs are disjoint by design, not just by convention: AWS 10.0.0.0/16 (sym-aws), IBM Cloud 10.64.0.0/16 (sym-roks) — required by Symphony’s no-NAT invariant. EGO host registration is IP-anchored, so a NAT’d or overlapping-prefix path silently breaks host identity rather than failing loudly. Terraform home: infra/interconnect/ (one state directory per convention), plus DNS-forwarding resources in infra/aws (Route53 Resolver endpoints) and infra/ibm (Cloud DNS Custom Resolver). No GitOps placement — this is below the cluster, provisioned once via Terraform/CLI like the clusters it connects.

No NAT between grid hosts, ever is the load-bearing invariant of the whole design — both the VPN and Direct Link paths preserve real, routable IPs on both sides. The VPN PSK never enters git, created by scripts/create-secrets.sh from an environment variable.

Interfaces

Symphony port-matrix rows that must traverse the interconnect bidirectionally between sym-aws (management hosts) and sym-roks (burst compute hosts). IBM does not document channel initiation direction, so treat every row as bidirectional at the firewall/security-group layer regardless of which side conventionally opens the connection.

Peer Direction Port / Protocol Purpose

sym-roks compute LIM <→ sym-aws master LIM

bidirectional

7869 TCP+UDP

EGO_LIM_PORT — base port; must be identical on every host, both sites

sym-roks compute → sym-aws master vemkd

outbound from compute (client-first)

7870 TCP

EGO_KD_PORT — first connection a joining/dispatching host makes

sym-aws master vemkd <→ sym-roks compute PEM

bidirectional, long-lived idle

7871 TCP

EGO_PEM_PORT — carries the EGO_ENABLE_CHAN_KEEPALIVE-protected channel

both sites

bidirectional

7872 TCP

egosc (ESC_PORT)

both sites

bidirectional

7873 TCP

Repository service — service-package distribution to burst hosts

sym-roks client/compute <→ sym-aws SD

bidirectional

7874/7875 TCP

Session director (SD_ADMIN_PORT/SD_SDK_PORT)

sym-aws SSM <→ sym-roks SIM

bidirectional, long-lived idle

pinned range, e.g. 31000-31039

SSM — must be pinned in sd.xml per symphony/config/sd-ssm-ports.xml, since OS-random defaults are unusable across a firewalled WAN link

IBM VPC VPN gateway <→ AWS VGW

bidirectional

UDP 500/4500, ESP

IPsec site-to-site tunnel (PoC path) — carries all rows above

IBM Direct Link Connect <→ AWS Direct Connect hosted VIF

bidirectional

BGP (TCP 179) + data plane

Production path — carries all rows above once cut over

Route53 Resolver endpoint <→ IBM Cloud DNS Custom Resolver

bidirectional

53 TCP/UDP

Cross-cloud name resolution — secondary to $EGO_CONFDIR/hosts

Maturity

  • PLANNED for Phase 3; currently PARKED. The explicit decision to work AWS-only first was recorded 2026-07-09; infra/ibm and infra/interconnect have no Terraform yet. This component cannot proceed until the user supplies two things owed since Phase 0: an IBM Cloud API key and the VNI-allowlist ticket for OVN localnet UDN on ROKS — flagged as the longest lead-time item in the entire plan specifically so it doesn’t block Phase 3 later.

  • MTU must match exactly, not just be "close." IBM KB ibm16164145 documents the precise failure signature: a compute host sticks at unavail, its lim starts then hangs and stops logging, and pem/melim never start. This project already hit the same failure class within a single site before ever reaching the cross-cloud link, so the interconnect must not be treated as "close enough": the PoC (VPN) path runs an effective MTU ~1446 on the AWS side, MSS clamp 1360 on the IBM side — below the 1500 standardized everywhere else in the grid, and must be actively managed. The production (Direct Link) path plans 1500 end-to-end; jumbo frames are documented only for Dedicated Hosting (9214), and partner CSP ports commonly cap at 1500.

  • Latency is inferred, not measured. No vendor publishes a latency figure for either path. The working assumption — low single-digit-ms RTT between AWS us-east-1 and IBM us-east — rests on metro adjacency and is explicitly an inference to be validated in the PoC, not a documented number. No published Symphony RTT tolerance exists either — the closest IBM publishes is a client/session tuning study at ~28ms simulated WAN RTT, which is client-path tuning evidence, not a daemon-tolerance spec.

  • VNI cross-Transit-Gateway routability is inferred, not documented by name. IBM documents VNI-attached VMs getting "direct VPC connectivity" but never states whether that reachability extends across a Transit Gateway to a Direct Link-connected external network. The Phase-3 manual burst proof must validate this explicitly, including security-group rules for the port matrix above, before any provider automation is built on top of it.

  • Egress economics is a security-adjacent cost risk, not just a billing line. Every burst task’s input and output crosses clouds and is billed twice: AWS data-transfer-out on the return leg, and IBM Direct Link’s metered plan charges per-GB egress. At grid scale, tasks/day × (input+output payload) × egress rate can exceed the cost of the burst compute itself — this is the concrete trigger for falling back to Symphony multicluster federation, which keeps data local to each site instead of stretching one grid over the WAN. The metered-vs-unmetered Direct Link plan choice is deferred to measured Phase 2/3 volumes.

  • DNS is belt-and-braces, not a single point of failure. Route53 Resolver endpoints forward to IBM Cloud DNS Custom Resolver — a synthesized pattern, not jointly documented by AWS and IBM — layered with static $EGO_CONFDIR/hosts entries, IBM’s own documented pattern for removing DNS from the grid’s critical path.

Verify

# 1. MTU probe both directions -- must not fragment at the path's true MTU.
#    VPN path: expect failure above ~1446 (AWS side) / with MSS>1360 (IBM side) unless
#    the interconnect-specific MTU is deliberately configured on Symphony's NICs.
#    Direct Link path: expect success up to 1500, failure above it (no jumbo assumed).
ping -M do -s 1472 <peer-ip>   # 1472 + 28 bytes ICMP/IP header = 1500 total
ping -M do -s 1418 <peer-ip>   # 1418 + 28 = 1446, the VPN-path effective MTU

# 2. RTT / jitter measurement, both directions, sustained (not a single ping).
#    Feeds the EGO_CHAN_KEEPALIVE_TIME and EGO_DYNAMIC_HOST_WAIT_TIME tuning decision.
mtr --report --report-cycles 100 <peer-ip>

# 3. 24-hour keepalive soak -- Phase 3 exit criterion: "stable join across >=24h
#    (keepalives hold), task turnaround measured at real RTT."
egosh resource list -l | grep -i unavail    # run periodically over the soak window

# 4. Channel-direction validation -- IBM does not document initiation direction, so
#    confirm empirically which side opens vemkd<->PEM and SSM<->SIM.
netstat -tnp | grep -E '7869|7870|7871|3100[0-9]|3103[0-9]'   # adjust to the pinned SSM range

# 5. Confirm the pinned SSM range is actually in effect (not OS-random defaults).
grep -A2 'SSM_SDK_ADDR\|SSM_SIM_ADDR' $EGO_CONFDIR/../soam/*/configdir/sd.xml

# 6. Confirm $EGO_CONFDIR/hosts belt-and-braces entries exist and resolve correctly
#    independent of the DNS resolver chain.
cat $EGO_CONFDIR/hosts

Open items for whoever picks this up next: IBM Cloud API key and VNI-allowlist ticket are still owed by the user; no RTT/jitter numbers exist yet for AWS us-east-1 to IBM us-east; VPN gateway and Direct Link partner router idle-timeout values are unmeasured; Direct Link metered-vs-unmetered plan choice is deferred to Phase 2/3 volume data.

Return to Components for the full map, or continue to Install phases for the build sequence.