ADR 0005 — KEDA for the Burst Demo, Host Factory KubeVirt Provider for Production

Status: Accepted (D4)

Context

Symphony’s native burst mechanism is Host Factory (Advanced Edition): a scheduler-driven, 5-script provider interface (getAvailableTemplates, requestMachines, requestReturnMachines, getRequestStatus, getReturnRequests) that Symphony’s own symA requestor drives from real grid demand. No built-in Host Factory provider targets KubeVirt, so a custom provider is required for production burst regardless of what drives the demo.

KEDA (packaged on OpenShift as the Custom Metrics Autoscaler) was evaluated as a faster-to-build alternative: a KubeVirt VirtualMachinePool exposes the standard Kubernetes /scale subresource, and ScaledObject targets it generically with no KubeVirt-specific integration required — a working public example (Prometheus/RabbitMQ triggers scaling a VirtualMachinePool) confirms this mechanically works. VirtualMachinePool itself is Red Hat/KubeVirt-native fleet tooling for keeping N VM replicas Ready — not custom code, and this project’s KEDA demo (KEDA metrics shim) uses exactly that generic mechanism.

It was rejected as the production burst mechanism for three reasons:

  1. Scale-in is not grid-aware. KEDA/HPA-style scale-in deletes VMs without draining in-flight work, stranding running tasks for rescheduling and leaving ghost unavail hosts in EGO, versus Host Factory’s requestReturnMachines, which performs a negotiated, graceful return. Even KubeVirt 1.6’s newer VirtualMachinePool.spec.scaleInStrategy.proactive.selectionPolicy is ordinal / creation-time ordering only, not workload-state or idle/drain-aware — it can express "kill the newest VM first," never "kill the VM that’s actually idle."

  2. The demand signal is a custom build either way. No Symphony Prometheus exporter exists, so a KEDA path needs a new SYMREST poller (GET /platform/rest/symrest/v1/sessions → pendTasks) — engineering effort of the same order as wiring the stock symA requestor, which already understands Symphony demand natively.

  3. Two autoscalers on one pool is a control-loop conflict — running KEDA and a future Host Factory provider against the same VirtualMachinePool simultaneously would fight over the same scale target.

The Custom Metrics Autoscaler Operator reached GA at operator version 2.10.1 in the OCP 4.14 era, having run as Technology Preview through OCP 4.11-4.13 — a comparatively young operator lineage, distinct from whether its scaling semantics fit this workload at all.

Decision

Use KEDA scaling a VirtualMachinePool for the Phase 4 burst demo only (KEDA metrics shim): it demonstrates VM-level elasticity end-to-end with minimal new code, on the platform’s own generic /scale mechanics. Build the custom Host Factory KubeVirt provider — the one intentionally custom component in this design — as the production burst path, patterning on the FINOS Open Resource Broker (finos/open-resource-broker, ex Morgan Stanley + AWS, Apache-2.0) and Google’s symphony-gcp, both of which implement the identical 5-script Host Factory contract against Kubernetes/AWS/GCE targets. Details in Host Factory KubeVirt provider.

Consequences

Pros

  • The demo proves VM-level elasticity (the reference architecture’s core claim) using entirely off-the-shelf mechanics — VirtualMachinePool /scale + upstream KEDA — with no bespoke scale-in logic to build or maintain for the demo itself.

  • The production provider follows a documented, standard extension interface (IBM’s 5-script Host Factory contract) with two independent, real-world implementations already proving the pattern against different substrates — this is not a from-scratch interface design.

  • Keeping KEDA scoped to the demo avoids ever running two autoscalers against the same pool in production — no control-loop conflict to design around.

  • The demo’s honest gap (scale-in kills VMs mid-task) is documented and observed on purpose rather than silently accepted as a production design.

Cons

  • The demo shows a real gap on purpose: KEDA scale-in has no task-drain awareness, so demo scale-in events can strand in-flight work — acceptable only because it is explicitly a demo, not the production burst path.

  • The custom Host Factory KubeVirt provider is a genuinely multi-person-month build, not a weekend script — the FINOS ORB and google/symphony-gcp analogs establish both that the pattern works and the effort scale it takes (external kubeconfig auth, VM lifecycle-to-Host-Factory-state mapping, sysprep injection, lazy-return tuning).

  • Until the custom provider exists, there is no grid-aware production burst mechanism at all — the demo is a capability preview, not a stopgap that could be promoted to production as-is.

  • Two working code paths (demo KEDA/shim, future Host Factory provider) both have to be understood by operators, at least until the provider fully supersedes the demo.

Continue to ADR 0006: Eval, then BYOL.