ADR 0007 — OpenShift GitOps App-of-Apps as the Operating Model

Status: Accepted (D6)

Context

Every cluster in this design — the AWS primary, the future IBM Cloud burst site — runs a mix of operators, VM manifests, and application config that needs to converge the same way every time, on any target. GitOps is operator-native on both platforms: OpenShift GitOps (Argo CD, packaged as a Red Hat operator) installs identically whether the cluster is self-hosted IPI, ROSA, or ROKS.

Precedent this design follows. Red Hat’s own Cloud Experts guide for running VMs on ROSA blesses exactly this shape: manage VirtualMachine manifests via OpenShift GitOps using ApplicationSets with self-healing enabled, over a Kustomize base/overlay repo layout [cloud.redhat.com/experts/rosa/virt-gitops/]. That is the same structural pattern this repo’s gitops/apps/ and gitops/clusters/<cluster>/ split already follows — a shared app definition, overlaid per cluster.

Where this sits relative to Validated Patterns. Red Hat’s Validated Patterns (validatedpatterns.io) are positioned as a more executable evolution of reference architectures — deployable, testable, GitOps-automated artifacts, expressed through graduated maturity tiers (Sandbox / Tested / Maintained). This design borrows the vocabulary — the repo is GitOps-deployable in the same spirit — but is deliberately not claiming to be a Validated Pattern. The Maintained tier’s bar is "use only supported components," and this architecture has exactly one component that fails that bar by design: the Host Factory KubeVirt provider (see ADR 0001 for the sibling keystone decision, and Host Factory KubeVirt provider for the component itself). Calling this a Validated Pattern would overstate its supportability posture; the honest framing is "GitOps-deployable reference architecture with one customer-owned, unsupported-by-Red-Hat extension."

Decision

Adopt OpenShift GitOps (Argo CD) with an app-of-apps pattern, applied once per cluster by scripts/bootstrap-gitops.sh, as the sole operating model for cluster state: operators, HyperConverged, UDN/NADs, and namespaces live in gitops/clusters/<cluster>/; workloads (symphony-mgmt, symphony-compute-win, installer-repo, image-pipeline, keda) live in gitops/apps/, each with its own kustomization.yaml (Kustomize, not Helm). Direct oc apply/oc edit is reserved for bootstrap, debugging, and break-glass only — anything that must persist gets backported to git in the same session. See GitOps for the app-of-apps layout in detail.

Consequences

Pros

  • Operator-native on every target this design runs on (self-hosted IPI today, ROSA/ROKS later) — no cluster-specific install path to maintain.

  • Directly precedented by Red Hat’s own Cloud Experts guidance for VM manifests on ROSA (ApplicationSets + Kustomize base/overlay), not an invented pattern.

  • Reproducibility and auditability: cluster state is defined in git and reconciled by Argo, not accumulated through ad hoc oc sessions — every change has a commit.

  • Borrowing Validated Patterns' maturity vocabulary gives an explicit, citable way to state this repo’s GitOps deployability posture without overclaiming a support tier this design cannot honestly hold.

Cons

  • This is not a Validated Pattern, and the doc says so explicitly: the one custom component (Host Factory KubeVirt provider) would fail "supported components only" at the Maintained tier. Anyone comparing this repo to a Validated Pattern submission should not expect that level of support/testing-cadence commitment.

  • Bootstrap and secrets remain imperative by necessity: scripts/bootstrap-gitops.sh installs the GitOps operator itself (nothing can be GitOps-managed before GitOps exists), and scripts/create-secrets.sh creates OpenShift Secrets from env vars/local files precisely because secrets never enter git. GitOps only covers what comes after that seam.

  • Argo’s repo access needed a build-time fix: SSH access to the private git host was rejected by a source-IP restriction on the git host’s authorized_keys (identical key worked from the devbox but was rejected from every cluster NAT egress IP) — the durable fix was switching Argo’s repo credential from SSH to HTTPS+PAT rather than chasing an IP allowlist. Anyone reusing this pattern against a git host with source-IP restrictions should default to HTTPS+PAT, not assume SSH will work from in-cluster egress IPs.

Continue to ADR 0008: RHEL management VMs.