Deep dive · engineers

The first product (TFA), one release end to end

State as of 28 October 2026dev landscapeFollows illustrative version 202610.22.093015 from git push to a ready HelmRelease

TFA is a composition of components (helm, adf, databricks) delivered at one version by two independent pipelines. This page follows one release through every artifact, file and controller it touches, then hands you the runbook, the decisions behind it and the traps to avoid.

Deployables ppdevops.azurecr.io Compositions ghcr.io/powerplandev/tfa-composition Versions CalVer YYYYMM.D.HHMMSS Cluster pp-aks-cloud-eastus-main3 (dev-v1) Namespace {environment}-{product}-{landscape} dev/app01 delivery: flux · track: candidate

Summary

TL;DR in six lines

  1. Service repos and powerplan-tfa-adf build artifacts at CalVer and push them to ACR.
  2. CD · Development (push to main, or started by CD · Detect every 30 min) runs cd-detect: newest builds and product files vs the latest approved composition.
  3. cd-compose writes one product version: adf/databricks aliases, composition (ghcr), product chart last.
  4. Tracking environments' Flux pulls the -candidate chart; CD waits for HelmRelease Ready on app01.
  5. cd-approve runs automatically when every dev deploy is green: -approved everywhere and the Release pointer, the next detect baseline.
  6. qa and prod move only by CD · Promote (cd-promote): a pin commit in cloud-config, applied by environments.yml → env-apply. CD · Release gives an approved version a release number; its qa deploy is planned. Today there is no qa or prod: only dev/app01 exists, so Promote and Release stop at their first step (plan) until greenfield IaC creates those environments.
trace · tfa 202610.22.093015 (illustrative)

01Four invariants to keep in your head

Everything else on this page is a consequence of these. If a change you are about to make breaks one, stop and read the decision log.

INV-1

One product, one version

Every component carries <v> (CalVer YYYYMM.D.HHMMSS). The state is part of the version: <v>-candidate, <v>-approved. A bare CalVer pin is rejected.

INV-2

The composition decides

What a version contains is the composition, not "whatever is newest". Detect compares against the latest approved one; every deploy pulls it first and checks the chart digest built.

INV-3

Versions move two ways only

Lower environments: Flux tracking -candidate. qa and prod: a pin commit in cloud-config made by cd-promote. Nothing else changes a version. A release number (CD · Release) is a third label on the same approved bytes, not a new version.

INV-4

Values layer; secrets don't

Later layer wins at the same path; identity is last and fixed. Secrets are names in values and values only in Key Vault. Bad input fails the render, not the pod.

02Two pipelines, one hand-off

Each pipeline runs on its own. They meet at one point: a promotion goes from the product pipeline (cd-promote) as a pin commit into cloud-config, which the environment pipeline applies.

Product pipeline product-deploy-tfa

Owns
Versions: the composition of every component, <v>-candidate → <v>-approved
Runs when
Five workflows: CD · Development on a push to main, a PR (dry run), "Run workflow" or a start by CD · Detect (every 30 min, only when a component build or a product.yaml/configurations/ change moved); CD · Release, CD · Promote by hand; CD · Retention weekly built
Changes
ACR, GitHub Packages and GitHub Releases; cloud-config only through a promotion commit (GitHub App)
Reaches environments
Tracking envs (track: candidate) pull each candidate through Flux; app01 is watched until Ready, then the run approves it automatically (no reviewer)
qa / prod→cd-promoteapproved version
pin commit, gated by tfa-<landscape>-release

Environment pipeline cloud-config

Owns
Environments: environment.yaml, landscape values, per-environment values, pins
Runs when
A landscapes/** push (including a promotion commit), a pull request (render only), or "Run workflow" for one environment: environments.yml → env-apply
Changes
The cluster: env chart env-<environment>-<landscape>, then every enabled product (or the ones named)
Use it for
A new environment, value overrides for an existing one, a production version pin
Product pipelineservice repos + product-deploy-tfa + cloud-cicd-templates
Environment pipelinecloud-config · environments.yml → env-apply · runs on its own

Click a stage, then use ← / → to move through that pipeline.

03The pieces

The repositories (GitHub organisation in brackets), what each one does, and the files to open first. Which branch each one uses is in the repository README and the platform knowledge base; it changes often.

PieceRepo [org]Role
Platform charthelm-platform-service [PowerPlanDev]
2.4.1 released · 2.5.0 to publish
platform-services. Archetypes app (Deployment, Service, Ingress/HTTPRoute, HPA or KEDA, PDB, SA, ExternalSecret) and task (Job, hooks; CronJob with schedule), plus the libraries global and keda. KEDA and cron arrive in 2.5.0 to publish. Published at oci://ppdevops.azurecr.io/helm/platform/platform-services.
Orchestratororchestrator-component-release [PowerPlanDev] Python CLI: generate-service, generate-product-chart (--version <v>-candidate), write-composition, approve-composition, pipeline-plan, render-environment (--flux-dir, --composition; A-12 allowed list, resolved.json hash), adf-package, databricks-package, export-schema built. CI installs it from git with ORCHESTRATOR_TOKEN.
Shared CI/CDcloud-cicd-templates [PowerPlanDev] Golden paths gp-dotnet-service, gp-python, gp-adf, gp-databricks-bundle, helm-chart (and gp-nx-monorepo, not yet merged); product CD cd-plan, cd-candidate, cd-detect, cd-compose, cd-deploy-environment, cd-approve, cd-release, cd-promote, cd-retention; environments env-apply; actions version, service-chart built.
Service repospowerplan-tax-fixed-assets, powerplan-client (chart in deploy/tfa/), powerplan-active-reports-service, powerplan-content, powerplan-databricks-zerobus Each has a deploy/service-chart.yaml. CI builds the images, then the service chart.
ADF and Databricks repospowerplan-tfa-adf, powerplan-spotlight-db, powerplan-spotlight-api [PowerPlanDev] gp-adf: validate + export, adf-package, oras push powerplan/tfa-adf-artifact:<CalVer>. gp-databricks-bundle: databricks-package, oras push databricks/<bundle>:<CalVer> built (PRs: package only).
Product repoproduct-deploy-tfa [PowerPlanDev] product.yaml (kind: Product; components.helm chart + serviceCharts, components.adf, components.databricks spotlight-db + spotlight-api), configurations/{helm,adf,databricks}.yaml (the global.secrets pool is in helm.yaml), pipeline.config.yaml (the ladder; dev/app01 only today). Five thin workflows, one per process: cd-detect.yml, cd-development.yml, cd-release.yml, cd-promote.yml, cd-retention.yml (until 2026-10-26: one cd.yaml → cd-router).
Environmentscloud-config [PowerPlanDev] The landscapes/ tree. Workflow environments.yml → env-apply (the 2026-10-19 consolidation removed deploy-changed, deploy-environment, promote-product, bump-environment-pin).
Environment charthelm-platform-environments [PowerPlanSpikes]
0.4.1 released · 0.5.0 to publish
platform-environment: namespaces, ServiceAccount shared-pp-workload, the Key Vault SecretStore, and (under Flux) the OCIRepository, HelmRelease and values ConfigMaps for each product; 0.5.0 adds the config-hash annotations and the environment-level identity to publish. Published at oci://ppdevops.azurecr.io/helm/platform/platform-environment.

Service charts in TFA today (dev)

tax-fixed-assetstax-fixed-assets-api, tfa-migrations, redis
tfa-frontendtfa-client, flatfile-listeners, compodoc
active-reports-serviceactive-reports-service
content-deploycontent-deploy
tfa-zerobuszerobus-worker, zerobus-dispatcher (off by default; needs 2.5.0)

Artifacts and registries

  • ACR: images and helm/services/<chart> at build CalVer
  • ACR helm/products/tfa-app: <v>-candidate / <v>-approved as the chart's own version built
  • ACR powerplan/tfa-adf-artifact: build CalVer; <v>-candidate/-approved aliases built
  • ACR databricks/powerplan-spotlight-db, databricks/powerplan-spotlight-api: same scheme built
  • GitHub Packages ghcr.io/powerplandev/tfa-composition: <v>-candidate, <v>-approved built

Cluster add-ons: Flux Operator (source and helm controllers), External Secrets Operator, KEDA (AKS add-on 2.17), registry pull Secret acr-pull-secret. Key Vault access as the environment's identity through shared-pp-workload (or Secret keyvault-sp with authType: ServicePrincipal).

04Model and composition

A product is a composition of components with one version and one deployment per environment. helm is the product chart tfa-app (service charts, each holding services); adf the Data Factory ARM package; databricks a named map of Asset Bundles (TFA: spotlight-db, then spotlight-api), deployed in the order listed. Compose composes declared components only.

# ideas/product/composition.example.yaml (abridged)
apiVersion: v1
kind: Composition
product: tfa
version: 202610.22.093015
state: candidate            # approved after the deploy ladder
source: {repository: PowerPlanDev/product-deploy-tfa, commit: …}
components:
  helm:
    chart: helm/products/tfa-app
    version: 202610.22.093015-candidate
    packageDigest: sha256:…  # checked at every deploy
    serviceCharts: {tax-fixed-assets: 202610.21.141502, tfa-frontend: 202610.22.081144, …}
  adf:
    artifact: powerplan/tfa-adf-artifact
    version: 202610.22.093015-candidate
    digest: sha256:…
    built: 202610.18.110230
  databricks:
    spotlight-db:  {artifact: databricks/powerplan-spotlight-db,  version: …-candidate, built: …}
    spotlight-api: {artifact: databricks/powerplan-spotlight-api, version: …-candidate, built: …}
approval: {}                # approvedAt, approvedBy, run

Source of truth of every deploy

  • Models, write-composition / approve-composition and publishing built
  • Compose pushes tfa-composition:<v>-candidate to ghcr; every deploy pulls it first, checks the chart's packageDigest and deploys exactly what it names
  • Approval pushes :<v>-approved; the GitHub Release notes carry its address and digest (a pointer, no copy)
  • Detect follows the latest approved release to its composition

Access

  • Package linked to product-deploy-tfa (org.opencontainers.image.source), inherits its access
  • Compose/approve: GITHUB_TOKEN with packages: write; cloud-config gets Actions read on the package
  • ACR holds only deployables

Decision D-03: Flux tracking stays

  • Flux semver tracking stays: a tracking env's helm follows the newest -candidate chart; adf/databricks move when a deploy runs
  • Skew kept small: publish order adf/databricks → composition → chart last; product CD deploys the candidate's other components right after compose
  • Every deploy resolves adf/databricks from the composition of the chart version the HelmRelease is at

05Values layering: lowest first, later wins

helm component shown. The first three layers are baked into the product chart. Under Flux, the last three are ConfigMaps owned by the env chart release and listed in that order in the HelmRelease's valuesFrom.

1
Chart defaultsplatform-services values.yaml
platform
2
Service chartdeploy/service-chart.yaml: appSettings, env, envGroups, secrets, "$required", $url:
service team
3
Productproduct-deploy-tfa/configurations/helm.yaml: global (prefixes, secret pool) + one key per service chart
product ops
10
10-landscapelandscapes/<landscape>/environments.yaml → products.tfa
platform ops
20
20-overrides (optional)landscapes/<landscape>/<env>/tfa/helm.yaml
platform ops
30
30-identitycomputed: product, environment, landscape, tier, imageRegistry, workloadIdentityClientId. Cannot be overridden.
orchestrator
baked into tfa-appFlux ConfigMap tfa-values-*↑ later wins

Secrets never travel in values

  • The product pool (configurations/helm.yaml secrets) names Key Vault secrets.
  • Each service maps an env var to a pool name.
  • The chart renders one ExternalSecret per service; ESO syncs it through the SecretStore keyvault, read as the environment's identity (shared-pp-workload).

Merge rules

  • Helm merges maps and replaces lists.
  • Landscape and environment layers may set only the A-12 allowed list (env, appSettings, sizing, enabled, imagePullSecrets, 25 global keys); anything else fails the render.
  • The rendered values hold only the landscape and environment deltas, never the product tier.
  • Per-service keys nest by service chart, then service, then archetype: services.tax-fixed-assets.tax-fixed-assets-api.app…

Per component

Tier (later wins)helmadfdatabricks
chart defaults / service chartplatform-services values, service-chart.yaml––
productconfigurations/helm.yaml (baked into the chart)configurations/adf.yaml (in the composition)configurations/databricks.yaml
landscapeenvironments.yaml products.tfa––
environment<env>/tfa/helm.yaml (optional; ConfigMap 20-overrides)<env>/tfa/adf.yaml<env>/tfa/databricks.yaml
identitycomputed, cannot be overridden (ConfigMap 30-identity)––

Missing or empty file = {}; a file for an undeclared component fails. render-environment --composition merges adf and databricks (per bundle) built; the ADF deploy becomes a real az deployment group create into the environment's factory resource group and the bundle deploy a real databricks bundle deploy planned (both print their commands today).

06Where did this value come from?

Six real keys from TFA on dev/app01, traced hop by hop through the files that set them. Pick one. (render-environment writes the same answer to layers.json, and the content hash to resolved.json.)

07environment.yaml and Flux

With delivery: flux, the env chart (templates/flux.yaml) renders three things per product in its namespace. CI applies nothing to the product; it asks Flux to reconcile and waits.

# cloud-config/landscapes/dev/app01/environment.yaml
environment: app01
landscape: dev
stack: dev-v1
delivery: flux            # helm (default) or flux
workloadIdentityClientId: …  # the environment's identity id-<env>-<landscape>
products:
  tfa:
    enabled: true
    track: candidate      # or version: <CalVer>-approved (a pin)
keyVault:
  name: kv-app01-prod      # override until kv-app01-dev exists
  authType: WorkloadIdentity
  tenantId: …

OCIRepository tfa-chart

oci://ppdevops.azurecr.io/helm/products/tfa-app, secretRef: acr-pull-secret.
track → semver: ">=202610.0.0-0" + semverFilter: '.*-candidate$'; pin → tag.

HelmRelease tfa

releaseName: tfa (helm-installed releases are adopted, because app.kubernetes.io/instance is in the immutable selectors). chartRef → the OCIRepository; install and upgrade remediation retries 3.

ConfigMaps tfa-values-*

10-landscape, 20-overrides (optional), 30-identity; labelled reconcile.fluxcd.io/watch: Enabled, so a changed layer re-reconciles the release. With env chart 0.5.0 to publish they and the HelmRelease carry powerplan.com/config-hash and powerplan.com/config-commit.

08When it's stuck: a runbook

Symptoms you will actually see, where to look, and the usual cause. Commands are read-only unless flagged. Namespace for TFA on app01: app01-tfa-dev.

Known problems right now (environment pipeline failing, promotion not live): where things stand and environment status.
HelmRelease not Readyartifact revision … does not match chart version …
Lookkubectl describe helmrelease tfa -n app01-tfa-dev
CauseA tag that is an alias of another chart version (an old bare-CalVer chart); helm-controller needs tag = chart version. Current composes build the chart as <v>-candidate and approval repackages <v>-approved.
FixPin or track a version made by cd-compose/cd-approve; never re-tag a chart by hand.
Deploy stops: digest mismatchpackageDigest
LookThe cd-deploy-environment log, composition step.
CauseThe chart pulled at <v> is not the package the composition recorded (re-pushed or overwritten tag), or no composition exists for that version.
FixDeploy a version that cd-compose produced; investigate who pushed the tag.
Flux doesn't see the new candidateflux get sources oci
LookThe OCIRepository revision and its ref.
CauseTracking uses semver: ">=202610.0.0-0" + semverFilter: '.*-candidate$'; a pinned environment uses tag and never moves on its own.
FixCheck environment.yaml has track: candidate; CD requests a reconcile instead of waiting for the interval.
Pod CreateContainerConfigError
Lookkubectl get externalsecrets,secretstores -n app01-tfa-dev
CauseFor a few seconds: ESO hasn't synced yet (expected, no hooks). If it stays: a pool name missing in the vault, the identity lacks Key Vault Secrets User, or its federated credential does not match system:serviceaccount:<namespace>:shared-pp-workload on this cluster's issuer.
FixAdd the secret to the environment's Key Vault under the pool name; check the SecretStore is Ready and the credential subject.
Deploy job: namespace does not exist
LookThe deploy job log.
CauseThe product deploy never creates namespaces (no --create-namespace); the env chart owns them.
FixRun the environment pipeline for that environment first.
Env chart install fails patching namespaces
Lookenvironment job of env-apply.
CauseThe deploy principal has namespace-scoped RBAC only.
FixIt needs cluster-scope namespace write; AKS RBAC Admin at namespace scope is not enough.
Render fails before any cluster call
LookThe render-environment step output.
Cause<fill> or #{ in an output, a credential-like key, a key outside the A-12 allowed list, a stale pin, an unknown stack, names not matching folders, a file no layer reads, a missing "$required".
FixFix the named file in cloud-config or the product repo. --strict also rejects dummies.
An override "does nothing"
Looklayers.json from a local render; the tfa-values-20-overrides ConfigMap.
CauseA wrong service name (use the service chart alias: compodoc, not tfa-compodoc) applies to nothing, silently. Or the service's own env beats an env group.
FixUse the alias from service-chart.yaml; remove group-supplied variables from the service.
Source and release state
flux get sources oci -n app01-tfa-dev
flux get helmreleases -n app01-tfa-dev
kubectl describe helmrelease tfa -n app01-tfa-dev
The values Flux is applying
kubectl get configmap -n app01-tfa-dev | grep tfa-values
kubectl get configmap tfa-values-20-overrides -n app01-tfa-dev -o yaml
Secrets plumbing
kubectl get secretstores,externalsecrets -n app01-tfa-dev
kubectl describe externalsecret -n app01-tfa-dev
Env chart release
helm list -n environments
helm get values env-app01-dev -n environments
Caution: flux reconcile helmrelease tfa -n app01-tfa-dev --with-source changes state: it is what CD runs. Don't run it against an environment you weren't asked to touch.

09Reproduce it on your machine

The renderer is a pure function: same files, same bytes, no network. That makes most problems reproducible offline.

Render one environment
orchestrator-component-release render-environment \
  --landscapes-root cloud-config/landscapes \
  --landscape dev --environment app01 --product tfa \
  --composition composition.yaml \
  --output-dir rendered [--flux-dir flux] [--strict]
→ values.yaml, adf.yaml, databricks.yaml, cluster.yaml, layers.json (key → layer), resolved.json (config hash). --flux-dir adds the three ConfigMap layers.
The whole chain, offline
# service chart against a local platform chart
generate-service ... --repository file://<abs path>/helm-platform-service
generate-product-chart ...
helm dependency update            # Linux, Helm 3.16.3 (WSL on Windows)
helm template tfa <chart> -f rendered/values.yaml
# orchestrator tests (coverage gate 90%)
cd orchestrator-component-release && poetry run pytest

10Decision log: why it is this way

The choices that shape the system, and what each one costs. Read before proposing "why don't we just…". Open one for the reason, the cost and its state. Full records: the platform's decision log.

D-01The suffix is the chart's own version
Why
helm-controller requires OCI tag = chart version; alias tags fail.
Cost
Approval repackages (new digest), so S2 requires an identical render.
State
built
D-02Compositions in GitHub Packages, not ACR
Why
Linked to product-deploy-tfa, inherits its access; ACR holds only deployables; the Release keeps a pointer, not a copy.
Cost
cloud-config needs Actions read on the package.
State
built
D-03Flux semver tracking stays
Why
Lower environments follow every candidate with no pin change.
Cost
helm can lead adf/databricks briefly. Mitigated: publish order (chart last); adf/databricks resolved from the composition of the chart version the HelmRelease is at.
State
decided 2026-10-18
D-04Configuration-only changes compose
Why
A prefix or pool-name change would otherwise wait for the next build.
How
The composition records the product repo commit; detect diffs product.yaml/configurations/ since then.
State
built
D-05Products and environments are separate
Why
cloud-config owns environments; production changes are reviewed commits, not CD side effects.
Cost
Promotion is a hand-off: cd-promote commits the pin with a GitHub App, then follows the cloud-config run and verifies.
State
built
D-06releaseName stays tfa
Why
app.kubernetes.io/instance is in the immutable selectors; the HelmRelease adopts the helm-installed release.
Cost
Renaming a product's release means recreating its workloads.
State
built
D-07No Kustomize
Why
Helm, Flux's source and helm controllers, and the orchestrator cover it; one rendering model.
Cost
Environment variation must be expressible as values layers.
State
built
D-08No templates in configuration
Why
Plain values plus "$required" and "$url:": every file reads as what it renders. 37 templates → 0.
Rule
A setting that needs a template means the platform is missing a form.
State
built
D-09Declare only what is published
Why
Compose composes declared components only; databricks was declared once its bundles were published.
Cost
A new component is a product.yaml change plus its build.
State
helm + adf + databricks (spotlight-db, spotlight-api)
D-10One managed identity per environment
Why
id-<environment>-<landscape>, declared at the top of environment.yaml; every product namespace's shared-pp-workload uses it, a product may override.
Cost
Products of one environment share vault access and RBAC; max 20 federated credentials per identity.
State
decided 2026-10-21; env chart 0.5.0 to publish
D-11Environments may set only an allowed list
Why
Deny by default: sizing, env/appSettings, enabled, pull secrets and 25 global keys; everything else belongs to the service or product.
Cost
A new key is a reviewed change in the orchestrator.
State
built (service-layer narrowing planned)
D-12Config identified by commit + hash
Why
The cluster shows which cloud-config commit and content hash it runs; no hash-named objects.
Cost
None beyond two annotations.
State
hash built; annotations with env chart 0.5.0 to publish
D-13One workflow per process; approval on green
Why
Under the one cd.yaml → cd-router entry point every run listed the other flows as skipped, and deploy also meant approve. Now each process (Detect, Development, Release, Promote, Retention) has its own name, concurrency and permissions. Approval: "Automatic on green, because the deploy will be gated with QA testing."
Cost
Approval says only "ran green in dev"; adf/databricks deploys are still placeholders. The human gates are Release (QA) and Promote.
State
decided 2026-10-24/25; built, on main 2026-10-26; Release's qa deploy planned
D-14A human release number, typed at Release
Why
QA and customers talk in 1.4.0, not CalVer. Release republishes the approved bytes under the number; nothing is rebuilt, and the composition records its approved source.
Cost
A third artifact state (amends CD-A-03); the orchestrator and env chart must accept it as a pin before qa can deploy it.
State
decided 2026-10-25; repackage built, deploy planned (parked)
D-15Last 20, or 60 days
Why
Auto-approval creates an approved version per green dev run. Old ones are untagged everywhere; the newest, release sources and anything pinned in cloud-config are never removed.
Cost
Untag only (manifests stay in ACR).
State
decided 2026-10-25; built, report-only until CD_RETENTION_DELETE is set

11Gotchas: things that will bite you

Collected from the deploy notes and the platform chart guide. Most fail loudly; the dangerous ones are silent.

1
A wrong service name applies to nothing, silently

Overrides use the service chart's aliases (tfa-client, flatfile-listeners, compodoc), not repo names.

2
A service's own env beats an env group

Remove group-supplied variables from service-chart.yaml, or the landscape value is hidden.

3
Lists replace, maps merge

Helm deep-merges maps; a list in a later layer replaces the whole list.

4
CRLF breaks things

Python on Windows writes CRLF; use newline="\n". landscapes/ files were CRLF once.

5
Quote the YAML that looks like syntax

Strings starting with {, >, |, *, or containing {{.

6
Derived names must be "$required" or $url:

The validator rejects every ENV_DERIVED_KEYS name with a literal, even a constant. Constants go in configurations/helm.yaml at the same path.

7
An empty env: fails validation

Delete the key instead of leaving it empty.

8
Pins carry the suffix

YYYYMM.D.HHMMSS-candidate or -approved. A bare CalVer is rejected.

9
Environments set only the allowed list

Flat keys become global.<key> and must be one of the 25 A-12 global keys; containerPort, probes, ingress, secrets mapping are never per environment.

10
The product deploy never creates namespaces

Create them through the environment pipeline; its principal needs cluster-scope namespace write.

11
Pods may start before ESO syncs

CreateContainerConfigError for a few seconds is normal; minutes is not.

12
Run Helm in WSL on Windows

The chain is tested on Linux with Helm 3.16.3.

13
Federated credential names must be unique

Creating a credential with an existing name overwrites its subject: one environment's namespace silently loses Key Vault access.