Deep dive · engineers
The first product (TFA), one release end to end
TFA is a composition of components (helm, adf, databricks) delivered at one version by two independent pipelines.
This page follows one release through every artifact, file and controller it touches, then hands you the runbook, the decisions behind it and the traps to avoid.
Summary
TL;DR in six lines
- Service repos and
powerplan-tfa-adfbuild artifacts at CalVer and push them to ACR. - CD · Development (push to
main, or started by CD · Detect every 30 min) runscd-detect: newest builds and product files vs the latest approved composition. cd-composewrites one product version: adf/databricks aliases, composition (ghcr), product chart last.- Tracking environments' Flux pulls the
-candidatechart; CD waits for HelmRelease Ready on app01. cd-approveruns automatically when every dev deploy is green:-approvedeverywhere and the Release pointer, the next detect baseline.- qa and prod move only by CD · Promote (
cd-promote): a pin commit in cloud-config, applied byenvironments.yml→env-apply. CD · Release gives an approved version a release number; its qa deploy is planned. Today there is no qa or prod: only dev/app01 exists, so Promote and Release stop at their first step (plan) until greenfield IaC creates those environments.
01Four invariants to keep in your head
Everything else on this page is a consequence of these. If a change you are about to make breaks one, stop and read the decision log.
One product, one version
Every component carries <v> (CalVer YYYYMM.D.HHMMSS). The state is part of the version: <v>-candidate, <v>-approved. A bare CalVer pin is rejected.
The composition decides
What a version contains is the composition, not "whatever is newest". Detect compares against the latest approved one; every deploy pulls it first and checks the chart digest built.
Versions move two ways only
Lower environments: Flux tracking -candidate. qa and prod: a pin commit in cloud-config made by cd-promote. Nothing else changes a version. A release number (CD · Release) is a third label on the same approved bytes, not a new version.
Values layer; secrets don't
Later layer wins at the same path; identity is last and fixed. Secrets are names in values and values only in Key Vault. Bad input fails the render, not the pod.
02Two pipelines, one hand-off
Each pipeline runs on its own. They meet at one point: a promotion goes from the product pipeline (cd-promote) as a
pin commit into cloud-config, which the environment pipeline applies.
Product pipeline product-deploy-tfa
- Owns
- Versions: the composition of every component,
<v>-candidate→<v>-approved - Runs when
- Five workflows: CD · Development on a push to
main, a PR (dry run), "Run workflow" or a start by CD · Detect (every 30 min, only when a component build or aproduct.yaml/configurations/change moved); CD · Release, CD · Promote by hand; CD · Retention weekly built - Changes
- ACR, GitHub Packages and GitHub Releases; cloud-config only through a promotion commit (GitHub App)
- Reaches environments
- Tracking envs (
track: candidate) pull each candidate through Flux; app01 is watched until Ready, then the run approves it automatically (no reviewer)
cd-promoteapproved versionpin commit, gated by
tfa-<landscape>-releaseEnvironment pipeline cloud-config
- Owns
- Environments:
environment.yaml, landscape values, per-environment values, pins - Runs when
- A
landscapes/**push (including a promotion commit), a pull request (render only), or "Run workflow" for one environment:environments.yml→env-apply - Changes
- The cluster: env chart
env-<environment>-<landscape>, then every enabled product (or the ones named) - Use it for
- A new environment, value overrides for an existing one, a production version pin
Click a stage, then use ← / → to move through that pipeline.
03The pieces
The repositories (GitHub organisation in brackets), what each one does, and the files to open first. Which branch each one uses is in the repository README and the platform knowledge base; it changes often.
| Piece | Repo [org] | Role |
|---|---|---|
| Platform chart | helm-platform-service [PowerPlanDev]2.4.1 released · 2.5.0 to publish |
platform-services. Archetypes app (Deployment, Service, Ingress/HTTPRoute, HPA or KEDA, PDB, SA, ExternalSecret) and task (Job, hooks; CronJob with schedule), plus the libraries global and keda. KEDA and cron arrive in 2.5.0 to publish. Published at oci://ppdevops.azurecr.io/helm/platform/platform-services. |
| Orchestrator | orchestrator-component-release [PowerPlanDev] |
Python CLI: generate-service, generate-product-chart (--version <v>-candidate), write-composition, approve-composition, pipeline-plan, render-environment (--flux-dir, --composition; A-12 allowed list, resolved.json hash), adf-package, databricks-package, export-schema built. CI installs it from git with ORCHESTRATOR_TOKEN. |
| Shared CI/CD | cloud-cicd-templates [PowerPlanDev] |
Golden paths gp-dotnet-service, gp-python, gp-adf, gp-databricks-bundle, helm-chart (and gp-nx-monorepo, not yet merged); product CD cd-plan, cd-candidate, cd-detect, cd-compose, cd-deploy-environment, cd-approve, cd-release, cd-promote, cd-retention; environments env-apply; actions version, service-chart built. |
| Service repos | powerplan-tax-fixed-assets, powerplan-client (chart in deploy/tfa/), powerplan-active-reports-service, powerplan-content, powerplan-databricks-zerobus |
Each has a deploy/service-chart.yaml. CI builds the images, then the service chart. |
| ADF and Databricks repos | powerplan-tfa-adf, powerplan-spotlight-db, powerplan-spotlight-api [PowerPlanDev] |
gp-adf: validate + export, adf-package, oras push powerplan/tfa-adf-artifact:<CalVer>. gp-databricks-bundle: databricks-package, oras push databricks/<bundle>:<CalVer> built (PRs: package only). |
| Product repo | product-deploy-tfa [PowerPlanDev] |
product.yaml (kind: Product; components.helm chart + serviceCharts, components.adf, components.databricks spotlight-db + spotlight-api), configurations/{helm,adf,databricks}.yaml (the global.secrets pool is in helm.yaml), pipeline.config.yaml (the ladder; dev/app01 only today). Five thin workflows, one per process: cd-detect.yml, cd-development.yml, cd-release.yml, cd-promote.yml, cd-retention.yml (until 2026-10-26: one cd.yaml → cd-router). |
| Environments | cloud-config [PowerPlanDev] |
The landscapes/ tree. Workflow environments.yml → env-apply (the 2026-10-19 consolidation removed deploy-changed, deploy-environment, promote-product, bump-environment-pin). |
| Environment chart | helm-platform-environments [PowerPlanSpikes]0.4.1 released · 0.5.0 to publish |
platform-environment: namespaces, ServiceAccount shared-pp-workload, the Key Vault SecretStore, and (under Flux) the OCIRepository, HelmRelease and values ConfigMaps for each product; 0.5.0 adds the config-hash annotations and the environment-level identity to publish. Published at oci://ppdevops.azurecr.io/helm/platform/platform-environment. |
Service charts in TFA today (dev)
tax-fixed-assets | tax-fixed-assets-api, tfa-migrations, redis |
tfa-frontend | tfa-client, flatfile-listeners, compodoc |
active-reports-service | active-reports-service |
content-deploy | content-deploy |
tfa-zerobus | zerobus-worker, zerobus-dispatcher (off by default; needs 2.5.0) |
Artifacts and registries
- ACR: images and
helm/services/<chart>at build CalVer - ACR
helm/products/tfa-app:<v>-candidate/<v>-approvedas the chart's own version built - ACR
powerplan/tfa-adf-artifact: build CalVer;<v>-candidate/-approvedaliases built - ACR
databricks/powerplan-spotlight-db,databricks/powerplan-spotlight-api: same scheme built - GitHub Packages
ghcr.io/powerplandev/tfa-composition:<v>-candidate,<v>-approvedbuilt
Cluster add-ons: Flux Operator (source and helm controllers), External Secrets Operator, KEDA (AKS add-on 2.17), registry pull Secret acr-pull-secret. Key Vault access as the environment's identity through shared-pp-workload (or Secret keyvault-sp with authType: ServicePrincipal).
04Model and composition
A product is a composition of components with one version and one deployment per environment. helm is the product chart tfa-app (service charts, each holding services);
adf the Data Factory ARM package; databricks a named map of Asset Bundles (TFA: spotlight-db, then spotlight-api), deployed in the order listed. Compose composes declared components only.
# ideas/product/composition.example.yaml (abridged)
apiVersion: v1
kind: Composition
product: tfa
version: 202610.22.093015
state: candidate # approved after the deploy ladder
source: {repository: PowerPlanDev/product-deploy-tfa, commit: …}
components:
helm:
chart: helm/products/tfa-app
version: 202610.22.093015-candidate
packageDigest: sha256:… # checked at every deploy
serviceCharts: {tax-fixed-assets: 202610.21.141502, tfa-frontend: 202610.22.081144, …}
adf:
artifact: powerplan/tfa-adf-artifact
version: 202610.22.093015-candidate
digest: sha256:…
built: 202610.18.110230
databricks:
spotlight-db: {artifact: databricks/powerplan-spotlight-db, version: …-candidate, built: …}
spotlight-api: {artifact: databricks/powerplan-spotlight-api, version: …-candidate, built: …}
approval: {} # approvedAt, approvedBy, run
Source of truth of every deploy
- Models,
write-composition/approve-compositionand publishing built - Compose pushes
tfa-composition:<v>-candidateto ghcr; every deploy pulls it first, checks the chart'spackageDigestand deploys exactly what it names - Approval pushes
:<v>-approved; the GitHub Release notes carry its address and digest (a pointer, no copy) - Detect follows the latest approved release to its composition
Access
- Package linked to
product-deploy-tfa(org.opencontainers.image.source), inherits its access - Compose/approve:
GITHUB_TOKENwithpackages: write;cloud-configgets Actions read on the package - ACR holds only deployables
Decision D-03: Flux tracking stays
- Flux semver tracking stays: a tracking env's helm follows the newest
-candidatechart; adf/databricks move when a deploy runs - Skew kept small: publish order adf/databricks → composition → chart last; product CD deploys the candidate's other components right after compose
- Every deploy resolves adf/databricks from the composition of the chart version the HelmRelease is at
05Values layering: lowest first, later wins
helm component shown. The first three layers are baked into the product chart. Under Flux, the last three are ConfigMaps owned by the env chart release and listed in that order in the HelmRelease's valuesFrom.
platform-services values.yamldeploy/service-chart.yaml: appSettings, env, envGroups, secrets, "$required", $url:product-deploy-tfa/configurations/helm.yaml: global (prefixes, secret pool) + one key per service chartlandscapes/<landscape>/environments.yaml → products.tfalandscapes/<landscape>/<env>/tfa/helm.yamltfa-values-*↑ later winsSecrets never travel in values
- The product pool (
configurations/helm.yamlsecrets) names Key Vault secrets. - Each service maps an env var to a pool name.
- The chart renders one
ExternalSecretper service; ESO syncs it through the SecretStorekeyvault, read as the environment's identity (shared-pp-workload).
Merge rules
- Helm merges maps and replaces lists.
- Landscape and environment layers may set only the A-12 allowed list (env, appSettings, sizing, enabled, imagePullSecrets, 25 global keys); anything else fails the render.
- The rendered values hold only the landscape and environment deltas, never the product tier.
- Per-service keys nest by service chart, then service, then archetype:
services.tax-fixed-assets.tax-fixed-assets-api.app…
Per component
| Tier (later wins) | helm | adf | databricks |
|---|---|---|---|
| chart defaults / service chart | platform-services values, service-chart.yaml | – | – |
| product | configurations/helm.yaml (baked into the chart) | configurations/adf.yaml (in the composition) | configurations/databricks.yaml |
| landscape | environments.yaml products.tfa | – | – |
| environment | <env>/tfa/helm.yaml (optional; ConfigMap 20-overrides) | <env>/tfa/adf.yaml | <env>/tfa/databricks.yaml |
| identity | computed, cannot be overridden (ConfigMap 30-identity) | – | – |
Missing or empty file = {}; a file for an undeclared component fails. render-environment --composition merges adf and databricks (per bundle) built; the ADF deploy becomes a real az deployment group create into the environment's factory resource group and the bundle deploy a real databricks bundle deploy planned (both print their commands today).
06Where did this value come from?
Six real keys from TFA on dev/app01, traced hop by hop through the files that set them. Pick one. (render-environment writes the same answer to layers.json, and the content hash to resolved.json.)
07environment.yaml and Flux
With delivery: flux, the env chart (templates/flux.yaml) renders three things per product in its namespace. CI applies nothing to the product; it asks Flux to reconcile and waits.
# cloud-config/landscapes/dev/app01/environment.yaml
environment: app01
landscape: dev
stack: dev-v1
delivery: flux # helm (default) or flux
workloadIdentityClientId: … # the environment's identity id-<env>-<landscape>
products:
tfa:
enabled: true
track: candidate # or version: <CalVer>-approved (a pin)
keyVault:
name: kv-app01-prod # override until kv-app01-dev exists
authType: WorkloadIdentity
tenantId: …
OCIRepository tfa-chart
oci://ppdevops.azurecr.io/helm/products/tfa-app, secretRef: acr-pull-secret.
track → semver: ">=202610.0.0-0" + semverFilter: '.*-candidate$'; pin → tag.
HelmRelease tfa
releaseName: tfa (helm-installed releases are adopted, because app.kubernetes.io/instance is in the immutable selectors). chartRef → the OCIRepository; install and upgrade remediation retries 3.
ConfigMaps tfa-values-*
10-landscape, 20-overrides (optional), 30-identity; labelled reconcile.fluxcd.io/watch: Enabled, so a changed layer re-reconciles the release. With env chart 0.5.0 to publish they and the HelmRelease carry powerplan.com/config-hash and powerplan.com/config-commit.
08When it's stuck: a runbook
Symptoms you will actually see, where to look, and the usual cause. Commands are read-only unless flagged. Namespace for TFA on app01: app01-tfa-dev.
HelmRelease not Readyartifact revision … does not match chart version …
kubectl describe helmrelease tfa -n app01-tfa-dev<v>-candidate and approval repackages <v>-approved.cd-compose/cd-approve; never re-tag a chart by hand.Deploy stops: digest mismatchpackageDigest
cd-deploy-environment log, composition step.<v> is not the package the composition recorded (re-pushed or overwritten tag), or no composition exists for that version.cd-compose produced; investigate who pushed the tag.Flux doesn't see the new candidateflux get sources oci
ref.semver: ">=202610.0.0-0" + semverFilter: '.*-candidate$'; a pinned environment uses tag and never moves on its own.environment.yaml has track: candidate; CD requests a reconcile instead of waiting for the interval.Pod CreateContainerConfigError
kubectl get externalsecrets,secretstores -n app01-tfa-devKey Vault Secrets User, or its federated credential does not match system:serviceaccount:<namespace>:shared-pp-workload on this cluster's issuer.Deploy job: namespace does not exist
--create-namespace); the env chart owns them.Env chart install fails patching namespaces
environment job of env-apply.Render fails before any cluster call
render-environment step output.<fill> or #{ in an output, a credential-like key, a key outside the A-12 allowed list, a stale pin, an unknown stack, names not matching folders, a file no layer reads, a missing "$required".cloud-config or the product repo. --strict also rejects dummies.An override "does nothing"
layers.json from a local render; the tfa-values-20-overrides ConfigMap.compodoc, not tfa-compodoc) applies to nothing, silently. Or the service's own env beats an env group.service-chart.yaml; remove group-supplied variables from the service.flux get sources oci -n app01-tfa-dev
flux get helmreleases -n app01-tfa-dev
kubectl describe helmrelease tfa -n app01-tfa-devkubectl get configmap -n app01-tfa-dev | grep tfa-values
kubectl get configmap tfa-values-20-overrides -n app01-tfa-dev -o yamlkubectl get secretstores,externalsecrets -n app01-tfa-dev
kubectl describe externalsecret -n app01-tfa-devhelm list -n environments
helm get values env-app01-dev -n environments
flux reconcile helmrelease tfa -n app01-tfa-dev --with-source changes state: it is what CD runs. Don't run it against an environment you weren't asked to touch.09Reproduce it on your machine
The renderer is a pure function: same files, same bytes, no network. That makes most problems reproducible offline.
orchestrator-component-release render-environment \
--landscapes-root cloud-config/landscapes \
--landscape dev --environment app01 --product tfa \
--composition composition.yaml \
--output-dir rendered [--flux-dir flux] [--strict]
values.yaml, adf.yaml, databricks.yaml, cluster.yaml, layers.json (key → layer), resolved.json (config hash). --flux-dir adds the three ConfigMap layers.# service chart against a local platform chart
generate-service ... --repository file://<abs path>/helm-platform-service
generate-product-chart ...
helm dependency update # Linux, Helm 3.16.3 (WSL on Windows)
helm template tfa <chart> -f rendered/values.yaml
# orchestrator tests (coverage gate 90%)
cd orchestrator-component-release && poetry run pytest10Decision log: why it is this way
The choices that shape the system, and what each one costs. Read before proposing "why don't we just…". Open one for the reason, the cost and its state. Full records: the platform's decision log.
D-01The suffix is the chart's own version
- Why
- helm-controller requires OCI tag = chart
version; alias tags fail. - Cost
- Approval repackages (new digest), so S2 requires an identical render.
- State
- built
D-02Compositions in GitHub Packages, not ACR
- Why
- Linked to
product-deploy-tfa, inherits its access; ACR holds only deployables; the Release keeps a pointer, not a copy. - Cost
cloud-configneeds Actions read on the package.- State
- built
D-03Flux semver tracking stays
- Why
- Lower environments follow every candidate with no pin change.
- Cost
- helm can lead adf/databricks briefly. Mitigated: publish order (chart last); adf/databricks resolved from the composition of the chart version the HelmRelease is at.
- State
- decided 2026-10-18
D-04Configuration-only changes compose
- Why
- A prefix or pool-name change would otherwise wait for the next build.
- How
- The composition records the product repo commit; detect diffs
product.yaml/configurations/since then. - State
- built
D-05Products and environments are separate
- Why
cloud-configowns environments; production changes are reviewed commits, not CD side effects.- Cost
- Promotion is a hand-off:
cd-promotecommits the pin with a GitHub App, then follows the cloud-config run and verifies. - State
- built
D-06releaseName stays tfa
- Why
app.kubernetes.io/instanceis in the immutable selectors; the HelmRelease adopts the helm-installed release.- Cost
- Renaming a product's release means recreating its workloads.
- State
- built
D-07No Kustomize
- Why
- Helm, Flux's source and helm controllers, and the orchestrator cover it; one rendering model.
- Cost
- Environment variation must be expressible as values layers.
- State
- built
D-08No templates in configuration
- Why
- Plain values plus
"$required"and"$url:": every file reads as what it renders. 37 templates → 0. - Rule
- A setting that needs a template means the platform is missing a form.
- State
- built
D-09Declare only what is published
- Why
- Compose composes declared components only; databricks was declared once its bundles were published.
- Cost
- A new component is a
product.yamlchange plus its build. - State
- helm + adf + databricks (spotlight-db, spotlight-api)
D-10One managed identity per environment
- Why
id-<environment>-<landscape>, declared at the top ofenvironment.yaml; every product namespace'sshared-pp-workloaduses it, a product may override.- Cost
- Products of one environment share vault access and RBAC; max 20 federated credentials per identity.
- State
- decided 2026-10-21; env chart 0.5.0 to publish
D-11Environments may set only an allowed list
- Why
- Deny by default: sizing, env/appSettings, enabled, pull secrets and 25 global keys; everything else belongs to the service or product.
- Cost
- A new key is a reviewed change in the orchestrator.
- State
- built (service-layer narrowing planned)
D-12Config identified by commit + hash
- Why
- The cluster shows which cloud-config commit and content hash it runs; no hash-named objects.
- Cost
- None beyond two annotations.
- State
- hash built; annotations with env chart 0.5.0 to publish
D-13One workflow per process; approval on green
- Why
- Under the one
cd.yaml→cd-routerentry point every run listed the other flows as skipped, anddeployalso meant approve. Now each process (Detect, Development, Release, Promote, Retention) has its own name, concurrency and permissions. Approval: "Automatic on green, because the deploy will be gated with QA testing." - Cost
- Approval says only "ran green in dev"; adf/databricks deploys are still placeholders. The human gates are Release (QA) and Promote.
- State
- decided 2026-10-24/25; built, on
main2026-10-26; Release's qa deploy planned
D-14A human release number, typed at Release
- Why
- QA and customers talk in
1.4.0, not CalVer. Release republishes the approved bytes under the number; nothing is rebuilt, and the composition records its approved source. - Cost
- A third artifact state (amends CD-A-03); the orchestrator and env chart must accept it as a pin before qa can deploy it.
- State
- decided 2026-10-25; repackage built, deploy planned (parked)
D-15Last 20, or 60 days
- Why
- Auto-approval creates an approved version per green dev run. Old ones are untagged everywhere; the newest, release sources and anything pinned in cloud-config are never removed.
- Cost
- Untag only (manifests stay in ACR).
- State
- decided 2026-10-25; built, report-only until
CD_RETENTION_DELETEis set
11Gotchas: things that will bite you
Collected from the deploy notes and the platform chart guide. Most fail loudly; the dangerous ones are silent.
Overrides use the service chart's aliases (tfa-client, flatfile-listeners, compodoc), not repo names.
env beats an env groupRemove group-supplied variables from service-chart.yaml, or the landscape value is hidden.
Helm deep-merges maps; a list in a later layer replaces the whole list.
Python on Windows writes CRLF; use newline="\n". landscapes/ files were CRLF once.
Strings starting with {, >, |, *, or containing {{.
"$required" or $url:The validator rejects every ENV_DERIVED_KEYS name with a literal, even a constant. Constants go in configurations/helm.yaml at the same path.
env: fails validationDelete the key instead of leaving it empty.
YYYYMM.D.HHMMSS-candidate or -approved. A bare CalVer is rejected.
Flat keys become global.<key> and must be one of the 25 A-12 global keys; containerPort, probes, ingress, secrets mapping are never per environment.
Create them through the environment pipeline; its principal needs cluster-scope namespace write.
CreateContainerConfigError for a few seconds is normal; minutes is not.
The chain is tested on Linux with Helm 3.16.3.
Creating a credential with an existing name overwrites its subject: one environment's namespace silently loses Key Vault access.
12Where to read more
Docs in the repos
- helm-platform-service · CHART-GUIDE.md: the platform chart
- cloud-config · landscapes/README.md: tiers and environments
- cloud-cicd-templates · docs/golden-paths: compose, deploy, approve
- product-deploy-tfa · product.yaml: the product and its components
Workflows to read first
- product-deploy-tfa · cd-detect.yml, cd-development.yml, cd-release.yml, cd-promote.yml, cd-retention.yml
- cloud-cicd-templates · cd-plan, cd-candidate, cd-detect, cd-compose, cd-deploy-environment, cd-approve, cd-release, cd-promote, cd-retention, env-apply
- cloud-config · environments.yml
- helm-platform-environments · templates/flux.yaml
Diagrams and slides
- All diagrams and slide decks, by audience
- Delivery architecture and product pipeline sequence
- Diagram sources: the
specs/folder (see the README)