Skip to content

CELINE Infrastructure (infra)

This repository contains the infrastructure-as-code used to deploy and operate the CELINE platform across local, staging, and production environments.

It defines: - Kubernetes infrastructure - Helm / Helmfile-based deployments - Encrypted secrets handling - Environment-specific configuration

The repository is operator-oriented and assumes familiarity with Kubernetes tooling.

Overview

CELINE infrastructure follows a declarative and reproducible model based on:

  • Helm charts as the primary deployment unit
  • Helmfile to coordinate multiple Helm releases
  • Helm plugins for diffing and secrets integration
  • SOPS for encrypted configuration
  • Task as a convenience wrapper for common operational commands
  • Minikube for local development

No imperative deployment scripts are used.
Infrastructure is applied using Helmfile-driven workflows.

Repository Layout

infra/
├── charts/           # CELINE and third-party Helm charts
├── envs/             # Environment bindings (symlinks)
├── defaults/         # Default configurations for charts
├── helmfile.d/       # helmfile catalogue of Helm charts
└── .sops.yaml/.sops  # SOPS-encrypted secrets

Required Tooling (Local Setup)

Local setup is mandatory. Install the following tools:

  • task
    https://taskfile.dev/docs/installation

  • minikube
    https://minikube.sigs.k8s.io/docs/start

  • kubectl
    https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

  • helm
    bash curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4 | bash

  • helm-diff (required by Helmfile)
    bash helm plugin install https://github.com/databus23/helm-diff --verify=false

  • helm-secrets (required by Helmfile)
    bash helm plugin install https://github.com/jkroepke/helm-secrets/releases/download/v4.7.5/secrets-4.7.5.tgz --verify=false helm plugin install https://github.com/jkroepke/helm-secrets/releases/download/v4.7.5/secrets-getter-4.7.5.tgz --verify=false helm plugin install https://github.com/jkroepke/helm-secrets/releases/download/v4.7.5/secrets-post-renderer-4.7.5.tgz --verify=false

  • helmfile
    https://helmfile.readthedocs.io/en/latest/#installation

  • skaffold
    https://skaffold.dev/docs/install/#standalone-binary

Missing any of the above will result in a broken setup.

Local Kubernetes Environment (Minikube)

Start Minikube with sufficient resources:

minikube start --cpus=4 --memory=8192

Ensure your kube context is set correctly:

kubectl config use-context minikube

Local DNS Configuration (*.celine.test)

CELINE services rely on Ingress host-based routing.

For local development, services are exposed under *.celine.test where *.test is resolved by minikube

Add the following entry to /etc/hosts:

192.168.49.2 api.celine.test webapp.celine.test assistant.celine.test dashboard.celine.test s3.celine.test keycloak.celine.test marquez.celine.test mqtt.celine.test sso.celine.test prefect.celine.test superset.celine.test

Notes: - Replace 192.168.49.2 with the output of minikube ip if different - Hostnames must match ingress definitions - OAuth redirect URIs depend on these domains

Plain localhost will not work due to OIDC issuer mismatch.

Secrets Management (SOPS)

All secrets are stored encrypted at rest, in envs/<env>/secrets.sops.yaml, with the AGE key at .sops/<env>/key.txt.

They are edited in place, never through a plaintext copy:

task secrets:edit:dev                    # sops edit in VS Code, re-encrypted on save
task secrets:edit:dev SOPS_EDITOR=vim    # any other editor (an exported SOPS_EDITOR works too)
task secrets:dump:dev | grep smtp        # read-only, to stdout

sops edit keeps the file's data key, so only the values you change get new ciphertext and two edits of different keys merge. The sops.lastmodified / sops.mac lines always conflict: keep either side, then recompute the MAC without touching a value:

SOPS_EDITOR="sed -i '\$a zz_remac: x'" sops edit --ignore-mac envs/<env>/secrets.sops.yaml
SOPS_EDITOR="sed -i '/^zz_remac/d'" sops edit envs/<env>/secrets.sops.yaml

task sops:create-env -- <env> and task sops:copy-dev-keys create an encrypted stub when the file is missing.

Helmfile integrates with helm-secrets to decrypt secrets at deploy time.

Plaintext secrets must never be committed.

Email (SMTP)

Services that send email read one shared smtp block from the environment's values and secrets:

smtp:
  host: ""            # empty: no email
  port: 587
  ssl: false
  starttls: true
  auth: true          # defaults to true when `user` is set
  user: ""            # keep in secrets
  password: ""        # keep in secrets
  from: ""
  fromDisplayName: ""
  replyTo: ""

A service can use a different provider through its own block, keycloak.smtp or superset.smtp:

  • Provider settings (host, port, ssl, starttls, auth, user, password) are taken as a whole. The service's own block is used when its host is set. Otherwise the shared block is used. Values are never mixed across blocks.
  • Sender settings fall back one at a time. Each field comes from the service's own block when set there, and otherwise from the shared block. The Keycloak fields are from, fromDisplayName and replyTo. Superset's is superset.smtp.mailFrom, which falls back to smtp.from.
  • Keycloak email is off when keycloak.smtp.enabled is false, even if a host resolves.
  • Superset's image always uses STARTTLS, so it ignores ssl and starttls.

The Keycloak realm import applies SMTP to a new realm only. An existing realm is not changed by the import.

The provisioning service does not send email itself: it asks Keycloak to. Its provisioning.email_mode defaults to dev, which emails nobody except the addresses in provisioning.email_dev_recipients. Set deliver only in an environment that should email participants.

Onboarding

The REC onboarding service runs as two releases on one host, onboarding.<domain>: onboarding (the API, charts/celine-onboarding) and frontend-onboarding (the UI, charts/celine-frontend-onboarding). Both are off until the environment sets onboarding.enabled: true, which also adds the onboarding database to postgres-db and ONBOARDING_URL to the community release.

The member wizard is public. Only /api/admin and /api/me go through oauth2-proxy, and every /api path goes straight to the API, so the API sees the ingress controller as its peer.

Value Effect
onboarding.enabled installs both releases, the database and ONBOARDING_URL
onboarding.forwarded_allow_ips the ingress controller's pod address range, uvicorn's FORWARDED_ALLOW_IPS. Rate limits, consent evidence and audit rows use the address it resolves. Default 0.0.0.0, an address no peer has: no proxy is trusted and every caller shares the ingress's address, so set the range per environment. The chart refuses * and a /0 network
onboarding.encryption_key required, secret: the Fernet key for personal data at rest. Losing it loses the data
postgres_db.onboarding required, secret: the database role's password. It goes into a connection URI unescaped, so use a URL-safe value
onboarding.client_secret, onboarding.cli_client_secret secret: svc-onboarding and svc-onboarding-cli, the same keys policies-shell syncs to the realm
onboarding.image_tag, onboarding.ui_image_tag the API image tag, and the UI's (defaults to the API's)
onboarding.host defaults to onboarding.<domain>
onboarding.dataspace_enabled false by default. A template declaring consent.data_sharing also needs a connector, or the API does not start. The add-on layer's dataspace.enabled turns it on as well (Add-on layer)
onboarding.sms_provider none by default: no phone verification. The service's own default, log, only prints the code to the pod log. brevo also needs dpa_sms_signed and brevo_api_key
onboarding.smtp overrides the shared smtp block, as in Email (SMTP). STARTTLS or plain only: a resolved block with ssl: true stops the render
onboarding.templates.files, onboarding.templates.binary_files the community templates, <slug>/<path> to text or base64, mounted from a ConfigMap (1 MiB at most)
onboarding.persistence {enabled, size, storage_class}: a volume for uploaded documents. Off by default, because document scanning is off
onboarding.env further plain settings for the API, NAME: value

charts/celine-onboarding/examples/example-rec.values.yaml is a complete example. The API runs its Alembic migrations in an init container. After a template changes, import it:

kubectl -n celine-<env> exec deploy/onboarding -- \
  /app/.venv/bin/onboarding-cli import-templates --filter <slug>

ROI calculator

The ROI calculator is public by design: roi.<domain> needs no sign-in and the API behind it takes no token. celine-frontend-roi routes only the calculator's API paths to roi (ingress.apiPaths); any other /api path - /api/v1/estimates, the stored calculator inputs, readable only with the realm role platform-admin - never reaches roi through the public host. What one caller can cost is bounded per client address twice: at the edge, on the API Ingresses only, and inside roi.

The feedback form (ingress.authPaths, /api/v1/feedback and below) is the one exception: it is for visitors who already hold an SSO session, e.g. from the webapp. Those paths pass oauth2-proxy's auth check (auth-url, no auth-signin), which hands roi the user's token; without a session the request gets a plain 401 and no redirect, so nobody is asked to sign in on roi.<domain>. roi verifies the token against the audience oauth2_proxy.

Value Effect
roi.forwarded_allow_ips the ingress controller's pod address range, uvicorn's FORWARDED_ALLOW_IPS. roi's rate limits and the client address it stores use the address it resolves. Unset (or REQUIRED-ingress-pod-cidr), every visitor shares the ingress's address and one budget. * stops the render
roi.rate_limit_calculators_per_minute, roi.rate_limit_feedback_per_minute roi's per-address limits; roi answers 429 with Retry-After. Default 30, 5
roi.estimates_max_writes_per_minute stored estimates per minute for the whole service; above it results are served and not stored. Default 60
roi.client_ip_retention_days age after which a stored client address is cleared (estimates and feedback). Default 30
frontend_roi.edge_limit_rps, frontend_roi.edge_limit_connections ingress-nginx limit-rps / limit-connections on the API paths, per client address. Default 2, 10. They assume the controller sees the real client address

The roi settings other than forwarded_allow_ips are read by roi releases after v1.10.3; an older image ignores them.

Environment signal and token verification

Every celine service, the operator shells (assistant, dataset-api, rec-registry), the pipelines and Superset get CELINE_ENV, the environment's name. Only dev relaxes anything; any other value, unset included, is hardened, and a hardened service refuses to start with a development value (default database password, client secret equal to the client id, missing issuer or key). Images before the celine-sdk release that carries posture ignore the variable.

The frontends built from celine-frontend (webapp, assistant, community, grid, roi) get the same value as PUBLIC_CELINE_ENV: anything but prod shows a banner saying the UI is a development version. Outside prod, with prod_reads.domain set, they also get PUBLIC_CELINE_PROD_URL, https://<the app's host>.<prod_reads.domain>, which the banner links; without it the banner has no link. Frontend images before the banner ignore both.

Value Effect
celine_env overrides CELINE_ENV. Default: the helmfile environment's name. dev outside the environment dev stops the render
qdrant.api_key secret: the key Qdrant enforces (apiKey) and the assistant sends (QDRANT_API_KEY, through ai-assistant-secrets). Required by the assistant outside dev. Unset: neither is rendered. Sync ai-assistant and ai-assistant-shell before qdrant
nudging.vapid_private_key secret: VAPID_PRIVATE_KEY, the Web Push signing key, through nudging-secrets (never a plain env value). nudging.vapid_public_key and nudging.vapid_subject stay plain values. A new key pair invalidates the push subscriptions already made
nudging.click_tracking_secret secret: CLICK_TRACKING_SECRET, the HMAC key of tracked notification links, through nudging-secrets. Required by nudging outside dev from the release that carries posture. Older images sign with VAPID_PRIVATE_KEY when it is unset, so setting it invalidates the links already sent
celine_audit.pseudonym_key secret: CELINE_AUDIT_PSEUDONYM_KEY, the HMAC key the celine-sdk access audit pseudonymises person ids with, through each service's <release>-secrets. One random key per environment (openssl rand -hex 32), so a person has the same pseudonym in every service; changing it changes every pseudonym. Unset: not rendered, and the SDK falls back to unkeyed SHA-256
community.webapp_api_url, community.roi_api_url WEBAPP_API_URL and ROI_API_URL of the community release: where the manager inbox reads participant and ROI-calculator feedback. Default: http://webapp:8014 and http://roi:8000. A community image that refuses development values also refuses a host.docker.internal upstream, which is what an unset one falls back to
mosquitto.go_auth_log_level mosquitto-go-auth's auth_opt_log_level, default info. debug writes whole bearer tokens to the broker log, so it stops the render outside the environment dev

Derived, not set:

  • Superset trusts the tokens of one issuer: CUSTOM_SECURITY_MANAGER_KEYCLOAK_ISSUER is https://keycloak.<domain>/realms/<keycloak.realm>, the realm oauth2-proxy signs users in with, and CUSTOM_SECURITY_MANAGER_KEYCLOAK_AUDIENCE is auth_setup.clientID (oauth2_proxy). Plain env on web, worker, beat and the run-setup container: from celine-superset ADR-0003, Superset does not start outside dev without the issuer. superset.acceptSelfSignedCerts still turns TLS verification of the key fetch off.
  • The assistant gets OAUTH2_ISSUER (the same realm URL), OAUTH2_AUDIENCE (auth_setup.clientID) and OAUTH2_TRUST_HEADERS=false; it reads the key set from CELINE_OIDC_JWKS_URI (celine-services), or discovers it from the issuer on images that do not read that variable.

Ingress controller

Every chart here writes plain Ingress objects for the nginx class, with nginx.ingress.kubernetes.io/* annotations. Where the cluster already runs a controller for that class, infra installs none. Where it does not, the environment asks for one, and infra installs Traefik with its NGINX provider. That provider reads the same annotations, so no chart changes (ingress-nginx itself is retired upstream). Where both run side by side, Traefik takes a class of its own (ingress_controller.class and controller_class below) and every chart follows it.

Value Effect
ingress_controller.enabled false unless set. true installs the traefik release (group core)
ingress_controller.namespace default ingress
ingress_controller.class the IngressClass it creates and serves, default nginx. Every chart's Ingresses, and the ACME HTTP-01 solver, use it
ingress_controller.controller_class the controller that IngressClass names, default k8s.io/ingress-nginx. Where the cluster also runs an ingress-nginx, set a class and a controller of Traefik's own: ingress-nginx answers every IngressClass that names k8s.io/ingress-nginx, so the two would otherwise share every Ingress
ingress_controller.replicas default 2, with a PodDisruptionBudget of one
ingress_controller.service.annotations load-balancer annotations for the provider the cluster runs on
ingress_controller.service.externalTrafficPolicy default Local, which keeps the client address that whitelist-source-range, the rate limits and forwarded_allow_ips rely on
ingress_controller.resources default requests 250m / 256Mi, memory limit 512Mi
tls_setup.cert_manager.leader_election_namespace where cert-manager takes its leader-election lock. Default kube-system, the chart's own; set it to cert-manager on clusters that refuse writes to kube-system

Traefik differs from ingress-nginx on a few annotations: auth-url forward auth behaves differently (test the oauth2-proxy login end to end), custom-headers works once its header names are allowed (globalAllowedResponseHeaders, ingress-nginx's global-allowed-response-headers: infra allows the "Security headers" list below, and ingress_controller.allowed_response_headers adds more; any other header turns every path of the Ingress into a 503), and configuration-snippet is only partly supported. Overrides go in envs/<env>/0010-core/traefik/values.yaml.gotmpl.

Network perimeter

An environment can close its namespace celine-<env> to the rest of the cluster (release celine-network-policy, group core). Off unless asked, so an environment that does not set it renders unchanged.

Value Effect
network_policy.enabled false unless set. true installs two NetworkPolicies: every pod refuses incoming connections, except from the namespace itself, the ingress controller's namespace and cnpg
network_policy.ingress_namespace the ingress controller's namespace. Default: ingress_controller.namespace where infra installs Traefik, else ingress-nginx
network_policy.allow_namespaces further namespaces allowed in (a dataspace calling celine, a metrics scraper), default none
network_policy.pod_security.{enforce,warn,audit} Pod Security Admission level per mode (privileged, baseline, restricted), set as namespace labels by a presync hook; any other value stops the render
network_policy.pod_security.version the policy version the labels pin, default latest

Ingress only: egress stays open. The cluster's CNI must enforce NetworkPolicy (a default minikube does not, so dev renders it without effect).

task test:network-policy

Pod Security restricted

Every workload the celine-<env> namespace receives in prod passes Pod Security restricted, by infra defaults: no privilege escalation and no capabilities on every container (the shells, policies-shell's init containers, postgres-db's Jobs, Keycloak and its dbchecker, Redis, the app broker, qdrant), the postgres-db Jobs as the postgis image's postgres user (uid 26), and qdrant without its root chown init container (updateVolumeFsOwnership: false: the pod's fsGroup gives the volume to its group). The seccomp profile is left to the cluster (GKE Autopilot sets RuntimeDefault). An environment enforces it with network_policy.pod_security.enforce: restricted. auth-setup's dashboard (nginx:alpine, root) does not pass: keep it off where restricted is enforced.

task test:pod-security

Images by digest

An environment pins images by digest in one map, by repository and tag:

Value Effect
image_digests.<repository>.<tag> sha256:<64 hex>. A container whose repository:tag is listed runs repository:tag@sha256:…; the app.kubernetes.io/version label keeps the tag. Keyed by tag: a tag bumped without its digest runs unpinned, never the old image under the new name. A value that is not a sha256 digest stops the render. Unset: images by tag
ingress_controller.image_tag, oauth2_proxy.image_tag name the Traefik / oauth2-proxy tag (otherwise the chart's), so the map can pin it
postgres_db.pgbouncer_image_tag names the pooler's pgbouncer tag (otherwise the CNPG operator's), so the map can pin it
postgres_db.pin_cluster_image true: the postgis Cluster's image is pinned too. Off by default because a changed imageName restarts Postgres; the postgres-db Jobs take the digest as soon as it is listed

Keycloak (ghcr.io/celine-eu/keycloak) and its dbchecker render as repository@digest (keycloakx's own value). Not pinnable this way: qdrant 1.17.0 (no digest value; a tag@digest tag breaks the chart's semverCompare), the CNPG operator and the pooler's init container (the operator image; changing it upgrades the instance manager). Redis is pinned by its chart. sync_tables.image is one reference: write tag@sha256:… there.

task test:image-digest

Redis

One Redis (release redis, group core) holds the oauth2-proxy sessions, and on environments that run them, Prefect's messaging and Superset's broker and cache.

Value Effect
redis.password secret (secrets.sops.yaml). Set: Redis requires it (requirepass); it lives only in the Secret redis-auth (key password), which oauth2-proxy reads as OAUTH2_PROXY_REDIS_PASSWORD. A changed password restarts both through a checksum/redis-auth annotation. Unset: Redis is open, as before. prod refuses to render without it
Database Client
0 oauth2-proxy (sessions); today also Prefect and Superset's Celery broker

Prefect and Superset do not read the password yet: an environment that runs them keeps redis.password unset until they do.

task test:redis

Sizing

Clusters that bill by resource request, or fill in a large default for pods that declare none, need every pod sized. These keys are optional; unset, the pods render as before.

Value Effect
postgres_db.resources the Postgres cluster's requests and limits. Default requests 2 CPU / 8Gi, limits 4 / 12Gi
postgres_db.parameters single Postgres parameters over the defaults. Move shared_buffers (default 2GB) and effective_cache_size (6GB) with the memory
postgres_db.pooler_resources each pgbouncer pod. Default none
postgres_db.job_resources the post-deploy Jobs (grants, table tuning). Default none
tls_setup.cert_manager.resources.<part> controller, webhook, cainjector, startupapicheck. Default none
auth_setup.dashboard_resources the dashboard Deployment. Default none
legal.resources the legal host. Default none
ingress_controller.resources see "Ingress controller"

The CELINE services default to requests 100m / 256Mi and limits 500m / 512Mi (defaults/0050-celine-services/values.yaml.gotmpl), overridden per service with <service>.resources.

Pod disruption

A cluster autoscaler that scales a node down deletes the pods on it; a one-replica service is then down until its new pod is ready (GKE Autopilot, prod 2026-10-23: webapp twice in 13 minutes, Traefik answering "503 no available server"). Opt-in per environment and per release, keyed by the release name; unset, nothing renders differently:

pod_disruption:
  webapp:
    pdb: true              # a PodDisruptionBudget over the release's pods
    max_unavailable: 0     # default 0; 1 lets a two-replica release go one pod at a time
    pod_annotations: {}    # extra pod annotations

The autoscaler does not remove a node whose pods a budget does not let it evict. A rollout is not an eviction, so RollingUpdate and Recreate releases roll as before; a drain waits (GKE forces a node upgrade after an hour). cluster-autoscaler.kubernetes.io/safe-to-evict: "false" in pod_annotations also keeps the pod, but on Autopilot makes it an extended-duration pod, raised to at least 500m CPU.

Read by every release built on the celine-services library except the operator shells (defaults/0050-celine-services/pod-disruption.yaml.gotmpl), and by keycloak (the chart's budget), oauth2-proxy and redis (a budget through extraObjects: oauth2-proxy's own renders only above one replica, redis has none). sync_tables.sink has its own pod_annotations / pdb. Tests: task test:pod-disruption.

Sync environment marker

sync_tables (celine-eu/demo3) copies datasets from dev to local and from dev to prod, and only when both databases carry a marker naming their environment. Here the postgres-db release writes it: with postgres_db.sync_environment set to dev or prod, a Job runs COMMENT ON DATABASE datasets IS 'celine:env=<value>' as the database owner on every deploy. There is no default. Unset, the database carries no marker and every sync against it is refused. Any other value stops the render.

Read-only role on datasets

The postgres-db release grants datasets_ro read access to the schemas in grants.schemas (defaults/0010-core/postgres-db/values.yaml.gotmpl): USAGE, SELECT on every table, and SELECT on tables created later. A Job runs it as the database owner on every deploy, in one transaction with a 5 s lock timeout; if a table is busy, the Job rolls back and retries. Schemas that are missing or not the owner's are skipped with a warning. So are tables another role owns, which also never get the future-tables default. The Job's log ends with a per-schema count of tables and of readable tables.

Mirror schemas for the sync sink

Where a database receives copied tables, the sink writes them as mirror_writer, a role that owns the mirror schemas and nothing else: no CREATE on the database, no other schema. With postgres_db.mirror.enabled: true the postgres-db release adds:

Key Meaning
postgres_db.mirror.enabled true or false (default). Anything else stops the render
postgres_db.mirror.schemas required when on: the schemas the sink writes, the same as sync_tables.sink.allow_schemas. Not public, information_schema or pg_*
postgres_db.mirror_writer required when on, secret: the role's password. It goes into a connection URI unescaped, so use a URL-safe value

It renders the role, its connection string Secret postgis1-mirror-writer-connection-string (the sink's dsn_secret), the database owner's membership in mirror_writer, and a Job that runs as the owner on every deploy. The Job creates each missing schema owned by mirror_writer (CREATE SCHEMA ... AUTHORIZATION is why the owner needs the membership), refuses a schema that already exists under another owner, and gives datasets_ro USAGE, SELECT on what is there and the default for what the sink creates later. Until CNPG has reconciled the new role and membership, the Job fails and retries. The read-only grants above skip these schemas with a warning, since the owner does not own them.

Table sync (sync_tables)

charts/sync-tables runs sync_tables' exchange layer, with two roles enabled per environment (sync_tables.sink.enabled, sync_tables.push_source.enabled; the release is installed when either is on). The keys are listed in defaults/0050-celine-services/sync-tables/values.yaml.gotmpl.

  • Sink: a Deployment holding a database's write DSN (from an existing Secret), writing only the schemas in allow_schemas. It is reached on one route, sync.<domain>/sync/v1, through the ingress with mutual TLS: the ingress serves a certificate signed by a CA used only for this sync and accepts only client certificates from that CA. Its NetworkPolicy admits the ingress controller's namespace and lets it reach DNS and the database.
  • Push source: a CronJob that reads its database and dials a sink (wss:// only), trusting the sync CA alone. A run is not retried: the next one resumes. Its profile (the table list) comes from the environment as YAML text.

The sink refuses to start on a database without its environment marker (see "Sync environment marker"): it exits and restarts until the postgres-db release has written it. Its /healthz shows the process is up, not the database. The push source reads its config.yaml (parallelism, batch size) from the image's /app, so nothing is mounted there.

The CA, certificates and token come from sync-tables certs. The image runs as uid 10001, under the restricted Pod Security profile.

Security headers

Outside dev, every public host - the frontends, the legal host and the API gateway - answers with security response headers set by ingress-nginx, not by the applications. Each release renders a ConfigMap <release>-security-headers and names it on its Ingresses with nginx.ingress.kubernetes.io/custom-headers. This is not a snippet annotation, so the controller's default allow-snippet-annotations: false and annotations-risk-level: High stay as they are.

Header Value
X-Content-Type-Options nosniff
Referrer-Policy strict-origin-when-cross-origin; no-referrer on the API gateway
X-Frame-Options DENY
Permissions-Policy no camera, microphone, geolocation, payment or USB; the onboarding UI keeps the camera for document capture
Strict-Transport-Security max-age=31536000; includeSubDomains, where the Ingress terminates TLS (ingress.tls.enabled; the gateway's tls.enabled)
Content-Security-Policy per host, in the chart's securityHeaders.csp; origins of the platform come from domain (sso.<domain>, keycloak.<domain>). Not on the SvelteKit frontends, which send their own (below)

An Ingress whose backend sets its own headers carries no annotation, because the controller would replace them: the assistant API (on the gateway, on the assistant host and under /api/assistant of the webapp host), the community API and the onboarding API.

The SvelteKit frontends send their own Content-Security-Policy (kit.csp in each app's svelte.config.js, celine-frontend and onboarding ui/): every page is rendered on request and its inline scripts - SvelteKit's bootstrap, the theme script - carry a fresh nonce, so no host allows inline scripts. A static ConfigMap cannot carry a per-request nonce, and the controller replaces a header of the same name, so these charts set csp.sentByApp: true and their ConfigMap holds no enforced policy; the render stops if an environment gives them one. Grid keeps its Report-Only trial of the other directives here, without script-src or default-src. An API Ingress of its own on such a host keeps an enforced policy from the controller (securityHeaders.apiCsp, ConfigMap <release>-api-security-headers): roi's calculator API allows nothing (default-src 'none'). Paths that share the page's Ingress - /oauth2 on every frontend host, /api on webapp and grid - get the other headers but no policy. Deploy the frontend images that send the policy before this chart change, or the hosts are briefly without one.

Host Policy Allows besides the host itself
webapp, assistant, community enforced, sent by the app scripts by nonce, inline styles, data:/blob: images; community: OpenStreetMap tiles
onboarding enforced, sent by the app scripts by nonce, inline styles, data:/blob: images
grid scripts enforced by the app (nonce, a blob: worker); the rest Report-Only here inline styles, the CARTO and ArcGIS map hosts
roi enforced, sent by the app scripts by nonce, inline styles, data:/blob: images, the a/b/c OpenStreetMap tile hosts, the static map image, Nominatim (no wildcard)
legal enforced inline styles only; no script
api gateway enforced nothing (default-src 'none')

The Keycloak host gets only X-Content-Type-Options: nosniff (ConfigMap keycloak-security-headers from the auth-setup release, named by the keycloak release's custom-headers): Keycloak sends its own policy, X-Frame-Options and Referrer-Policy, which a header of the same name would replace. It also reaches the controller's 404 on the paths outside the public rules, / included.

Before the first deployment with these headers, the controller must allow the header names. It refuses any other with a 503 on every path of the Ingress that names it. In the ingress-nginx ConfigMap (Helm chart: controller.config):

global-allowed-response-headers: "X-Content-Type-Options,Referrer-Policy,X-Frame-Options,Permissions-Policy,Strict-Transport-Security,Content-Security-Policy,Content-Security-Policy-Report-Only"
Value Effect
security_headers.enabled true unless set. false renders neither ConfigMap nor annotation; only the environment dev may set it, elsewhere it stops the render
security_headers.csp.<release>.report_only true sends Content-Security-Policy-Report-Only for that release's host instead
security_headers.csp.<release>.directives merged over the chart's directives; null drops one. Values may use {{ .Values.ingress.domain }} and {{ .Values.ingress.host }}

<release> is the helmfile release: frontend-webapp, frontend-assistant, frontend-community, frontend-grid, frontend-roi, frontend-onboarding, legal, api-gateway. A Report-Only host is promoted by setting report_only: false once the browser console shows no report on its pages. task test:security-headers renders all of it.

The deployment's legal documents (privacy notices, terms of use, data-sharing notices), from the generic image ghcr.io/celine-eu/celine-legal (github.com/celine-eu/celine-legal) and the deployment's own data. Release legal (charts/celine-legal), at legal.<domain>, public: legal documents are read before anyone signs in, so there is no oauth2-proxy. It is off until the environment sets legal.enabled: true and legal.image_tag.

One host, two sites that never meet:

  • the root serves the published releases only. Applications link it and record acceptances against it; a document with nothing published answers "not yet available" and is not offered to them;
  • /draft/<revision>/ (with legal.draft) serves the data as it is now, for the parties to review, every page marked not valid. The revision is the commit of the deployment's data; the link is the access. /draft/<alias>/ redirects to the current draft, so reviewers keep one link across commits: legal.draft_alias, default the image's latest, which anyone can guess. Set your own where the draft is private.

One container, non-root, read-only root filesystem, probed on /healthz. The data is a ConfigMap mounted at /data; a change to it restarts the pod.

Value Effect
legal.enabled installs the release
legal.image_tag required when enabled: a released celine-legal version, vX.Y.Z
legal.draft serves the draft at /draft/<revision>/
legal.draft_alias the name /draft/<alias>/ redirects from to the current draft (celine-legal v0.3.0+; an older image ignores it). Empty: the image's default, latest
legal.release pins the stable site to vX.Y.Z; the pod refuses a release that is not the newest published
legal.features.<name> true/false: overrides a derived feature (below)
legal.host defaults to legal.<domain>
legal.base_url where applications find the legal host: defaults to https://<legal.host> when the release is installed. Set it alone to use a legal host served elsewhere. Set it to "" while nothing is published: the applications keep their own documents, and the host serves the draft only
legal.applications_read_draft every application below reads the draft, https://<legal.host>/draft/<legal.draft_revision>, instead of the base URL: for an environment that reviews the texts in the applications, before any release. Needs legal.enabled and legal.draft. Acceptances are then given on draft versions
legal.draft_revision the draft's 40-hex commit, the same one the release's environment file gives the host; required by applications_read_draft. The address changes with it: after a change to the data, apply legal first, then the applications

The data comes from the release's environment file, envs/<env>/0050-celine-services/celine-legal/values.yaml.gotmpl, which belongs to the deployment: chart values data.files (path under /data → text: register.yaml, parties.yaml, the identities, sharing-offers.yaml, editions/…) and draft.revision. The chart refuses to render without register.yaml, or with a draft and no 40-hex revision.

The features the texts describe are derived from the applications' own switches, so a text never describes what the environment does not run:

Feature On when
sms_verification onboarding.sms_provider is a dev provider (log, console, dev), or brevo with onboarding.dpa_sms_signed
document_extraction onboarding.env sets EXTRACTION_ENABLED true, LLM_BASE_URL and LLM_VISION_MODEL
dso_data_exchange onboarding.dataspace_enabled
ai_assistant always (the assistant is released with the services)

All but ai_assistant are off where onboarding is not installed.

With a base URL, the applications use the community's own documents:

  • webapp and onboarding get LEGAL_BASE_URL. Community links the registry leaves empty, the terms gate and the onboarding consent documents resolve there;
  • frontend-assistant gets PUBLIC_LEGAL_BASE_URL, for the AI notice's privacy link;
  • Keycloak gets TERMS_URL and PRIVACY_URL (the host's slot pages, which name no community unless the request does), for the login and email footer.

Without one, every application behaves as before.

The import checks each boundary id with the Digital Twin, so the Digital Twin must be up.

Keycloak realm bootstrap

The realm's platform level (Organizations, sign-in settings, languages, themes, lifespans, brute force, smtpServer, the realm role platform-admin and who holds it) is written by celine-policies keycloak bootstrap, never by the realm import. It runs in the policies-shell pod's init containers: plan (with a realm export first), apply, then a check that must find nothing left to change. Only then does the shell run keycloak sync. sync-orgs and sync-users refuse a realm that has not been through both, and sync-users refuses outright unless ENV is exactly dev: a deployed realm gets its users from onboarding, not from a YAML.

Both reach Keycloak through its Service (keycloak.internal_url), never the public host, which serves neither /admin nor /realms/master (see Keycloak admin access). Outside dev, bootstrap also hardens the master realm (brute force, a second factor for its admins), and does it only as its own master-realm client svc-celine-policies-bootstrap. Its first run creates that client with the master admin (keycloak.username/password); every later run, and sync, signs in as the client, because the master admin's password grant stops working once the admin has a second factor. The procedure (first console sign-in, secret rotation, recovery) is in celine-policies docs/deployment.md, "The admin second factor and the master realm".

The run's export, plan, apply and check output is stored under <bucket>/<environment>/<UTC time>-<policies_shell.image_tag>/. Pin policies_shell.image_tag to a release: the tag is the version of platform.yaml.

Value Effect
policies_shell.bootstrap.bucket, policies_shell.bootstrap.s3_secret where the run is stored; a Secret with S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_ENDPOINT_URL. Unset: logs only
policies_shell.bootstrap.allow_destructive passes --allow-destructive, for a plan that turns a setting off or removes a list entry. Set for one release only
policies_shell.env CELINE_KEYCLOAK_ENV for bootstrap and sync. Unset: dev in the helmfile environment dev, prod in any other. Anything but dev is production to celine-policies
policies_shell.bootstrap.client_secret CELINE_KEYCLOAK_BOOTSTRAP_CLIENT_SECRET, through a Secret, to the bootstrap init container and the shell. Required outside dev, at least 32 characters (openssl rand -hex 32): the render stops without it. The client holds master's admin role, so treat the secret as the master credential. Rotate it in the console first (master > Clients > svc-celine-policies-bootstrap > Credentials), then here
policies_shell.bootstrap.client_id CELINE_KEYCLOAK_BOOTSTRAP_CLIENT_ID. Unset: svc-celine-policies-bootstrap
keycloak.internal_url the Keycloak Service bootstrap, sync and provisioning use. Default http://keycloak-keycloakx-http (same namespace)
keycloak.brute_force.enabled CELINE_KEYCLOAK_BRUTE_FORCE_ENABLED, only when set. Unset: brute-force protection is on outside dev, on the platform realm and on master; false turns it off on both. The realm import reads keycloak.brute_force.failureFactor (max login failures, default 5); there is no maxLoginFailures in Keycloak, and an import naming one stops Keycloak from starting
keycloak.admin_mfa CELINE_KEYCLOAK_ADMIN_MFA_REQUIRED, only when set: true/false, or {enabled: true\|false}; anything else stops the render. Unset: on outside dev. bootstrap owns the flow (the realm import has none): holders of the realm role platform-admin, and master's admins, who sign in with a password need TOTP or a recovery code, and one with neither enrols both; a passkey sign-in needs nothing more. Turning it off on a realm that has it needs allow_destructive. task test:keycloak-access checks the render
keycloak.platform an overlay on platform.yaml: realm_settings.supportedLocales, and platform_admin.users, the usernames that hold the realm role platform-admin besides the operator realm admin
the resolved smtp block (see Email (SMTP)) CELINE_KEYCLOAK_SMTP_*; user and password through a Secret
auth_setup.realmAdminUser, realmAdminEmail, realmAdminPassword CELINE_KEYCLOAK_REALM_ADMIN_*: the operator realm admin, created once (password through a Secret) and granted the realm role platform-admin
auth_setup.clientSecret, domain OAUTH2_PROXY_CLIENT_SECRET, CELINE_DOMAIN: sync owns the oauth2_proxy client, its secret and its redirect URIs (sso, superset, webapp, assistant on the domain)

Who is a platform admin. Exactly two levels (celine-policies ADR-0012). The realm role platform-admin, in realm_access.roles, is the only platform-wide grant. An organisation's own groups (admins > managers > editors > viewers) reach a token only inside organization.<alias>.groups and count only in that organisation. There are no realm groups and no realm roles admin/manager/editor/viewer, and no top-level groups claim. The Prefect and Marquez ingresses admit allowed_groups=role:platform-admin only (oauth2-proxy's keycloak-oidc provider names realm roles role:<name>, client roles role:<client>:<name>). auth_setup.adminRole names a different role for both gates, <client>:<role> for a client role; an empty value, a list or anything else stops the render. The MQTT broker has no superuser: mqtt-auth's /superuser never grants one.

task keycloak:bootstrap:check:<env> reports drift (exit 1). Do not run bootstrap by hand while the pod is starting. Rolling back is manual: put the platform-level keys back from the stored export.json, then pin the previous image tag so the next start does not re-apply.

Keycloak admin access

The public host keycloak.<domain> serves the platform realm and nothing else. Its Ingress lists two paths, /realms/<keycloak.realm> and /resources, so sign-in, tokens, discovery, JWKS, logout and the account console work, and everything else gets the ingress controller's 404: /admin (the admin console and the admin REST API), /realms/master, any other realm, /. These are plain path rules: no snippet annotation, which ingress-nginx refuses by default. A gateway firewall or WAF in front does not change this.

Value Effect
keycloak.public_realms the realms the public host serves. Default: [<keycloak.realm>]. master stops the render
keycloak.admin_forward_port the localhost port of the admin console, KC_HOSTNAME_ADMIN=http://localhost:<port>. Default 18080
keycloak.admin_public {enabled, allow_source_ranges}: the admin console on the public host, from an allowlist, outside prod only. Default off. See below

There is no admin ingress. The master realm and the admin console are reached only through kubectl port-forward to the Keycloak Service, from a machine with cluster credentials:

task keycloak:forward                       # current kubectl context and namespace
task keycloak:forward NAMESPACE=celine-dev  # or name the namespace
# -> Admin console: http://localhost:18080/admin/master/console/   (Ctrl-C to stop)

The task reads the port from the deployed KC_HOSTNAME_ADMIN; the console works only on that exact URL. localhost is a secure context, so plain http needs no certificate. KC_HOSTNAME stays the public URL: every realm's issuer, and every token minted inside the cluster, is unchanged.

Master's Frontend URL is set on every deploy. The console signs in to master through master's own URL, and KC_HOSTNAME_ADMIN does not change that (measured on Keycloak 26.6.0 and 26.7.3: without it the console calls the public host for master and never shows a login). So master's Frontend URL (its realm attribute frontendUrl) must be the same localhost URL. The policies-shell pod's bootstrap init container sets it on every start, after keycloak bootstrap:

  • the URL is http://localhost:<keycloak.admin_forward_port>, derived like KC_HOSTNAME_ADMIN, so the two cannot disagree (chart value bootstrap.masterFrontendUrl; empty leaves master alone);
  • it signs in as the bootstrap client through keycloak.internal_url, reads master, merges frontendUrl into the attributes it read (a PUT replaces the whole map) and writes only when the value differs, then checks master's issuer. It never prints the secret or the token;
  • without policies_shell.bootstrap.client_secret (allowed in dev only) it logs that it left master alone;
  • a failure fails the pod, like any other bootstrap step. Its output is in the init container's log (master frontendUrl: ...) and in the stored record as frontend-url.txt.

task keycloak:forward warns while master's issuer is not the forwarded URL. The first fix is to run the step again: kubectl rollout restart deployment/policies-shell, then kubectl logs deployment/policies-shell -c bootstrap | grep 'master frontendUrl'. If the shell cannot run, set it by hand with the forward running, as the bootstrap client (its secret is policies_shell.bootstrap.client_secret; read -rs keeps it out of the shell history; needs jq):

U=http://localhost:18080
read -rs S
T=$(curl -s -d grant_type=client_credentials -d client_id=svc-celine-policies-bootstrap \
      --data-urlencode client_secret="$S" "$U/realms/master/protocol/openid-connect/token" \
    | jq -r .access_token)
# merge into master's attributes: a PUT with frontendUrl alone would drop the others
curl -s -H "Authorization: Bearer $T" "$U/admin/realms/master" \
  | jq --arg u "$U" '{attributes: ((.attributes // {}) + {frontendUrl: $u})}' \
  | curl -s -X PUT -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
      -d @- "$U/admin/realms/master"
unset S T

Master's tokens then carry the issuer http://localhost:18080/realms/master; celine-policies checks only that it ends in /realms/master. The first console sign-in of a master admin outside dev enrols TOTP and recovery codes.

Outside prod, the console can move to the public host (keycloak.admin_public), for a browser impersonation that must start on the host whose cookies the platform sees:

keycloak:
  admin_public:
    enabled: true
    allow_source_ranges: [192.168.1.47/32]   # the addresses the ingress controller sees
  • a second Ingress, the chart's keycloak-keycloakx-console, serves /admin and /realms/master on keycloak.<domain> with nginx.ingress.kubernetes.io/whitelist-source-range; any other source gets 403. The public realm's Ingress is unchanged;
  • KC_HOSTNAME_ADMIN and master's Frontend URL become https://keycloak.<domain>, so the port-forward no longer serves the console; task keycloak:forward prints the public URL instead;
  • the render stops in prod (environment or celine_env), without a non-empty list of CIDRs, on a range that admits everything (/0), with an ingress class other than nginx (the annotation is ingress-nginx's), and with keycloak.admin_mfa off (or unset in dev);
  • the allowlist filters the address the controller sees. Behind a proxy or a NAT that is the proxy's address, not the client's: measure it in the controller's access log first.

Inside the cluster nothing administers Keycloak through the public host: the policies-shell bootstrap and sync, and provisioning, use keycloak.internal_url. Services keep validating tokens against the public realm URL, which the public host still serves.

task test:keycloak-access renders all of it. On a throwaway minikube with ingress-nginx v1.14.3, Keycloak 26.7.3 and the rendered Ingress (2026-10-04): - the public host answered 404 to /admin/*, /realms/master/*, / and other realms, .. and %2f variants included; - sign-in and the account console of the platform realm worked; - the console signed in through task keycloak:forward; - bootstrap's calls through the Service worked.

Add-on layer

A project that adds services next to CELINE (a dataspace, for instance) can switch on the pieces infra needs for it without editing infra or the environment's values.yaml: every group in helmfile.d/ also loads envs/<env>/addons.yaml.gotmpl, after values.yaml and before the secrets file, when that file exists.

  • Absent, nothing changes: the hook is an isFile test in each group's environments block, so an environment without the file renders exactly as before.
  • Present, it must render: it is a helmfile values template, so it can be the layer itself or a readFile of a file another checkout owns. A readFile whose target is missing stops every group. A relative readFile path is joined to infra/envs/<env>/ and cleaned as text, never through a symlinked environment directory: three ../ are the directory holding infra/, so from an envs/staging/ linked in from a deployment repository that has infra as a submodule, a checkout beside that repository is ../../../../<checkout>/…. An absolute path avoids the question.
  • envs/*/addons.yaml.gotmpl is gitignored: the file is written per machine, by the project that owns the layer.

Today the layer carries one feature, dataspace:. Everything is off unless dataspace.enabled is true; dataspace.enabled must be a boolean, and once it is on every input below that is marked required stops the render when missing.

dataspace:
  enabled: false                       # true or false; anything else stops the render
  databases: []                        # postgres-db: role and database names (required list)
  onboarding:
    enabled: true                      # the onboarding part; set false to keep it off
    identity_registry_url: ""          # required: IDENTITY_REGISTRY_URL
    connector_url: ""                  # DS_CONNECTOR_URL when set
    collector_aliases: []              # SVC_DS_COLLECTOR_<ALIAS>_SECRET each
    template_patches: {}               # <slug>/<path> -> map, deep-merged into the template
  dataset_api:
    extra_env: []                      # env entries appended to dataset-api's env
    client_id: ""                      # the realm client dataset-api calls out as, when set
  policies_shell:
    overlays: []                       # required: [{name, content}], realm declarations
    clients: []                        # required list: realm client ids needing a secret
  client_secrets: {}                   # secret: realm client id -> secret
onboarding:
  rec_assertion_hmac_key: ""           # secret: required once the dataspace reaches onboarding
Value Effect, with dataspace.enabled: true
dataspace.databases each name becomes a role and a database of that name in postgres-db, shaped like every other role (Secret <name>-db-secret with cnpg.io/reload, a managed role, a Database CR). The password is postgres_db.<name> in the secrets file, required. The Cluster's parameters are untouched
dataspace.onboarding.* onboarding runs with DATASPACE_ENABLED=true, whatever onboarding.dataspace_enabled says, unless dataspace.onboarding.enabled is false. While onboarding is enabled: identity_registry_url is required; client_secrets.svc-ds-onboarding is required (DS_ONBOARDING_CLIENT_SECRET); each alias in collector_aliases needs client_secrets.svc-ds-collector-<alias> (required), exported as SVC_DS_COLLECTOR_<ALIAS>_SECRET (upper-cased, - as _); onboarding.rec_assertion_hmac_key is required (REC_ASSERTION_HMAC_KEY)
dataspace.onboarding.template_patches each <slug>/<path> is deep-merged into onboarding.templates.files[<path>]: maps merge key by key, the patch wins, lists are replaced. A path that is not in templates.files stops the render. A patched template is written back as YAML, so its comments and key order are not kept
dataspace.dataset_api.extra_env appended after the dataset-api chart's own env (chart value extraEnvAppend). Each entry needs a name and a value or valueFrom
dataspace.dataset_api.client_id optional: the realm client dataset-api calls other services as (oidc.service.clientId), its secret client_secrets.<client_id> required. The audience it accepts does not change. Unset: svc-dataset-api as before
dataspace.policies_shell.overlays required, a list of {name, content}: mounted into policies-shell and passed to its start-up celine-policies keycloak sync as --overlay, in list order, after clients.yaml (chart value syncOverlays; celine-policies 1.7.0 or later). A name given twice stops the render. A changed overlay restarts the shell. Never --prune
dataspace.policies_shell.clients required list: each realm client id gets its secret as SVC_<ID>_SECRET in policies-shell (the id upper-cased, - as _: svc-ds-portal gives SVC_DS_PORTAL_SECRET) from client_secrets.<id>, required. A name the shell already sets stops the render

Removing the file, or setting dataspace.enabled: false, renders every environment exactly as without the layer.

Applying Infrastructure (Helmfile)

From the infra/ directory:

Apply an environment

helmfile -e dev apply

Diff changes before applying

helmfile -e dev diff

Destroy an environment

helmfile -e dev destroy

Apply a single release

helmfile -e dev apply --selector name=<release-name>

Operational Guidelines

  • Do not commit plaintext secrets
  • Encrypt secrets before apply
  • Prefer helmfile diff before apply
  • Avoid manual helm install
  • Keep environment changes isolated
  • Production environments require additional safeguards

Intended Audience

This repository is intended for: - Infrastructure engineers - Platform operators - CI/CD automation

It is not intended as a general developer quickstart.

Local Charts

Charts under charts/ are deployed from the working tree by helmfile.d/. defaults/ is the integration point between envs/**/values.yaml and each chart's values.yaml, so environment variables stay simple and are reused across charts.

Shared

  • celine-services — library chart (type: library) providing the shared deployment, service, ingress, secret and env templates every celine-* service chart includes
  • api-gateway — exposes all CELINE APIs under a single ingress, api.domain.tld/<service>

Services

  • celine-dataset-api — Dataset API
  • celine-dataset-api-shell — Dataset API CLI to manage datasets
  • celine-digital-twin — Digital Twin API
  • celine-flexibility-api — Flexibility API
  • celine-mqtt-auth — mosquitto-go-auth compatible API endpoint for MQTT auth/ACL
  • celine-nudging — Nudging API
  • celine-policies-shell — Policies CLI to manage Keycloak. On every start, init containers run keycloak bootstrap (the realm's platform level, from platform.yaml in the image) and store the run, then the shell runs keycloak sync; a failure of either fails the pod. See Keycloak realm bootstrap
  • celine-rec-registry — REC Registry API
  • celine-rec-registry-shell — REC Registry CLI to manage REC organizations and asset metadata
  • celine-ai-assistant — AI Assistant API
  • celine-roi — ROI API
  • celine-webapp — Participant webapp API
  • celine-onboarding — REC onboarding API (member wizard and operator console). See Onboarding
  • celine-grid — Grid resilience API

Frontends

  • celine-frontend-assistant — AI Assistant webapp
  • celine-frontend-roi — ROI webapp
  • celine-frontend-webapp — Participant webapp
  • celine-frontend-grid — Grid webapp
  • celine-frontend-onboarding — REC onboarding UI, and the host the onboarding API answers on
  • celine-legal — the legal host: the deployment's legal documents, public and scoped per community (API + UI). See Legal host

The frontends, the legal host and the API gateway carry the security headers.

Platform

  • auth-setup — secrets and configmaps for oauth2-proxy and keycloak
  • mqtt-setup — MQTT access for services
  • mqtt-ingestor — ingests every MQTT message on the configured topics into a database
  • marquez — Marquez OpenLineage endpoints and UI
  • mosquitto-go-auth — mosquitto with the mosquitto-go-auth module
  • pg-freezer — cold storage service mirroring table records to minio/s3 as parquet, then cleaning the tables
  • postgres-db — CNPG-specific configuration for the database/user maps
  • prefect-pipelines — collects and deploys the CELINE data pipelines
  • registry-accounts — docker/ghcr.io secrets for image pulling
  • s3-accounts — minio/s3 access credentials
  • tls-setup — TLS for local (self-signed) and production (Let's Encrypt) environments
  • CELINE pipelines: https://github.com/celine-eu/celine-pipelines
  • CELINE project: https://celineproject.eu/
  • CELINE docs: https://celine-eu.github.io/