celine-policies

Authentication, authorization, and identity management for the CELINE platform.

This repository provides three services:

mqtt_auth — A FastAPI HTTP backend for mosquitto-go-auth that validates JWTs and evaluates OPA (Rego) policies to control MQTT topic access. Outside CELINE_ENV=dev it requires CELINE_OIDC_AUDIENCE and an explicit issuer/JWKS (REQ-0009).
provisioning — A FastAPI service that is the only writer of participant accounts in the celine realm: it ensures an account, its REC organization and its org group, and it sweeps a community from the registry. It holds realm-wide Keycloak administration and therefore has no public route — see ADR-0007.
celine-policies CLI — A typer-based CLI that performs idempotent synchronization of OAuth scopes, service clients, users, and organizations into Keycloak.

Each CLI command owns one level of the realm, and the line is who drives the state, not how often it changes:

Level	Written by	Declared in
platform: realm features (Organizations, fine-grained admin permissions), sign-in settings, languages, themes, token lifespans, brute force, `smtpServer`, the realm role `platform-admin` and its holders, the retired realm groups and roles (ADR-0012)	`keycloak bootstrap`	`platform.yaml`, `CELINE_KEYCLOAK_PLATFORM_` overrides, `CELINE_KEYCLOAK_BRUTE_FORCE_ENABLED`, `CELINE_KEYCLOAK_ADMIN_MFA_REQUIRED`, `CELINE_KEYCLOAK_SMTP_`; the master realm through `CELINE_KEYCLOAK_BOOTSTRAP_CLIENT_SECRET` (ADR-0014)
clients: clients, scopes, scope bindings, audience mappers, realm claim scopes, service-account grants	`keycloak sync`	`clients.yaml`
organizations and users: REC organizations, org groups, participants, memberships	the provisioning service (`POST /reconcile/{community}` creates a REC's organization, org roles and groups, even with no members; `PUT /participants` files a participant, `PATCH /participants` corrects their names or address), `keycloak set-user-organization` (a manager's `managers` or `admins` group, run by a platform admin), `keycloak sync-orgs`; `keycloak sync-users` in local development only — see ADR-0010; on an intermediate (non-prod) realm, mirrors of prod's members by `keycloak mirror-members` — see ADR-0015	the registry, owners and REC YAML

A command never writes a level it does not own. It checks the levels it depends on and refuses, naming the command to run. So the order is bootstrap, then sync, then the organization and user commands, and re-running all three is how to answer "is this realm right?".

It also ships a custom Keycloak Docker image with the rec login and email themes (see keycloak/README.md).

Quick Start

# Install dependencies
uv sync

# Bootstrap Keycloak admin client and sync scopes/clients
task keycloak:bootstrap
task keycloak:sync

# Start the full stack (Keycloak, MQTT auth, Mosquitto, Redis, oauth2-proxy)
docker compose up -d

# Verify MQTT auth health
curl http://localhost:8009/health

Documentation

Document	Description
Getting Started	Setup, CLI commands, and first sync
Architecture	System design and component overview
API Reference	MQTT auth HTTP endpoints
Scopes & Permissions	OAuth scopes and client configuration
MQTT Integration	Topic patterns, ACL policies, broker config
Deployment	Docker Compose stack and configuration

Project Structure

celine-policies/
├── src/celine/
│   ├── mqtt_auth/          # FastAPI MQTT auth service
│   │   ├── main.py         # App factory (create_app)
│   │   ├── routes.py       # /user, /acl, /superuser (always 403) endpoints
│   │   ├── models.py       # Pydantic request/response models
│   │   └── config.py       # MqttAuthSettings (pydantic-settings)
│   ├── provisioning/       # The only writer of participant accounts
│   │   ├── provisioner.py  # The Admin API sequence; no typer, no FastAPI
│   │   ├── bundle.py       # Reading a REC bundle (file or registry: same parser)
│   │   ├── registry.py     # GET /admin/export on the live rec-registry
│   │   ├── service.py      # What each route does, with no FastAPI in it
│   │   ├── routes.py       # PUT/PATCH /participants, invitation, disable, reconcile
│   │   ├── invitation.py   # Who may be emailed, which actions, shared with sync-users
│   │   ├── api_models.py   # Wire models; the one place keycloak_id becomes user_id
│   │   ├── config.py       # ProvisioningSettings
│   │   └── main.py         # App factory (create_app)
│   └── policies/cli/       # celine-policies CLI
│       ├── main.py          # Typer entrypoint
│       └── keycloak/        # Keycloak management commands
│           ├── commands/    # bootstrap, sync, sync-users, sync-orgs, etc.
│           ├── client.py    # KeycloakAdminClient (async httpx)
│           ├── models.py    # Config models for clients.yaml
│           ├── settings.py  # KeycloakSettings, SyncUsersSettings
│           └── sync.py      # Sync plan computation and application
├── policies/celine/        # Rego policy files
│   ├── mqtt/acl.rego       # MQTT topic ACL rules
│   └── scopes.rego         # Shared scope/group helpers
├── clients.yaml            # Platform scopes and service client definitions
├── keycloak/               # Custom Keycloak image + rec login and email themes
├── config/
│   ├── keycloak/import/    # Realm import JSON
│   ├── mosquitto/          # mosquitto.conf
│   └── oauth2-proxy/       # oauth2-proxy.cfg
├── tests/                  # Pytest test suite
├── docker-compose.yaml     # Full development stack
├── Dockerfile              # MQTT auth service image
├── taskfile.yaml           # Task runner commands
└── pyproject.toml          # Package definition (uv + hatchling)

CLI Commands

celine-policies keycloak bootstrap       # Converge platform.yaml; create the admin-cli service account
celine-policies keycloak sync            # Sync clients.yaml scopes/clients to Keycloak
celine-policies keycloak sync-users      # Import users from REC registry YAML (ENV=dev only; see ADR-0010)
celine-policies keycloak sync-orgs       # Import organizations from owners YAML
celine-policies keycloak set-password    # Set a user's password (development realms only)
celine-policies keycloak seed-dev-users  # Create admin, manager, editor, viewer (development realms only)
celine-policies keycloak set-user-organization  # Assign user to org + groups
celine-policies keycloak mirror-members  # Mirror prod's members into an intermediate realm (never prod; ADR-0015)
celine-policies keycloak status          # Show current Keycloak state

`ENV` — production safety

Every client secret in clients.yaml is written as ${SVC_X_SECRET:-svc-x}, so an unset variable resolves the secret to the client id itself. That is the point locally; against a real realm it installs a credential anyone can derive from the client list, and nothing downstream ever complains.

keycloak sync therefore defaults to ENV=prod and refuses to run when any client would receive such a placeholder, naming each offending client and the variable that fixes it. It fails before authenticating, so a misconfigured deployment stops on its own machine rather than halfway through a live realm.

ENV=dev celine-policies keycloak sync    # accept the clients.yaml fallbacks
SVC_DATASET_SECRET=... celine-policies keycloak sync   # or supply real secrets

taskfile.yaml exports ENV=dev for the whole file, so task keycloak:sync and friends behave as before. Resolution order is CELINE_KEYCLOAK_ENV, CELINE_ENV, then plain ENV; only dev disables the check (the platform rule, celine.sdk.posture), and anything else — test, local, ci, development, a typo or nothing at all — is production. Declaring no secret: at all is always accepted: Keycloak then generates one, which is the recommended production shape.

One realm, more than one file

sync recomputes the grants of every client present in the file it is given. A file that describes only part of a realm therefore does not leave the rest alone: an absent client is an orphan and survives without --prune, but a client that stays is narrowed to whatever grants that file declares — silently, with nothing deleted and no flag involved.

So when a realm is declared by more than one party, pass every file and let the loader merge them:

celine-policies keycloak sync clients.yaml \
    --overlay clients.ds-host.yaml \
    --overlay /path/to/ds/clients.yaml

--overlay is repeatable. Merging happens before anything else, so the placeholder-secret guard and the scope-reference check see the whole realm rather than one file's view of it.

The rule is ownership, not precedence — no file is subordinate and there is no last-wins:


a client's identity	declared by exactly one file: `name`, `description`, `secret`, `scopes_prefix`, `service_account_enabled`, `hardcoded_claims`
a client's grants	added by any file, with `client_id` plus `default_scopes` / `optional_scopes` / `extra_audiences` and nothing else
a scope	declared once, or identically more than once; a conflicting redefinition is an error
`realm`, `oauth2_proxy_client`	stated by any file; two files disagreeing is an error
a `scopes_prefix`	claimed by one client only — it decides where every audience mapper for those scopes points

Adding without removing: `--additive`

A run that is not handed the complete file set narrows the clients it declares. Where that risk cannot be ruled out — an init container that runs sync on every pod start, with overlays that may be missing or stale — pass --additive:

celine-policies keycloak sync clients.yaml --overlay clients.ds-host.yaml --additive

It applies every create and update and no removal. Each removal a full run would have made is held back and listed under Held back by --additive, marked =, in the plan and in the result:

a full run would	under `--additive`
remove a scope assignment no file grants	kept
move a scope between default and optional	both halves kept; reported as one move
delete a stale `aud-` mapper (the oauth2-proxy client's included) or `claim-` mapper	kept
revoke an admin permission for a group no longer named	kept
drop a scope from an admin permission	the scope is kept; scopes the file adds are still granted
switch off a login flow (or `publicClient`) on a client that declares no `browser` block, while updating it	the flow is left on
take a sign-out address off a browser client's `post.logout.redirect.uris` that neither its declared list nor, through `+`, its redirect URIs cover	the address is kept, after the declared list
take `organization`, `groups`, `dataspace` off the realm default/optional scope lists	left on the lists; the scopes are still created and assigned to the oauth2-proxy client

A client's declared browser block is an update, not a removal: its redirectUris and webOrigins are written as declared, so a URI dropped from the file is dropped from the client under --additive too.

browser.post_logout_redirect_uris (default ["+"]) is written as the client attribute post.logout.redirect.uris, which is what Keycloak checks a logout's post_logout_redirect_uri against; + means the redirect URIs, so a frontend added to redirect_uris can sign out too. The list is compared as a set, so a converged client is not rewritten. A hand-set list the redirect URIs already cover becomes + under --additive with nothing held; an address beyond them is kept and listed, and once kept the next additive run writes nothing.

--additive --dry-run lists the same held-back items without writing anything. --additive cannot be combined with --prune, and the pair exits 2 before connecting.

Where the command cannot be handed an argument, set the environment instead:

input	effect
`CELINE_KEYCLOAK_SYNC_ADDITIVE=true` (or `1`, `yes`, `on`)	the run is additive
`CELINE_KEYCLOAK_SYNC_ADDITIVE=false` (or `0`, `no`, `off`), empty, or unset	the run is not additive (the default)
`--additive`	additive, whatever the variable says
`--no-additive`	not additive, whatever the variable says

The flag overrides the variable either way. --prune is refused with an additive run from either source (exit 2); --no-additive --prune is allowed. An additive run says where it came from in its banner (Additive: removals are held back (from …)). Any other value of the variable is an error before the command starts.

The trade is real: a grant dropped from the files is not taken away until a run without the flag. A deployment that only ever runs additive syncs never narrows a client, including when it means to. Run a plain sync (after a --dry-run) to apply a declared removal.

Which files this repository ships

file	declares	mounted
`clients.yaml`	celine's own services	always — it is a whole realm on its own
`clients.ds-host.yaml`	the grants celine adds to the dataspace's clients	only where a dataspace is deployed, and only with ds's file

The dataspace is optional, so the base file does not require it. clients.yaml declares nothing about ds — no client, no scope family — and syncing it alone produces a correct celine realm. Where a dataspace is deployed, its clients are declared by ds's own file and celine adds grants to them from clients.ds-host.yaml.

Forgetting ds's file is refused rather than survived, and no keyword is needed for it: every entry in the host overlay is grants-only, so without ds's declaration they name clients nobody owns, and the sync stops before authenticating rather than creating eight clients with no name and generated secrets.

A file that genuinely cannot be synced on its own may still say so, and the file that answers it identifies itself by name — not by path, because a deployment mounts a file wherever it likes:

# a base file that is only ever half a realm
requires: [ds]        # refuse to sync without the declaration called 'ds'

# the file that answers it
overlay: ds           # this is that declaration

Use it only where the base file is genuinely incomplete without the other. Putting requires: on a file that describes a working realm makes an optional component mandatory — which is why neither file above carries one.

A grant naming a scope no file declares is refused the same way, in every environment and with no flag to accept it: the scope would never be created, so the grant would be skipped and the service would get a 403 the first time it needed it. This used to be a warning that synced anyway and failed at the end, with the realm already rewritten — see ADR-0002. See ADR-0001 for why this is merged in the loader rather than pre-merged into a generated file.

Dataspace Integration

The CLI manages Keycloak resources for the CELINE dataspace layer (identity registry, onboarding, portal).

`dataspace` claim scope

ensure_realm_claim_scopes() creates a dataspace client scope with an oidc-usermodel-attribute-mapper that maps the Keycloak user attribute dataspace_did into a dataspace_did JWT claim (id, access, and userinfo tokens). The scope is assigned as a default scope on the oauth2_proxy client, so every user JWT automatically carries the claim when the attribute is set.

`identity-registry.admin` scope

A standard OAuth scope granting admin access to the dataspace identity-registry API. It follows the {service}.admin naming convention, and is declared in clients.ds-host.yaml because celine is the only party that grants it: ds omits every *.admin from what it carries into a host realm, on the grounds that a long-lived process should not hold a superset over every permission of a service.

Dataspace service clients

They follow the svc-ds-* naming convention to distinguish dataspace services from platform services (svc-*), and they are declared by ds, not here — ds owns their identity and the identity-registry.*, connector.*, provenance.* and catalog.* vocabularies. This repository used to carry a hand-pasted copy of that declaration; it drifted, and the copy is gone. See ADR-0001.

What celine decides about them is in clients.ds-host.yaml, one grants-only entry per client:

svc-ds-identity-registry — identity-registry.admin. ds drops every *.admin from what it carries into a host realm; this realm drives the registry as an operator surface, so it grants the superset and declares it.
svc-ds-onboarding — rec-registry.members.write, rec-registry.lookup and rec-registry.read (the community reads, such as the console's duplicate delivery points, as svc-onboarding holds them). rec-registry is celine's own service, so ds's file cannot carry these grants.
svc-ds-portal — the host-side console grants, including dataset.query / dataset.read against celine's dataset-api.
svc-ds-dataset-api — dataset.admin and the svc-dataset-api audience. dataset.* is celine's vocabulary on celine's data plane; this client is the dataset API's outbound identity (spindoxlabs/ds#14).

The other four ds clients — svc-ds-connector, svc-ds-provenance, svc-ds-federated-catalog, svc-edc — need nothing from celine and appear in neither file.

Development

# Run MQTT auth dev server
task run

# Run tests
task test

# Release (semantic-release)
task release

License

Apache 2