`celine.governance` — the library

celine.governance is the one implementation of governance.yaml parsing, merging and validation. Every CELINE component that reads a governance file imports it: dataset-api, ds, celine-superset and the pipeline runner in this repository.

This page is for consumers importing the package. For the file format itself see the format reference; for the CLI see CLI.

The dependency contract

Runtime dependencies are pydantic, pyyaml and jsonschema. Nothing else, ever.

This is the feature, not a style preference. The package exists because celine-utils once required dbt, Meltano, Prefect and Keycloak in order to parse a YAML file — so no API service could import it, four teams wrote their own parser instead, and the four then disagreed about what the same file meant.

celine.governance must not import celine.utils, SQLAlchemy, OpenLineage, or anything from an optional extra. If a governance module needs something from the fat side, the dependency goes the other way round: celine.utils may import celine.governance, never the reverse.

A CI job asserts this rather than trusting review — adding one import line is easy and nothing in a diff makes it look expensive. See ADR-0001.

Installing just the core

uv add celine-utils          # core only — three dependencies

The extras are opt-in and none of them are needed to read governance:

Extra	Brings	For
(none)	pydantic, pyyaml, jsonschema	parsing governance
`openlineage`	`openlineage-python`	the typed `GovernanceDatasetFacet` class
`pipelines`	dbt, Meltano, Prefect, pandas, SQLAlchemy, …	running pipelines
`all`	everything above	pipelines plus the typed facet

There was an admin extra until 2.3.0. It is gone with the command tree it served — requesting it now installs successfully and warns does not have an extra named 'admin', so check the name rather than trusting the install to complain.

build_facet returns a plain dict and needs none of them, including openlineage.

Import path

The package ships from the celine-utils wheel today and is expected to become its own distribution. The import path celine.governance was chosen so that move is a directory rename with no consumer editing an import. Do not refer to it as celine.utils.governance — that path does not exist, and creating it would make the eventual split a breaking change for every downstream repository.

Reading a file

from pathlib import Path
from celine.governance import GovernanceResolver

resolver = GovernanceResolver.from_file(Path("governance.yaml"))
rule = resolver.resolve("datasets.ds_prod_gold.weather_hourly")

rule.access_level      # 'restricted'
rule.tags              # ['gold', 'weather']

A missing file is an error: from_file raises FileNotFoundError. Handing over a path is already a statement about which file you mean, so absence is what you asked for is not there — not nothing was asked for, which is auto_discover's case and still returns an empty configuration.

It used to warn and return that empty configuration instead. A connector whose governance/ directory had been renamed then started clean, served an empty dataset list and empty sharing offers, and said nothing while the files sat on disk; the first symptom was an unrelated end-to-end test failing on a fixture that was right there in the file. Catch FileNotFoundError if a missing file is a state your caller supports.

Constructors

Call	Use
`GovernanceResolver.from_file(path)`	a known path
`GovernanceResolver.from_dict(raw)`	a document already parsed, e.g. from an API response
`GovernanceResolver.from_file_with_override(base, overlay_name=None, *, infer_from_dir=False)`	base plus a deployer overlay beside it
`GovernanceResolver.auto_discover(app_name=None, project_dir=None)`	convention-based lookup — see discovery order

infer_from_dir is opt-in rather than default because the two callers this consolidated genuinely disagreed: one inferred the overlay name from the parent directory, the other returned the base unchanged. Neither default would have been correct for both, so each passes what it means.

A missing overlay is not an error, unlike a missing base: the overlay is located by convention rather than named by the caller, so its absence is the "nothing was asked for" case.

`parse_rule`

parse_rule(block) builds a single GovernanceRule from one raw block, accepting either a bare block or one nested under a governance: key. parse_rule(block, MyRule) builds a subclass of one — see extending the models.

It goes through model_validate on a dict containing only the keys the block actually declared. That is load-bearing. Pydantic records those keys in model_fields_set, and the whole merge layer reads it to tell unset from set to a falsy value. Constructing a GovernanceRule with keyword arguments marks every field as set, which silently degrades merging to "override always wins" and makes expose: false inexpressible. Build rules with parse_rule or model_validate, never with kwargs, if the result will be merged.

Merging

from celine.governance import merge_rules, merge_configs

resolved = merge_rules(base_rule, override_rule)
config   = merge_configs(base_config, overlay_config)

Function	Combines
`merge_rules(base, override, *, model_cls=None)`	two `GovernanceRule`s, with the per-field rules below
`merge_configs(base, override, *, model_cls=None)`	two whole documents — defaults with defaults, sources rule-wise, overlay-only sources added as-is
`merge_dataspace(base, override, *, model_cls=None)`	two `DataspaceConfig`s
`merge_models(base, override, cls)`	the generic `exclude_unset` overlay

The overlay uses exclude_unset — not exclude_defaults, and never truthiness. All three drop a field the source never mentioned; they differ on a field the source did mention whose value equals the default. exclude_defaults drops it, so it cannot tell silent from said no. Truthiness cannot express off at all once the base says on.

The full per-field table is in the format reference.

Validation

from pathlib import Path
from celine.governance import validate_file, validate, GovernanceValidationError

unknown = validate_file(Path("governance.yaml"))              # warns
unknown = validate_file(Path("governance.yaml"), strict=True) # raises

Function	Behaviour
`validate(data, *, source, strict=False)`	schema violations raise; unknown keys warn, or raise under `strict`. Returns the unknown keys
`validate_file(path, *, strict=False)`	the same, loading YAML from a path
`schema_errors(data, schema_name)`	every violation as a list of strings, sorted by document position. Raises nothing
`unknown_keys(data)`	keys the grammar does not define, per block
`load_schema(name)`	a packaged JSON Schema as a dict
`validate_owners(data)` / `validate_owners_file(path)`	strict always — see owners

schema_errors reports every error rather than the first: a governance file is edited by hand, and one problem per run turns a five-minute fix into five round trips.

Why unknown keys warn by default. Seventeen governance files had never been schema-checked when validation arrived. Making unknown keys fatal on day one would have turned adopting this package into a seventeen-file cleanup discovered at import time — in an exporter, in CI, at the worst possible moment. Warn, fix, then flip the default.

Schemas are read from the installed package via importlib.resources, never by walking from __file__, so validation works from a wheel and inside a container.

Exposure gates

from celine.governance import effective_expose, dataspace_expose, exposure_conflict

effective_expose(rule)    # listed in the catalogue and served by the API?
dataspace_expose(rule)    # offered into the dataspace?

if (why := exposure_conflict(rule)) is not None:
    log.error("%s: %s", name, why)

effective_expose falls back to dataspace.expose when expose is unstated; dataspace_expose has no fallback in either direction. exposure_conflict returns a sentence or None — it is reported rather than raised so a caller can collect every conflict in a run instead of failing on the first.

Never read rule.expose directly to decide catalogue visibility: it is tri-state, and None does not mean False. See the two gates.

from celine.governance import build_facet, is_empty, SCHEMA_URL

if not is_empty(rule):
    facet = build_facet(rule, producer="https://github.com/celine-eu/celine-utils")

Returns a plain dict with camelCase keys and imports nothing from OpenLineage, so a consumer gets the projection without the dependency. Absent values are omitted rather than emitted as null.

include_dataspace defaults to True, which matches the catalogue — the consumer that reads those fields back out. The lineage extractors historically did not project the dataspace block, so events already in Marquez carry no dataspace fields; an extractor adopting this function should pass include_dataspace=False to keep emitting what it emits today. Widening the payload is a deliberate change, not a side effect of deduplication.

SCHEMA_URL is a contract, not a configuration. It is embedded in every OpenLineage event already sitting in Marquez, so changing it does not break a build — it silently invalidates historical lineage.

Levels

from celine.governance import normalize_access_level, normalize_classification

normalize_access_level("SECRET")   # -> 'restricted'
normalize_classification("PII")    # -> 'pii'

Both accept None and return None. Both raise ValueError on a value outside the enum. secret is folded to restricted for compatibility with deployed files; the enums are GovernanceAccessLevel (open / internal / restricted), DataClassification (green / yellow / red / pii) and AccessRequirement (all / partner / contract).

Models

GovernanceRule is the resolved block; GovernanceConfig holds defaults and sources. Sub-models: DcatConfig, OntologyConfig, DataspaceConfig, TemporalCoverage, GovernanceOwner.

Every model sets extra="ignore", so a file carrying fields this package does not know — the EDC sub-objects ds adds, a new field from a newer release — parses rather than failing. Unknown keys at block level are preserved in rule.extra.

KNOWN_KEYS is the frozenset stating which keys the base grammar defines. It is what validate checks a block against when reporting unknown keys.

It is no longer what decides the split. parse_rule reads that off the model class it is given, so a consumer's subclass keeps its own fields instead of watching them land in extra. For GovernanceRule the two sets are identical, and a contract test holds them there.

The difference matters when they do drift: a field added to the model and forgotten in KNOWN_KEYS now parses correctly and is reported as an unknown key by validate — noisy and wrong, rather than the silent permanent absence it used to be. That silence is how the ontology block failed on introduction; it can no longer happen that way.

Extending the models

ds extends GovernanceRule with ODRL policy and a richer dataspace spec. Subclass rather than adding fields here: what belongs in this package is the surface every consumer shares. A field only one component reads is a field the other three carry without meaning.

class SpecRule(GovernanceRule):
    policy: dict | None = None

base     = parse_rule(base_block, SpecRule)
override = parse_rule(override_block, SpecRule)

merged = merge_rules(base, override)   # -> SpecRule, policy and all

parse_rule, merge_rules, merge_dataspace and merge_configs all follow the class they are given: parse_rule from its model_cls argument, the merges from their operands. Nothing needs model_cls in the ordinary case.

Both operands must be the same class. Mixing them raises TypeError rather than picking one, because picking the more derived class invents fields the other operand never had and picking the base drops the ones it did — either way a silent decision about which half of the input survives. Validate both into one class first, or pass model_cls to say deliberately what the result is:

merge_rules(base, override, model_cls=GovernanceRule)   # narrow on purpose

Until 3.0 these functions named their class instead. A subclass handed to any of them came back as a base-class instance with the subclass's fields dropped — no error, just a smaller object.

The package ships a PEP 561 py.typed marker, so a subclass is type-checkable rather than a subclass of Any.

celine.governance — the library