Requirements
What this repository must do, as opposed to what it currently does. Each
requirement carries an identifier, and a test declares what it verifies with a
@verifies REQ-#### tag on its own line directly above the test. The trace matrix is
the projection of the two — generated, never hand-maintained.
Scope is deliberately the shared surface: the properties other repositories depend on, where a regression here breaks something there. Pipeline execution is out of scope for now — it has integration coverage but no unit evidence, so requirements there would need evidence written alongside them. Platform administration is out of scope permanently: it was removed in 2.3.0 (ADR-0004).
A requirement here states a property and its consequence. It is not an ADR — the reasoning for a decision lives in decisions — and it is not documentation of behaviour, which lives in the pages these link to.
The register
| ID | Requirement | Verified by |
|---|---|---|
| REQ-0001 | celine.governance imports on core dependencies alone |
tests/test_governance_contract.py, .github/workflows/governance-thin.yaml |
| REQ-0002 | Catalogue and dataspace exposure are separate gates, ANDed | tests/test_governance_exposure.py |
| REQ-0003 | The package works on every Python version it claims | tests/test_supported_python_range.py, .github/workflows/test.yaml |
| REQ-0004 | The published facet schema URL keeps resolving | tests/test_governance_contract.py |
| REQ-0005 | An unknown key never silently changes a dataset's governance | tests/test_governance_contract.py, tests/test_governance_subclassing.py |
| REQ-0006 | A declared dependency resolves to a producer, or is reported | tests/test_governance_graph.py |
| REQ-0007 | A schedule contradicting the graph is reported, and an uncertain one is not asserted | tests/test_governance_graph.py |
REQ-0001 — celine.governance imports on core dependencies alone
celine.governance and every module beneath it MUST import successfully with only
pydantic, pyyaml and jsonschema installed.
It MUST NOT import celine.utils, SQLAlchemy, OpenLineage, Keycloak, pandas, dbt,
Meltano or Prefect, at module scope or inside a function. Where a governance module
needs something from the fat side, the dependency inverts: celine.utils may import
celine.governance, never the reverse.
Consequence if violated: dataset-api, ds and celine-superset cannot import
this package without inheriting an orchestration stack, which is the condition that
produced four disagreeing parsers. The violation appears at a consumer's install,
not here — a developer environment has every extra installed.
Rationale: ADR-0001. Behaviour: the governance library.
REQ-0002 — Catalogue and dataspace exposure are separate gates, ANDed
expose MUST gate catalogue listing and API serving. dataspace.expose MUST gate
the dataspace offer. Dataspace access requires both.
exposeMUST be tri-state. Unset MUST fall back todataspace.expose, so a file written before the split keeps its catalogue behaviour unchanged.dataspace.exposeMUST NOT fall back in either direction.expose: falsewithdataspace.expose: trueMUST be reported as a conflict rather than resolved. It MUST be reported rather than raised, so a caller can collect every conflict in one run.- An override that states
expose: falseMUST withdraw a dataset the baseline exposed.
Consequence if violated: either data is published that the catalogue never advertised, or an offer someone deliberately made is silently dropped. Both are security-relevant, which is why neither direction may be chosen implicitly.
Rationale: ADR-0003. Behaviour: the two gates.
REQ-0003 — The package works on every Python version it claims
The Programming Language :: Python :: 3.x classifiers in pyproject.toml are the
single declaration of the supported range. requires-python MUST agree with them,
and CI MUST build its matrix from them rather than from a second list.
Every submodule of celine.governance and celine.utils MUST import on every
version in that range, with the extras installed.
Consequence if violated: the supported range becomes aspirational. celine-sdk
lost two of 553 submodules to a version-gated import while its own suite passed, and
this package once declared a 3.12 floor it did not need — inherited through an
extras-only dependency — which locked celine-superset out entirely, since it ships
inside apache/superset:6.0.0 on Python 3.10.
REQ-0004 — The published facet schema URL keeps resolving
celine.governance.facet.SCHEMA_URL MUST remain
https://celine-eu.github.io/schema/GovernanceDatasetFacet.schema.json, and that URL
MUST keep serving a schema the emitted facets validate against.
The schema files MUST stay at schema/ in the repository root, and MUST remain
readable from the installed package through importlib.resources — not by walking
from __file__, which fails inside a wheel or a container.
Consequence if violated: no build breaks. Every OpenLineage event already emitted and sitting in Marquez carries this URL, so changing or moving it silently invalidates historical lineage, and nothing reports it.
Rationale: ADR-0002.
REQ-0005 — An unknown key never silently changes a dataset's governance
A key the grammar does not define MUST be preserved in rule.extra rather than
dropped, and MUST be reported by validate — as a warning by default, and as an
error under strict=True.
The split between a field and extra MUST follow the model class being parsed into,
including a consumer's subclass. A subclass split against the base grammar's key set
has the same failure as a forgotten KNOWN_KEYS entry once had: the field reads as its
default forever, and nothing warns.
KNOWN_KEYS MUST list every field of GovernanceRule that a file may declare. It is
what validate reports unknown keys against, so a field present in the model and absent
from it is reported as unknown — a legitimate field called a mistake.
Parsing MUST go through model_validate on the keys a file actually declared, so
model_fields_set distinguishes unset from set to a falsy value.
Consequence if violated: a misspelled key — access_levl: open — validates
against the schema, is discarded by the model, and the dataset silently takes the
default. A model class narrower than the file it is handed does the same thing to
fields that are spelled correctly. Under kwargs construction the merge degrades to
"override always wins" and expose: false becomes inexpressible, which is how a
withdrawal that validated clean left a dataset in the catalogue.
Behaviour: unknown keys, the companion's knowledge.
REQ-0006 — A declared dependency resolves to a producer, or is reported
depends_on MUST name datasets, never pipelines, and the producing pipeline MUST
be resolved by matching each entry against the sources keys of every governance file
in the scanned set.
- Matching MUST be case-sensitive
fnmatchapplied in both directions, so either the dependency or thesourceskey may be the glob. - An entry resolving to no producer MUST be reported, unless it declares
external: true. An entry declaringexternal: truethat does resolve MUST be reported as informational rather than as a fault. - A dataset declared by two governance files MUST be reported.
- A pipeline that cannot be placed in a run order MUST be reported and MUST NOT be emitted in a tier.
- Absent
depends_onMUST remain distinct fromdepends_on: []: the first states that a pipeline has not declared its inputs, the second that it has none. - An unknown key at the root of a governance document MUST be reported by
validate, on the same terms as an unknown key inside a block. - The graph MUST be computable within the dependencies of REQ-0001.
active: falseMUST mark a pipeline as not meant to run, and an active pipeline reading from an inactive one MUST be reported.
Consequence if violated: each app is its own dbt project, so ref() never crosses
an app boundary and nothing else in the platform can see the ordering across pipelines
— it lives in cron offsets and hand-maintained tier tables. A dependency that resolves
to nothing and is not reported produces a graph that looks authoritative and is
silently incomplete, which is worse than no graph: an ordering is read as an
instruction. The both-directions rule is what keeps this from happening per
deployment: ds_dev_* is the default everywhere, but a consumer resolves the schema
through CELINE_SILVER_SCHEMA / CELINE_GOLD_SCHEMA, which a deployment may point
anywhere.
Behaviour: depends_on,
the CLI.
REQ-0007 — A schedule contradicting the graph is reported, and an uncertain one is not asserted
Given a deployment's scheduled flows, the graph MUST report a producer/consumer pair whose crons contradict the declared edge:
- Collision — the two can start in the same minute, so which run the consumer sees depends on which container starts first.
- Inversion — both fire hourly, never in the same minute, and the consumer is always earlier in the hour, so it always reads the previous run's output.
Two things it MUST NOT do:
- Assert an ordering it cannot know. Governance is per app; a deployment schedules
flows, and one app can deploy several on independent crons. Where either side
deploys more than one, the pairing may not be the one that moves the data, and the
finding MUST be reported as advisory and excluded from
--strict. - Compare minutes across periods. Pairs on different periods MUST be reported only on an identical cron expression; reasoning about a real offset between an hourly and a daily flow needs the run duration, which is in no file.
Schedules MUST NOT be read from governance.yaml. A cron is a deployment fact — one app
runs several flows, and the same pipeline runs on different schedules in different
deployments — while a governance file is one per app and is shared across every
deployment that installs it.
Consequence if violated: the failure this catches is silent by construction. A consumer that runs before its producer succeeds every time, on stale data, and no test or alert fires — the ordering was only ever expressed as a cron offset, which is a convention nothing enforces. The advisory split matters just as much: a check that reports a pairing it cannot verify as though it were certain gets ignored, and then so does the one that was right.
Behaviour: governance graph.