loader

etl_craft.config.loader

Reading and validating craft-connector.yml.

The file is written by the team and only read by the engine. Its sections come in this order: Secrets (where variables are read from, and the default profile), Orchestration (mode, limits, DAG defaults and the Email relay), Engine, then the optional Warehouse and Cloning. Every section that varies by environment holds one block per profile (dev, prod, or any names). The active one is the section's own Profile, else Secrets.Profile; a section with a single profile block needs no selection. A profile block's settings override the section's own.

Every problem is a ConfigurationError naming the file and the setting.

SECTIONS module-attribute

SECTIONS = ('Secrets', 'Orchestration', 'Engine', 'Warehouse', 'Cloning', 'Docs_site')

The top-level sections, in the order the file must present them.

settings_by_section

settings_by_section() -> dict[str, frozenset[str]]

Return every setting the loader accepts, by where it is written.

Engine and Warehouse hold their section settings and the fields of a profile block; a warehouse's own separate fields (such as Snowflake's account) are included.

load_config

load_config(path: str | Path) -> ConnectorConfig

Read, parse and validate the craft-connector.yml at path.

parse_config

parse_config(raw: Any, path: Path) -> ConnectorConfig

Validate an already-loaded craft-connector.yml mapping; path is where it came from.

Relative paths in it (Secrets.Path) resolve against path's directory.