config
etl_craft.config ¶
craft-connector.yml: finding it, reading it, and what each connection target accepts.
The file is written by the team and only read by the engine; secrets are always variable names, never values.
CloningConfig
dataclass
¶
Copying Engine DB tables into the warehouse.
external_volume and base_location place the copies' Iceberg storage on warehouses
that name it explicitly (Snowflake).
ConnectionProfile
dataclass
¶
The active Engine or Warehouse connection.
extra holds the auth mode's own fields (key_file, client_id, ...), the dialect's
profile fields, and secret_var when the profile names a secret. schema is where the
profile's tables live: the Engine DB's own tables, or the warehouse schema cloning writes to.
An empty schema leaves the database's default.
ConnectorConfig
dataclass
¶
Everything craft-connector.yml configures.
dependency_gates is what a local run does with its dependencies on other pipelines.
warehouse_table_format is the default for tables the engine creates; a task may choose
another with CFG_TASK_PARAMETERS.TABLE_FORMAT. log_dir is where each task attempt's
log file is written. config_path is where the file was
read, so every task process reads the same one. settings records where each value
came from, in file order, for doctor.
The directory holding the file is the project directory (conventionally etl-craft/).
It holds the project's sql_files/, ingestion_scripts/ and migrations/.
project_dir
property
¶
The directory holding craft-connector.yml, or the current one without a file.
sql_files_dir
property
¶
Where SOURCE_SQL_FILE names are found: sql_files/ in the project.
ingestion_scripts_dir
property
¶
Where Python ingestion scripts are found: ingestion_scripts/ in the project.
migrations_dir
property
¶
The project's own Engine DB migrations: migrations/ in the project.
DagDefaults
dataclass
¶
Defaults for the DAGs generate-yml writes.
None means not set, so the generator uses its own default. allow_schedule false
writes schedule: null, so an environment holds every pipeline but runs one only when
triggered.
DocsSiteConfig
dataclass
¶
The catalog site generate-docs writes and publish-docs serves.
schedule is when it is written again, as a cron expression, for the DAG
generate-yml --docs writes; output is its folder, catalog/ in the project
directory when not set. authtoken_var names the variable holding the ngrok authtoken,
read only when the site is published; domain is the team's reserved ngrok domain, and
allowed_ips the CIDR ranges publish-docs serves, every address when empty.
EmailProfile
dataclass
¶
How email is sent: through an SMTP relay, or the host's own sendmail program.
With transport sendmail the message is handed to sendmail_path (the program
mailx and mail use), which delivers it through the host's mail system; host,
port and the login settings do not apply. from_address is the sender either way, and
from_name, when set, the name shown with it.
secret_var
property
¶
The variable holding the relay password or OAuth client secret.
ExecutionLimits
dataclass
¶
Deployment-wide limits: a task's time limit, parallel tasks, SLA alerts and gate waits.
task_timeout_seconds of 0 means no limit. gate_wait_minutes is how long a
cross-pipeline gate waits for a running upstream, 0 for not at all.
Every run of a pipeline with SLA_IN_HOURS is marked MET or BREACHED. With
enforce_sla on, a run that misses its SLA also sends an SLA email, through the Email
settings.
SettingSource
dataclass
¶
Where one setting's value came from: a variable, or the text as written.
where is the setting's path in the file, such as Engine.dev.user.
looks_like_a_missing_variable
property
¶
Whether the text was used as written although it reads like a variable name.
SourceConfig
dataclass
¶
Where the variables the file names are read from: the environment, or a .env file.
resolve_config_path ¶
Return the config file to read: --config, then $ETL_CRAFT_CONFIG, then a search.
The search looks in start (the current directory by default) and then each parent, the
way .git is found, because a scheduler's working directory is rarely the project's. In
each directory it looks for craft-connector.yml (the directory is the project) and for
etl-craft/craft-connector.yml (the directory holds the project). Finding both in one
directory is a ConfigurationError: which one is meant cannot be told. With nothing
found, it returns etl-craft/craft-connector.yml in start, so the error names where
the file was expected.
load_config ¶
Read, parse and validate the craft-connector.yml at path.
parse_config ¶
Validate an already-loaded craft-connector.yml mapping; path is where it came from.
Relative paths in it (Secrets.Path) resolve against path's directory.
profile_needs_secret ¶
Whether connecting with profile reads a secret.
profile_secret ¶
Return profile's secret, or "" when its auth mode presents none.
none, sts and sso without a client secret have nothing to present; a key file's
passphrase is read when the profile names one.
resolve_secret ¶
Read profile's secret from the secrets source, as it is now.
Raises ConfigurationError if the variable is no longer set.