model
etl_craft.config.model ¶
The parsed contents of craft-connector.yml.
Only the selected profile of each section is kept, with every value already resolved. Secrets are kept as the names of the variables that hold them, never their values.
SettingSource
dataclass
¶
Where one setting's value came from: a variable, or the text as written.
where is the setting's path in the file, such as Engine.dev.user.
looks_like_a_missing_variable
property
¶
Whether the text was used as written although it reads like a variable name.
ConnectionProfile
dataclass
¶
The active Engine or Warehouse connection.
extra holds the auth mode's own fields (key_file, client_id, ...), the dialect's
profile fields, and secret_var when the profile names a secret. schema is where the
profile's tables live: the Engine DB's own tables, or the warehouse schema cloning writes to.
An empty schema leaves the database's default.
SourceConfig
dataclass
¶
Where the variables the file names are read from: the environment, or a .env file.
CloningConfig
dataclass
¶
Copying Engine DB tables into the warehouse.
external_volume and base_location place the copies' Iceberg storage on warehouses
that name it explicitly (Snowflake).
DocsSiteConfig
dataclass
¶
The catalog site generate-docs writes and publish-docs serves.
schedule is when it is written again, as a cron expression, for the DAG
generate-yml --docs writes; output is its folder, catalog/ in the project
directory when not set. authtoken_var names the variable holding the ngrok authtoken,
read only when the site is published; domain is the team's reserved ngrok domain, and
allowed_ips the CIDR ranges publish-docs serves, every address when empty.
EmailProfile
dataclass
¶
How email is sent: through an SMTP relay, or the host's own sendmail program.
With transport sendmail the message is handed to sendmail_path (the program
mailx and mail use), which delivers it through the host's mail system; host,
port and the login settings do not apply. from_address is the sender either way, and
from_name, when set, the name shown with it.
secret_var
property
¶
The variable holding the relay password or OAuth client secret.
DagDefaults
dataclass
¶
Defaults for the DAGs generate-yml writes.
None means not set, so the generator uses its own default. allow_schedule false
writes schedule: null, so an environment holds every pipeline but runs one only when
triggered.
ExecutionLimits
dataclass
¶
Deployment-wide limits: a task's time limit, parallel tasks, SLA alerts and gate waits.
task_timeout_seconds of 0 means no limit. gate_wait_minutes is how long a
cross-pipeline gate waits for a running upstream, 0 for not at all.
Every run of a pipeline with SLA_IN_HOURS is marked MET or BREACHED. With
enforce_sla on, a run that misses its SLA also sends an SLA email, through the Email
settings.
ConnectorConfig
dataclass
¶
Everything craft-connector.yml configures.
dependency_gates is what a local run does with its dependencies on other pipelines.
warehouse_table_format is the default for tables the engine creates; a task may choose
another with CFG_TASK_PARAMETERS.TABLE_FORMAT. log_dir is where each task attempt's
log file is written. config_path is where the file was
read, so every task process reads the same one. settings records where each value
came from, in file order, for doctor.
The directory holding the file is the project directory (conventionally etl-craft/).
It holds the project's sql_files/, ingestion_scripts/ and migrations/.
project_dir
property
¶
The directory holding craft-connector.yml, or the current one without a file.
sql_files_dir
property
¶
Where SOURCE_SQL_FILE names are found: sql_files/ in the project.
ingestion_scripts_dir
property
¶
Where Python ingestion scripts are found: ingestion_scripts/ in the project.
migrations_dir
property
¶
The project's own Engine DB migrations: migrations/ in the project.
parse_api_address ¶
Validate a host and port, including bracketed IPv6 and ephemeral port zero.