Skip to content

etl-craft

craft-connector.yml

venkatcg00/etl-craft

craft-connector.yml¶

Every setting the loader accepts, by section, generated from the loader itself. A key it does not accept stops every command with the key named. Each setting is explained where it is shown: the annotated reference, or the example that uses it.

Orchestration settings, and each section's Profile, may be written at the section level or in a profile block (dev:, prod:, ...); Engine and Warehouse connection fields are written in a profile block.

`Secrets`¶

Setting	Shown in
`Path`	`craft-connector.example.yml`
`Profile`	`craft-connector.example.yml`
`Source_type`	`craft-connector.example.yml`

`Orchestration`¶

Setting	Shown in
`Allow_schedule`	`craft-connector.example.yml`
`Api_address`	`craft-connector.example.yml`
`Catchup`	`craft-connector.example.yml`
`Dependency_gates`	`craft-connector.example.yml`
`Depends_on_past`	`craft-connector.example.yml`
`Email_on_failure`	`craft-connector.example.yml`
`Email_recipients`	`craft-connector.example.yml`
`Enforce_sla`	`craft-connector.example.yml`
`Gate_wait_minutes`	`craft-connector.example.yml`
`Global_dag`	`craft-connector.example.yml`
`Log_dir`	`craft-connector.example.yml`
`Max_parallel_tasks`	`craft-connector.example.yml`
`Mode`	`craft-connector.example.yml`
`Name`	`craft-connector.example.yml`
`Retries`	`craft-connector.example.yml`
`Retry_delay_minutes`	`craft-connector.example.yml`
`Shutdown_grace_seconds`	`craft-connector.example.yml`
`Tags`	`craft-connector.example.yml`
`Task_timeout_seconds`	`craft-connector.example.yml`
`Timezone`	`craft-connector.example.yml`

`Orchestration.Email`¶

Setting	Shown in
`auth_mode`	`craft-connector.example.yml`
`ca_file`	`craft-connector.example.yml`
`client_id`	`craft-connector.example.yml`
`from_address`	`craft-connector.example.yml`
`from_name`	`craft-connector.example.yml`
`host`	`craft-connector.example.yml`
`port`	`craft-connector.example.yml`
`scope`	`craft-connector.example.yml`
`secret`	`craft-connector.example.yml`
`sendmail_path`	`craft-connector.example.yml`
`tls_mode`	`craft-connector.example.yml`
`token_url`	`craft-connector.example.yml`
`transport`	`craft-connector.example.yml`
`user`	`craft-connector.example.yml`

`Engine`¶

Setting	Shown in
`auth_mode`	`craft-connector.example.yml`
`cert_file`	`auth-postgres.yml`, `auth-trino.yml`
`client_id`	`craft-connector.example.yml`
`issuer`	`auth-postgres.yml`
`jdbc_url`	`craft-connector.example.yml`
`key_file`	`craft-connector.example.yml`
`Name`	`craft-connector.example.yml`
`region`	`auth-postgres.yml`
`role_arn`	`auth-postgres.yml`
`schema`	`craft-connector.example.yml`
`scope`	`craft-connector.example.yml`
`secret`	`craft-connector.example.yml`
`token_url`	`craft-connector.example.yml`
`user`	`craft-connector.example.yml`

`Warehouse`¶

Setting	Shown in
`account`	`auth-snowflake.yml`, `warehouse-snowflake-iceberg.yml`, `warehouse-snowflake.yml`
`auth_mode`	`craft-connector.example.yml`
`catalog`	`auth-databricks.yml`, `auth-duckdb-iceberg.yml`, `warehouse-databricks-iceberg.yml`, `warehouse-databricks.yml`, `warehouse-duckdb-iceberg.yml`
`catalog_uri`	`auth-duckdb-iceberg.yml`, `warehouse-duckdb-iceberg.yml`
`cert_file`	`auth-postgres.yml`, `auth-trino.yml`
`client_id`	`craft-connector.example.yml`
`database`	`auth-snowflake.yml`, `warehouse-snowflake-iceberg.yml`, `warehouse-snowflake.yml`
`iceberg_warehouse`	`auth-duckdb-iceberg.yml`, `warehouse-duckdb-iceberg.yml`
`issuer`	`auth-postgres.yml`
`jdbc_url`	`craft-connector.example.yml`
`key_file`	`craft-connector.example.yml`
`Name`	`craft-connector.example.yml`
`region`	`auth-postgres.yml`
`role`	`auth-snowflake.yml`, `warehouse-snowflake-iceberg.yml`, `warehouse-snowflake.yml`
`role_arn`	`auth-postgres.yml`
`s3_endpoint`	`auth-duckdb-iceberg.yml`, `warehouse-duckdb-iceberg.yml`
`s3_key_id`	`auth-duckdb-iceberg.yml`, `warehouse-duckdb-iceberg.yml`
`s3_region`	`warehouse-duckdb-iceberg.yml`
`s3_secret`	`auth-duckdb-iceberg.yml`, `warehouse-duckdb-iceberg.yml`
`s3_url_style`	`warehouse-duckdb-iceberg.yml`
`s3_use_ssl`	`warehouse-duckdb-iceberg.yml`
`schema`	`craft-connector.example.yml`
`scope`	`craft-connector.example.yml`
`secret`	`craft-connector.example.yml`
`Table_format`	`craft-connector.example.yml`
`token`	`auth-databricks.yml`, `auth-duckdb-iceberg.yml`, `auth-postgres.yml`, `auth-snowflake.yml`, `auth-trino.yml`, `warehouse-databricks-iceberg.yml`, `warehouse-databricks.yml`, `warehouse-snowflake-iceberg.yml`, `warehouse-snowflake.yml`
`token_url`	`craft-connector.example.yml`
`user`	`craft-connector.example.yml`
`warehouse`	`auth-snowflake.yml`, `warehouse-snowflake-iceberg.yml`, `warehouse-snowflake.yml`

`Cloning`¶

Setting	Shown in
`Base_location`	`warehouse-snowflake-iceberg.yml`
`Enabled`	`craft-connector.example.yml`
`External_volume`	`warehouse-snowflake-iceberg.yml`
`Scope`	`craft-connector.example.yml`

`Docs_site`¶

Setting	Shown in
`Allowed_ips`	`docs-site.yml`
`Authtoken`	`docs-site.yml`
`Domain`	`docs-site.yml`
`Output`	`docs-site.yml`
`Schedule`	`docs-site.yml`