connection
etl_craft.warehouse.connection ¶
Connecting to the warehouse.
The warehouse can be any database SQLAlchemy has a dialect for. Its driver is never imported
here: at each new connection, the creator builds a SQLAlchemy URL carrying the credential and
asks that URL's own dialect to connect. That URL is never the engine's logged one, so a
credential never appears in logs. How the credential is presented (URL username and password,
URL query, or driver arguments) is each warehouse dialect's present.
READ_ONLY_WAIT_SECONDS
module-attribute
¶
How long a read-only check (validate, doctor) waits for a single-writer warehouse.
WAREHOUSE_LOCK_KEY
module-attribute
¶
The Engine DB lock that queues writers of a single-writer warehouse.
warehouse_dialect ¶
Return the dialect the active warehouse profile and the default table format select.
build_warehouse_engine ¶
Build a SQLAlchemy engine for the active warehouse profile.
The profile is checked against its dialect first, so a missing auth field fails before
any connection. Each new connection authenticates afresh and runs the dialect's
on_connect; pools using a minted credential are recycled before it expires.
warehouse_creator ¶
warehouse_creator(dialect: WarehouseDialect, profile: ConnectionProfile, secret: str, url: WarehouseUrl) -> Callable[[], Any]
Return a function that opens one authenticated DBAPI connection to the warehouse.
An embedded warehouse (a DuckDB file) is addressed by its path; a server keeps its host, port and query whatever the auth mode.
dbapi_connect ¶
Open one raw DBAPI connection through url's own SQLAlchemy dialect.
The dialect's connect is used rather than the driver's, because a dialect may do its
own setup there (DuckDB's parses its configuration and loads extensions). extra is
added after the dialect's own connect arguments, for settings that must never travel in a
URL, such as a private key's path and passphrase.
is_single_writer ¶
Whether the configured warehouse admits only one writing process at a time.
is_in_memory ¶
Whether the configured warehouse is an in-memory DuckDB database.
Every task runs in its own process, so each would see its own empty database; doctor
refuses such a configuration.
single_writer_lock ¶
single_writer_lock(config: ConnectorConfig, engine_db: Engine | None = None, *, wait_seconds: float = 0) -> Iterator[None]
Queue behind other writers when the warehouse admits one writing process; else nothing.
The lock is held in the Engine DB, which every process that could contend can reach,
including workers on other machines. Waiters queue in order rather than retrying against
the warehouse's own lock error. wait_seconds 0 waits indefinitely; otherwise a
LockTimeoutError says another task is still using the warehouse. A task whose own
script writes to the warehouse is wrapped in this lock too, without an engine being opened
for it.
open_warehouse ¶
open_warehouse(config: ConnectorConfig, engine_db: Engine | None = None, *, wait_seconds: float = 0, **engine_kwargs: Any) -> Iterator[Engine]
Open the warehouse for one unit of work, queued behind other writers where needed.
This is how the engine reaches the warehouse. For a warehouse that accepts concurrent writers it takes no lock, so waves stay fully parallel. The engine is disposed afterwards.
verify_iceberg_catalog ¶
Check that a Trino warehouse's catalog stores Iceberg tables; return the problem, or None.
Trino's table format comes from the catalog, and a deployment often has several. A URL
naming a Hive catalog would have the engine create Hive tables while treating them as
Iceberg, so doctor checks it. Every other warehouse returns None: its format is
fixed by the connection.
warehouse_schema_problem ¶
Return why the warehouse profile's schema cannot be used, or None.
It must already exist in the profile's catalog: the engine never creates warehouse schemas.