cloning
etl_craft.services.cloning ¶
Cloning: copying the Engine DB tables into the warehouse after each pipeline run.
With Cloning.Enabled, every run ends by mirroring the Engine DB tables its Scope names
(cfg: the CFG_ tables, aud: the AUD_ tables, all, or none) into the
warehouse profile's schema, so a team can query its pipelines' configuration and history
from inside its own warehouse.
Each mirror has the Engine DB table's name and columns, with portable types: whole numbers
BIGINT, other numbers DECIMAL(38, 10), timestamps the warehouse's timestamp with time
zone, and everything else text; JSON values are copied as their text. A mirror is created when
it is missing, and its rows replaced on every clone, one transaction per table where the
warehouse has them, many rows to an INSERT. A mirror is never dropped, so views built on it keep
working: a column an upgrade adds to the Engine DB table is added to the mirror, and a column
the mirror has that the Engine DB table no longer does is left in place, NULL in every row.
Cloning writes only the mirrors, and only in the warehouse profile's schema, which must already exist. It refuses a warehouse schema that is the Engine DB's own, where it would empty the tables it reads. Two clones never overlap: the second waits, then copies the newer state.
After a run, a failed clone is logged with its cause and does not change how the run ended;
etl-craft clone runs it by hand and fails with the cause (CloningError, naming the
table).
BATCH_ROWS
module-attribute
¶
How many rows are read from the Engine DB at a time.
MAX_BOUND_VALUES
module-attribute
¶
Values bound in one INSERT: many rows per statement, within every driver's parameter limit.
A cloud warehouse answers each statement in a second or two, so a row per statement would take hours on a busy Engine DB; a statement per few dozen rows takes minutes at most.
PREFIXES
module-attribute
¶
PREFIXES: Mapping[CloningScope, tuple[str, ...]] = {CloningScope.CFG: ('CFG_',), CloningScope.AUD: ('AUD_',), CloningScope.ALL: ('CFG_', 'AUD_'), CloningScope.NONE: ()}
The table-name prefixes each scope copies.
ClonedTable
dataclass
¶
One mirror written: its name, the rows copied, whether it was created, columns added.
cloning_problem ¶
Return why cloning cannot run with this configuration, or None.
Nothing connects: this is the Warehouse section, the storage the mirrors need, and whether the warehouse schema is the Engine DB's own.
clone ¶
Mirror the Engine DB tables of Cloning.Scope into the warehouse schema.
ConfigurationError when the configuration cannot work; a database error fails with the
table it was copying. Returns nothing when cloning is off or its scope is none.
run_hooks ¶
Return the hooks a run gets: default_hooks, then a clone once it has ended.
mirror_type ¶
Return the warehouse type a mirror gives an Engine DB column.