Guides

How to model and run pipelines with etl-craft.

• Pipelines and tasks: the CFG_ rows a pipeline is made of, the task handlers, and the settings every task shares.
• Dependencies and run conditions: dependency types, ALL, ANY and N conditions, waves, and which tasks run, wait or are skipped.
• SQL tasks: the SELECT, inline or from sql_files/, the pipeline-id tokens, the eight actions, and what the engine checks and logs.
• Business rules: writing rules, flagging and clearing rows, waves and retries.
• Ingestion scripts: the script contract, INPUT_PARAMS, offsets, and where a script's output goes.
• Email alerts: the alert task, its outcomes and tokens, SLA emails, and sending through SMTP or sendmail.
• Validating pipelines: validate, every check it makes, and what fails.
• Inspecting pipelines: list, graph, steps and history.
• Lineage and documentation versions: column lineage across tasks and pipelines, and docs-version.
• The catalog site: generate-docs, the pages, lineage graphs and search.
• Running a pipeline: dependency waves, resuming a run, connection tests, orchestrator steps, and the SLA.
• Running a task and reading its logs: etl-craft run --task_code, retries, time limits, and where each attempt's log goes.

• Stepping in: mark, cancel, pause, backfill, rerun and bypasses: marking a task or a run, stand-in runs, cancelling, pausing, skipping a run, backfilling, running a task again or without its dependencies, relaxing dependency gates, and the record every change leaves; local mode.

• Service operations and JSON: calling the same operations from Python and the CLI, execution identities, actors and schema-versioned documents.