catalog

etl_craft.engine.repository.catalog

What the documentation catalog reads: pipelines, tasks and rules, with their latest runs.

RUN_HISTORY module-attribute

RUN_HISTORY = 30

How many recent runs of each pipeline and task the catalog shows.

LastRun dataclass

A pipeline's or task's latest run: its status, when, and a task's counts and error.

PipelineRow dataclass

An active pipeline and its latest run, if it has run.

TaskRow dataclass

An active task and its latest finished run, if it has one.

RuleRow dataclass

An active business rule of an active task.

RunSummary dataclass

One run of a pipeline, for its history.

seconds property

seconds: float | None

How long the run took, or None while it has not ended.

TaskRunSummary dataclass

One run of a task, for its history.

seconds property

seconds: float | None

How long the latest attempt took, or None while it has not ended.

Consumption dataclass

An upstream run a downstream run consumed; the tasks are set for a task dependency.

fetch_catalog_pipelines

fetch_catalog_pipelines(conn: Connection) -> list[PipelineRow]

Return every active pipeline with its latest run.

fetch_catalog_tasks

fetch_catalog_tasks(conn: Connection) -> list[TaskRow]

Return every active task of an active pipeline with its latest finished run.

fetch_catalog_rules

fetch_catalog_rules(conn: Connection) -> list[RuleRow]

Return every active business rule of an active task.

fetch_documentation_versions

fetch_documentation_versions(conn: Connection) -> dict[int, int]

Return each documented task's latest documentation version, by task id.

fetch_pipeline_runs

fetch_pipeline_runs(conn: Connection, limit: int = RUN_HISTORY) -> dict[str, list[RunSummary]]

Return the latest limit runs of every active pipeline, by code, newest first.

fetch_task_runs

fetch_task_runs(conn: Connection, limit: int = RUN_HISTORY) -> dict[int, list[TaskRunSummary]]

Return the latest limit runs of every active task, by task id, newest first.

fetch_consumption

fetch_consumption(conn: Connection) -> list[Consumption]

Return every logged consumption by a known downstream run, oldest first.