catalog_graph

etl_craft.services.catalog_graph

A table's lineage graph, laid out and drawn as SVG for its catalog page.

From one table the graph walks table edges upstream to the first sources and downstream to the last consumers, across tasks and pipelines. Each table sits in a column by its distance from the focus: upstream to the left (level -1, -2, ...), downstream to the right. Within a column, tables are ordered by where their neighbours are, a few sweeps each way, so edges cross less.

Each table box lists the columns that lineage connects to the other tables shown (every column, for the focus table). Column edges run from a source column's row to a target column's row: solid for a copy, dashed for a derived value, whose expression is the edge's tooltip. A column made from no source column, such as COUNT(*) or a constant, is marked. A task whose columns cannot be traced still links its tables, header to header, dotted.

Every edge ends in an arrow at the table made from its source. Each box has a header that shows or hides its columns and a button that opens the table's page. The drawing carries what the page script needs: each table's level and position, and each edge's two tables and columns, so the script can collapse tables and stack the boxes again, filter by direction and depth, and highlight every path through a column.

MAX_ROWS module-attribute

MAX_ROWS = 30

Columns listed per table before the rest are summed up; the focus table lists all.

MAX_TABLES module-attribute

MAX_TABLES = 200

Tables drawn at most; the farthest levels are left out beyond it, and the page says so.

ROW_MARKS module-attribute

ROW_MARKS = {'writes': '→', 'reads': '←', 'checks': '✓'}

How a DAG box marks the tables its task writes, reads and checks with business rules.

DEPENDENCY_STYLES module-attribute

DEPENDENCY_STYLES = {'SUCCESS': 'success', 'FAILURE': 'failure', 'ALWAYS': 'always'}

The line each dependency type is drawn with; HAS_DATA is has-data.

Node dataclass

A table in the drawing.

height property

height: float

The box's height: header, one row per column, one for the hidden ones.

row_y

row_y(column: str) -> float

Return the vertical middle of a column's row, from the top of the drawing.

LineageDrawing dataclass

A laid-out graph: its tables, column edges, table-only edges and size.

levels property

levels: tuple[int, int]

The furthest upstream and downstream levels drawn.

DagDrawing dataclass

A pipeline's tasks laid out by dependency, with the tables each one reads and writes.

lineage_drawing

lineage_drawing(catalog: Catalog, focus: str) -> LineageDrawing

Lay out the lineage of table focus, as far up and down as it goes.

render_svg

render_svg(drawing: LineageDrawing, link: Callable[[str], str]) -> str

Return the drawing as SVG; link gives the page URL of a table by name.

dag_drawing

dag_drawing(catalog: Catalog, pipeline: str) -> DagDrawing

Lay out the tasks of pipeline: each after the tasks it waits for, left to right.

A task is placed one column after the furthest task of its pipeline it depends on; the tasks of other pipelines it waits for sit in a column of their own on the left. Each box lists the table its task writes, then the tables it reads, then the tables its business rules check.

render_dag_svg

render_dag_svg(drawing: DagDrawing, catalog: Catalog, task_link: Callable[[str], str], table_link: Callable[[str], str]) -> str

Return the DAG as SVG, every task and table a link to its page.