validate

etl_craft.services.validate

validate: check every active pipeline's metadata, reporting every problem at once.

A run finds a mistake in a task's definition only when it reaches that task, and some mistakes not even then: an alert that runs before the tasks it reports on, or a HAS_DATA dependency on a task that never reports rows. validate reads the Engine DB and the project's files, and checks each active pipeline and task with the same code a run uses. It connects to nothing else and runs nothing: SQL is read and checked, ingestion scripts are parsed, not imported.

Each finding is a FAIL (a run would fail, hang or silently do something else) or a WARN (it works, but look at this, such as a parameter no handler reads). In remote mode, every rule the orchestrator does not support is a FAIL (see execution.remote).

ROWS_REPORTED module-attribute

ROWS_REPORTED = frozenset(WRITERS)

The SQL actions that report the rows in their target, which a HAS_DATA edge needs.

FULL_REPLACE module-attribute

FULL_REPLACE = frozenset({SqlAction.CREATE_TABLE, SqlAction.OVERWRITE_TABLE})

The SQL actions that rebuild every row of their target, and so every ROW_ID.

KNOWN_PARAMETERS module-attribute

KNOWN_PARAMETERS: dict[str, frozenset[str]] = {Handler.SQL: SQL_PARAMETERS, Handler.BUSINESS_RULES: frozenset(), Handler.EMAIL_ALERT: email_alert.PARAMETERS}

The parameters each handler reads. A PYTHON task's script may read any, so none is unknown.

CHECKS module-attribute

CHECKS: dict[str, Callable[[Connection, TaskContext, _Metadata], list[str]]] = {Handler.SQL: _sql, Handler.BUSINESS_RULES: _business_rules, Handler.PYTHON: _python, Handler.EMAIL_ALERT: _email_alert}

Each handler's checks; each raises or returns the problems it finds.

Finding dataclass

One problem: where it is (a pipeline, or PIPELINE.TASK), how bad, and what it is.

Report dataclass

What validate checked and found.

failed property

failed: bool

Whether any finding is a FAIL.

fail

fail(where: str, message: str) -> None

Record a problem that fails, hangs or silently changes a run.

warn

warn(where: str, message: str) -> None

Record something that works but deserves a look.

validate

validate(engine: Engine, config: ConnectorConfig, pipeline_code: str | None = None) -> Report

Check every active pipeline, or only pipeline_code; return every finding.