migrations

etl_craft.engine.migrations

migrate: bringing an existing Engine DB up to date.

Two ordered streams of *.sql files: the packaged ENGINE stream, which always runs first, and the team's optional PROJECT stream. SCHEMA_MIGRATIONS records each applied file by stream, filename and SHA-256. Files are applied in filename order within their stream, each in its own transaction together with its ledger row, so a failure rolls both back and stops before any later file. Before anything runs, every applied file must still be present and unchanged: a released migration file is never edited.

MigrationFile dataclass

One migration file, read once: its stream, path, SQL and checksum.

version property

version: str

The file's name, which identifies it within its stream.

Stream dataclass

The migration files of one stream, in the order they apply.

resolve_project_migrations_dir

resolve_project_migrations_dir(explicit: Path | str | None = None, project_default: Path | None = None) -> Path | None

Return the project migrations directory, or None when there is no project stream.

--migrations-dir first, then $ETL_CRAFT_MIGRATIONS_DIR, then project_default (migrations/ in the project directory) when that directory exists.

read_stream

read_stream(source: str, directory: Path) -> Stream

Read every *.sql file in directory, in filename order.

migration_streams

migration_streams(engine: Engine, project_dir: Path | str | None = None, project_default: Path | None = None) -> list[Stream]

Return the packaged ENGINE stream, followed by the PROJECT stream when there is one.

verify_ledger

verify_ledger(ledger: dict[tuple[str, str], str], streams: list[Stream]) -> None

Raise MigrationError unless every applied file is present and unchanged.

pending_migrations

pending_migrations(engine: Engine, project_dir: Path | str | None = None, *, project_default: Path | None = None) -> list[str]

Return the migrations migrate would apply, without applying any.

Raises MigrationError as migrate would, for an applied file that is missing or was edited.

apply_pending_migrations

apply_pending_migrations(engine: Engine, project_dir: Path | str | None = None, *, project_default: Path | None = None, wait_seconds: float = 0) -> list[str]

Apply every pending ENGINE, then PROJECT, migration; return the filenames applied.

project_dir is --migrations-dir; project_default is the project's migrations/, used when neither it nor $ETL_CRAFT_MIGRATIONS_DIR is given.

Concurrent runs are serialized by the migrate lock, so a second one waits and then finds nothing left to do.

mark_packaged_migrations_applied

mark_packaged_migrations_applied(conn: Connection) -> list[str]

Record the packaged ENGINE migrations as applied without running them.

init-db calls this: the packaged schema already includes every packaged migration. A project's migrations are not in the schema, so they stay pending. The caller holds the MIGRATE lock and owns the transaction that creates the schema and records this ledger.