gates
etl_craft.execution.gates ¶
Cross-pipeline dependencies: waiting for the upstream, judging its last run, and trackers.
A pipeline's dependencies on other pipelines (CFG_PIPELINE_DEPENDENCY) are checked before a
new run of it starts; a task's dependencies on tasks in other pipelines (CFG_TASK_DEPENDENCY
rows naming another pipeline) are checked before the task runs. Each check:
- Waits while the upstream's latest run is
IN-PROGRESS. The first look is at 70% of the upstream's average run length, then 80%, 90% and so on; one check waits at mostOrchestration.Gate_wait_minutes(an hour unless set) and looks at most 30 times, across all of its dependencies. - Judges the upstream's latest finished run. The dependency is satisfied only when that run has a newer run id or an allowed newer published revision, and its status satisfies the dependency type. An older run that would have satisfied it does not count: the last run decides.
- Once the downstream succeeds, logs that run as consumed in
AUD_DEPENDENCY_CONSUMPTION, one row per downstream run (or task), dependency, upstream run and revision; the dependency's latest row is what it last consumed. A downstream that fails or is skipped consumes nothing, so its retry sees the same upstream run.
Runs that are part of a backfill are invisible to all of this: they never satisfy a dependency, never fail one, and are never waited for.
Orchestration.Dependency_gates relaxes this in local mode: with warn a dependency that is
not satisfied is reported as bypassed and the run or task goes ahead; with off nothing is
checked or waited for, and every dependency is reported as bypassed. Only upstream runs that
satisfied their dependency are consumed.
FIRST_LOOK_FRACTION
module-attribute
¶
The first look at a running upstream is at this fraction of its average run length.
LOOK_FRACTION_STEP
module-attribute
¶
Each later look is this much more of the average run length.
MAX_LOOKS
module-attribute
¶
How many times one check looks at running upstreams, across all its dependencies.
WAIT_LIMIT_SECONDS
module-attribute
¶
How long one check waits for running upstreams, across all its dependencies, unless
Orchestration.Gate_wait_minutes says otherwise.
MIN_LOOK_INTERVAL_SECONDS
module-attribute
¶
The shortest pause between two looks, for an upstream already past its expected length.
DEFAULT_RUN_SECONDS
module-attribute
¶
The run length assumed for an upstream with no finished run to average.
Clock
dataclass
¶
How a check sleeps and reads the time; tests replace both.
WaitBudget
dataclass
¶
CrossPipelineCheck
dataclass
¶
What a gate found for a task's dependencies on tasks in other pipelines.
consumed maps each satisfied dependency to the upstream task run that satisfied it, for
the gate to record once the task succeeds. definitive is false when the gate did not
really check, so its reasons never record the task SKIPPED. bypassed holds the
dependencies Dependency_gates let through without being satisfied.
TrackedGate ¶
The cross-pipeline gate for tasks, backed by AUD_DEPENDENCY_CONSUMPTION.
__init__ ¶
__init__(clock: Clock | None = None, policy: GatePolicy = GatePolicy.ENFORCE, wait_seconds: float = WAIT_LIMIT_SECONDS) -> None
Wait and read the time with clock, and treat what is not satisfied by policy.
A running upstream is waited for at most wait_seconds.
check ¶
check(engine: Engine, task_id: int, needed: int, *, budget: WaitBudget | None = None, waiter: Callable[..., datetime | None] | None = None) -> CrossPipelineCheck
Check the task's cross-pipeline dependencies until needed are satisfied.
Dependencies after that are neither checked nor waited for. Under warn the ones not
satisfied are bypassed; under off none is checked and all are bypassed.
PipelineGateResult
dataclass
¶
A pipeline's dependencies on other pipelines: the reasons any are unsatisfied.
GateResult
dataclass
¶
One nonblocking check: final judgement or the next persisted look.
next_look_delay ¶
next_look_delay(average_seconds: float, elapsed_seconds: float, fraction: float, remaining_seconds: float) -> float
Return how long to wait before looking at a running upstream again.
The look is due when the upstream has run fraction of its average length; never sooner
than MIN_LOOK_INTERVAL_SECONDS from now, and never past the check's deadline.
satisfies ¶
Whether a finished upstream run satisfies a dependency of dependency_type.
judge ¶
judge(dependency_type: str, run: FinishedRun | None, last_consumed: ConsumedRun | int | None, *, consume_repairs: bool = True) -> tuple[int | None, str]
Return the upstream run that satisfies a dependency, or None and why none does.
Only the latest finished run is judged. A higher run id is new; the same id is new only
with a higher revision and consume_repairs. Its status must satisfy dependency_type.
check_pipeline_dependencies ¶
check_pipeline_dependencies(engine: Engine, pipeline_id: int, clock: Clock | None = None, policy: GatePolicy = GatePolicy.ENFORCE, wait_seconds: float = WAIT_LIMIT_SECONDS, *, budget: WaitBudget | None = None, waiter: Callable[..., datetime | None] | None = None) -> PipelineGateResult
Check every dependency of pipeline_id on other pipelines; all must be satisfied.
The check stops at the first unsatisfied dependency, without waiting on the rest. Under
warn it checks them all and bypasses the ones not satisfied; under off it checks
none and bypasses them all.
check_gate ¶
check_gate(engine: Engine, pipeline_run_id: int, pipeline_id: int, *, task_id: int | None = None, needed: int = 0, clock: Clock | None = None, policy: GatePolicy = GatePolicy.ENFORCE, wait_seconds: float = WAIT_LIMIT_SECONDS) -> GateResult
Check without sleeping or holding a worker; resume the run's durable wait budget.