runner
etl_craft.execution.runner ¶
run --task_code: run one task of a pipeline, in its own process.
The caller selects the task's run by id or key, or by a single non-terminal identity. In
local mode it runs only when it may: not again once SUCCESS or SKIPPED under the run, not
while it is already IN-PROGRESS, and only when enough of its dependencies are satisfied. A
task whose dependencies can never be satisfied under the run is recorded SKIPPED.
In remote mode the orchestrator decides, so the task runs whenever it is told to, with none of
those checks (see remote). Run again after it succeeded, it is a new attempt that skips
nothing; run after the run ended, it reopens the run, which --finalize-only ends again.
The task itself runs in a freshly started interpreter, supervised with its time limit. Its
output goes to the attempt's log file, whose tail is kept in TASK_LOG. When that process
ends without recording an outcome (it crashed, was killed, or ran out of time), the task is
recorded FAILED with the reason. While it runs, the run is watched: once an operator
cancels it (etl-craft cancel), the task process is stopped and the task is CANCELLED.
CHILD_MODULE
module-attribute
¶
The module a task attempt runs as, with python -m.
CANCEL_POLL_SECONDS
module-attribute
¶
How often a running task looks whether an operator cancelled its run.
NO_CROSS
module-attribute
¶
What a preflight that stopped before the cross-pipeline gate reports from it.
BACKFILL_ASSUMED
module-attribute
¶
BACKFILL_ASSUMED = frozenset({DependencyType.SUCCESS, DependencyType.HAS_DATA, DependencyType.ALWAYS})
The dependency types on another pipeline that a backfill counts as met without checking.
TaskOutcome
dataclass
¶
How run --task_code ended: SUCCESS, FAILED or SKIPPED, and why.
reopened is the status of the ended run that force put back IN-PROGRESS to run
the task; the caller ends that run again (pipeline.force_task).
Override
dataclass
¶
An operator's override of the checks before a task runs, with the reason for it.
rerun runs the task again although it already ended SUCCESS or SKIPPED, as a
new attempt that skips nothing, reopening the run if it had ended (--rerun). Otherwise
the task runs without its dependencies being checked (--ignore-dependencies). Neither
checks the task's dependencies or consumes an upstream run, and each is recorded in
AUD_RUN_INTERVENTIONS.
ChildOptions
dataclass
¶
How the task process is started: its module, log settings and kill grace period.
Setting cancel stops a running task process; the pipeline runner sets it when the run is
interrupted. cancel_poll_seconds is how often a running task looks whether an operator
cancelled its run.
run_task ¶
run_task(engine: Engine, config: ConnectorConfig, pipeline_code: str, task_code: str, *, force: bool = False, gate: TrackedGate | None = None, child: ChildOptions | None = None, override: Override | None = None, selector: RunSelector = runlog.ACTIVE_RUN, admission: CrossPipelineCheck | None = None, automatic_retries: bool = True) -> TaskOutcome
Run one task under its pipeline's active run and return how it ended.
force runs the task even when it already succeeded or its dependencies are not met, and
reopens an ended run; it is local mode's override, refused in remote mode, where the task
always runs as the orchestrator says. override is an operator's recorded override (see
Override), also local mode's. Raises MetadataError for an unknown code and
RunStateError when there is no run to bind to.
attempt_log_path ¶
attempt_log_path(config: ConnectorConfig, pipeline_code: str, pipeline_run_id: int, task_code: str, attempt: int) -> Path
Return the log file of one task attempt, under Orchestration.Log_dir.
run_cancelled ¶
Whether an operator cancelled pipeline_run_id; a failed look counts as no.