python_scripts

etl_craft.handlers.python_scripts

HANDLER=PYTHON: run an ingestion script from the project's ingestion_scripts/.

The task's SCRIPT_NAME names the script, a path inside ingestion_scripts/; the script defines run(task), or run() when it needs nothing from the task (see etl_craft.scripting). It runs in the task's own process, so what it prints and logs goes to the attempt's log, and the task's time limit applies to it.

Its INPUT_PARAMS task parameter, when set, must be a JSON object; the script gets it as a dictionary. The script gets the offset its last successful run stored, and returns the rows it wrote, recorded as the source, target and insert counts, and optionally a new offset, returned to the child for storage with the successful attempt outcome. A stored offset keeps its type: a script that returns another type fails. A backfill run gets no offset and stores none: the script reads its source for task.run_date. Everything that can be wrong with the script or what it returns fails the task with a message naming the script and the problem.

PARAMETERS module-attribute

PARAMETERS = frozenset({'SCRIPT_NAME', 'INPUT_PARAMS'})

The task parameters an ingestion-script task reads.

SCRIPT_LOGGER module-attribute

SCRIPT_LOGGER = 'etl_craft_script'

The logger a script is given as task.logger, named after its task.

run

run(context: TaskContext, engine_db: Engine) -> HandlerResult

Run the task's ingestion script and return the counts it reports.

parse_input_params

parse_input_params(value: str | None) -> dict[str, Any]

Return INPUT_PARAMS as a dictionary; it must be a JSON object. Absent means empty.

load_script

load_script(path: Path, name: str) -> Callable[..., Any]

Import the script and return its run; HandlerError saying what is wrong.

script_definition_problem

script_definition_problem(path: Path, name: str) -> str | None

Read the script without running it; return why its run cannot be called, or None.

The script is parsed, not imported, so nothing in it runs. A run that is not a plain top-level function (imported, or assigned) is taken on trust.

check_result

check_result(result: Any, name: str, stored: StoredOffset | None) -> ScriptResult

Check what the script returned; HandlerError naming each problem.