text
etl_craft.core.text ¶
Text parsing shared across the layers: JDBC URLs, secrets files, SQL text and identifiers.
Everything here works on strings alone; reading files and opening connections belong to the callers.
LINEAGE_RUN_DATE
module-attribute
¶
The $$run_date lineage and validation read a task's SELECT with: fixed, so a task's
lineage does not look changed every day.
WRITE_KEYWORDS
module-attribute
¶
WRITE_KEYWORDS = ('INSERT', 'UPDATE', 'DELETE', 'MERGE', 'TRUNCATE', 'DROP', 'ALTER', 'CREATE', 'GRANT', 'REVOKE', 'COPY')
Statements a read-only SELECT has no business containing.
METADATA_CODE_PATTERN
module-attribute
¶
Pipeline and task codes: one leading ASCII letter, at most 128 characters.
JdbcUrl
dataclass
¶
The parts of a jdbc:<scheme>://host[:port]/database[?query] URL.
database is the whole path, which some warehouses write as catalog/schema.
catalog
property
¶
The first segment of the path: the catalog in a catalog/schema path.
is_secret_name ¶
Whether a name contains a credential label that must not expose its value.
public_url_query ¶
Keep driver settings while omitting credential-like keys from logged URLs.
jdbc_scheme ¶
Return the lower-cased vendor of a jdbc:<vendor>:... URL.
Raises ConfigurationError for anything that does not start that way.
parse_jdbc_url ¶
Split a jdbc:<scheme>://host[:port]/database[?query] URL into its parts.
The query string is kept, so settings such as sslmode=require reach the driver.
default_port fills in a missing port. Raises ConfigurationError for another shape;
vendors with their own URL form parse it in their dialect.
parse_env_file ¶
Parse .env text: one KEY=VALUE per line.
Blank lines, lines starting with # and lines without = are skipped. Keys and values
are stripped, and one matching pair of quotes around a value is removed. There are no escape
sequences, no multi-line values and no trailing comments.
unquote ¶
Remove one matching pair of wrapping quotes, " or ', and nothing else.
split_statements ¶
Split SQL on semicolons outside quoted strings, identifiers, bodies and comments.
as_subquery ¶
Return select_sql in parentheses, ready to embed in a larger statement.
A trailing ; is dropped, and each parenthesis is on its own line, so a SELECT that ends
in a -- comment does not comment out the closing one.
is_only_comments ¶
Whether statement holds nothing but comments and whitespace.
substitute_task_tokens ¶
substitute_task_tokens(sql: str, *, pipeline_run_id: int, refresh_type: str, pipeline_run_id_substitution: bool, filter_enabled: bool, source: str = 'SOURCE_SQL', force_all: bool = False, run_date: date = LINEAGE_RUN_DATE, run_date_substitution: bool = False, pipeline_id: int = 0, task_run_id: int = 0, pipeline_id_substitution: bool = False, task_run_id_substitution: bool = False) -> str
Replace execution-identity and run-date tokens, each only when its switch is on.
$$pipeline_id,$$pipeline_run_idand$$task_run_idbecome their named ids, enabled byPIPELINE_ID_SUBSTITUTION,PIPELINE_RUN_ID_SUBSTITUTIONandTASK_RUN_ID_SUBSTITUTIONrespectively.$$pipeline_run_id_filterbecomespipeline_run_id = <id>, or1=1for aFULLrefresh or whenforce_allasks for every row, whenfilter_enabledis on (the task'sPIPELINE_RUN_ID_FILTER).$$run_datebecomes the date the run runs as of, asDATE 'YYYY-MM-DD', whenrun_date_substitutionis on (the task'sRUN_DATE_SUBSTITUTION).
Raises HandlerError, naming source, for any other $$ token, for a token whose
switch is off, and for a switch that is on with its token absent: each is a mistake in the
task's definition, and running the SQL anyway would read the wrong rows. The id is an
integer the engine resolved, so it is written into the text directly.
strip_comments_and_literals ¶
Replace comments with a space and string literals with ''.
read_only_problem ¶
Return why sql does not look like a read-only SELECT, or None when it does.
The SQL must start with SELECT, WITH, TABLE or VALUES and contain none of
WRITE_KEYWORDS as a whole word outside comments and string literals, which catches a
data-modifying CTE such as WITH x AS (DELETE ... RETURNING *) SELECT .... This is a lint
for honest mistakes, not a security boundary: CFG_ rows are reviewed, and a determined
author can defeat any string check.
is_metadata_code ¶
Whether a code is safe in commands and cannot collide with control steps.
is_safe_identifier ¶
Whether name can be written unquoted into SQL and into a shell command.
Pipeline codes, task codes, column names and catalog names are interpolated as they are, so only letters, digits and underscores are allowed, not starting with a digit.
is_safe_object_ref ¶
Whether object_ref is schema.table or database.schema.table, safe identifiers.
is_safe_order_term ¶
Whether term is one ORDER BY item: a column, then optional direction and nulls.
For example updated_at DESC NULLS LAST. Expressions are not accepted.
split_object_ref ¶
split_object_ref(object_ref: str, *, param_name: str = 'TARGET_OBJECT') -> tuple[str | None, str, str]
Split schema.table or database.schema.table into (database or None, schema, table).
Raises HandlerError for anything else: a bare table name has no schema.
qualify ¶
Return database.schema.table for a table reference.
A reference that names its database is kept as written; schema.table gets catalog,
the active warehouse profile's database, so one row resolves to the development, test or
production table depending on the environment.
sha256_hex ¶
Return the SHA-256 hex digest of payload; migrations record it to detect edits.
suggest ¶
Return the candidates closest to unknown, for a "did you mean" hint.
Case-insensitive, since codes are routinely typed in the wrong case; candidates starting
with unknown follow the close matches.