Skip to content

business_rules

etl_craft.handlers.business_rules

HANDLER=BUSINESS_RULES: check a task's CFG_BUSINESS_RULES and flag the rows that fail.

A rule names a warehouse table (TARGET_TABLE, schema.table or database.schema.table), the column that identifies a row (BUSINESS_RULE_KEY_COLUMN) and a condition (BUSINESS_RULE_SQL): a correlated SELECT that returns a row when the table's row t breaks the rule, such as SELECT 1 FROM sales.customers c WHERE c.id = t.customer_id AND c.active = 'N'. For each rule the engine:

  1. finds the keys of the rows in scope that break it: SELECT DISTINCT t.<key> FROM <table> t WHERE <scope> AND EXISTS (<rule>);
  2. flags each such key not already flagged, as a row of AUD_BUSINESS_RULES_RESULTS with the rule's BUSINESS_RULE_TYPE;
  3. clears the flags of keys it flagged before whose rows in scope no longer break it, and of keys that no longer have a row in the table at all.

The scope is the rows the current run wrote (t.PIPELINE_RUN_ID = <run>), or every row when the task is forced; on a table with ACTIVE_FLAG (an SCD2_MERGE target) only active versions are judged. A forced run judges the whole table, so every flag it does not find again is cleared. Rules run in waves by SEQUENCE_NUMBER: the rules of a wave run in parallel, at most Orchestration.Max_parallel_tasks at once, and the next wave starts once every rule of this one has finished. A rule that fails is recorded FAILED and the rest of its wave still runs; then the task fails, naming every rule that failed. A rule that already succeeded under the task run is not run again, unless the task itself is run again after it succeeded.

Each rule records its outcome in AUD_BUSINESS_RULES_RUN_LOG in a transaction of its own.

KEY_CHUNK module-attribute 

KEY_CHUNK = 500

How many flagged keys one clearing query checks at a time.

RuleOutcome dataclass

What one rule did: the keys it flagged and cleared, or that it had already run.

run 

run(context: TaskContext, engine_db: Engine) -> HandlerResult

Run every active business rule of the task, wave by wave; return flagged and cleared.

check_rule 

check_rule(rule: BusinessRule, catalog: str) -> _Rule

Check a rule's definition before any rule runs; SqlGuardError naming what is wrong.