Command line¶

Every command accepts the global options, before or after the command name. Results are written to standard output; errors (error: ...) and log records go to standard error. The exit codes page lists what each exit status means.

usage: etl-craft [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                 [--log-format {text,json}] [--version]
                 COMMAND ...

Metadata-driven ETL orchestration engine.

positional arguments:
  COMMAND
    run                 Run a pipeline in dependency waves, one task of it, or its first or last
                        step.
    server              supervise active local pipeline runs
    token               create, list or revoke API tokens
    mark                Mark a task or a run SUCCESS, FAILED or SKIPPED, or record a stand-in run;
                        local mode.
    cancel              Stop a pipeline's run in progress and end it CANCELLED; local mode.
    reconcile           Reconcile expired attempt and run leases.
    rehash              Upgrade an existing merge target to hash version 2.
    upgrade-targets     Add pipeline and task-run identities to existing targets.
    pause               Stop a pipeline from running until it is resumed; local mode.
    resume              Let a paused pipeline run again; local mode.
    list                List the active pipelines.
    graph               Show a pipeline's task waves and dependencies.
    steps               List a pipeline's active tasks and their parameters.
    status              Show the selected run and every configured task.
    explain             Explain the selected task's state and what would let it run.
    history             Show a pipeline's recent runs, or one task's.
    audit               Show command requests and metadata changes with their actors.
    lineage             Show where tables and columns come from and where they go.
    docs-version        Record a new version of each task's DOCUMENTATION that changed.
    generate-docs       Write the catalog site: every pipeline, task and table, lineage graphs and
                        search.
    publish-docs        Serve the catalog site until stopped, at an ngrok link or on this machine.
    generate-yml        Write a pipeline's DAG, the global trigger DAG or the docs DAG, as YAML.
    clone               Copy the Engine DB tables Cloning.Scope names into the warehouse schema
                        now.
    validate            Check every active pipeline's tasks, parameters and dependencies; exit 1
                        on a problem.
    doctor              Check the configuration, secrets, connections and Engine DB; exit 1 if any
                        check fails.
    setup               Check the configuration, then create or bring up to date the Engine DB
                        tables.
    init-db             Create the Engine DB schema in an empty database.
    migrate             Apply the pending packaged and project migrations to the Engine DB.

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`run`¶

usage: etl-craft run [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                     [--log-format {text,json}] [--format {text,json}]
                     [--run-id RUN_ID | --run-key RUN_KEY] --pipeline_code PIPELINE_CODE
                     [--task_code TASK_CODE | --init-only | --finalize-only] [--force]
                     [--ignore-dependencies] [--rerun] [--with-downstream] [--skip]
                     [--run-date YYYY-MM-DD] [--backfill FROM:TO] [--reason REASON]

Run a pipeline in dependency waves, one task of it, or its first or last step.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --run-id RUN_ID       the exact pipeline run id
  --run-key RUN_KEY     the exact run key within this pipeline
  --pipeline_code PIPELINE_CODE
                        the pipeline to run
  --task_code TASK_CODE
                        run only this task, under the selected pipeline run
  --init-only           test connections, check the pipeline's dependencies and start its run; an
                        orchestrator's first step
  --finalize-only       end the selected pipeline run from its tasks' statuses; an orchestrator's
                        last step
  --force               run even if tasks already succeeded or their dependencies are not met;
                        local mode only
  --ignore-dependencies
                        with --task_code: run the task without checking its dependencies; local
                        mode only, with --reason, recorded
  --rerun               with --task_code: run the task again although it already succeeded,
                        reopening its run if it ended; local mode only, with --reason, recorded
  --with-downstream     with --rerun: run every task after it again too, in dependency order
  --skip                record a run SKIPPED on purpose, running nothing; local mode only, with
                        --reason
  --run-date YYYY-MM-DD
                        the date a new run runs as of (SQL's $$run_date); today unless given
  --backfill FROM:TO    run the pipeline once for each date from FROM to TO (YYYY-MM-DD), as
                        backfill runs; local mode only, with --reason
  --reason REASON       why, for --ignore-dependencies, --rerun, --skip or --backfill

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`server`¶

usage: etl-craft server [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                        [--log-format {text,json}]

supervise active local pipeline runs

options:
  -h, --help            show this help message and exit

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`token`¶

usage: etl-craft token [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                       [--log-format {text,json}]
                       {create,revoke,list} ...

create, list or revoke API tokens

positional arguments:
  {create,revoke,list}
    create              create a token and print its value once
    revoke              revoke a token immediately
    list                list safe token metadata

options:
  -h, --help            show this help message and exit

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`mark`¶

usage: etl-craft mark [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                      [--log-format {text,json}] [--format {text,json}]
                      [--run-id RUN_ID | --run-key RUN_KEY] --pipeline_code PIPELINE_CODE
                      [--task_code TASK_CODE] --status {SUCCESS,FAILED,SKIPPED} --reason REASON
                      [--rows ROWS] [--stale] [--new-run]

Mark a task or a run SUCCESS, FAILED or SKIPPED, or record a stand-in run; local mode.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --run-id RUN_ID       the exact pipeline run id
  --run-key RUN_KEY     the exact run key within this pipeline
  --pipeline_code PIPELINE_CODE
                        the pipeline whose run to mark
  --task_code TASK_CODE
                        mark this task of the run; without it, the run itself
  --status {SUCCESS,FAILED,SKIPPED}
                        the status
  --reason REASON       why; recorded with the change
  --rows ROWS           with SUCCESS, the row count a HAS_DATA dependency on the task reads
  --stale               with --task_code: reconcile expired attempts before marking; live leases
                        still refuse the mark
  --new-run             record a finished stand-in run instead, so downstream gates pass where
                        this pipeline cannot run

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`cancel`¶

usage: etl-craft cancel [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                        [--log-format {text,json}] [--format {text,json}]
                        [--run-id RUN_ID | --run-key RUN_KEY] --pipeline_code PIPELINE_CODE
                        --reason REASON

Stop a pipeline's run in progress and end it CANCELLED; local mode.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --run-id RUN_ID       the exact pipeline run id
  --run-key RUN_KEY     the exact run key within this pipeline
  --pipeline_code PIPELINE_CODE
                        the pipeline whose run to cancel
  --reason REASON       why; recorded with the change

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`reconcile`¶

usage: etl-craft reconcile [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                           [--log-format {text,json}] [--format {text,json}]
                           [--pipeline_code PIPELINE_CODE] [--task_code TASK_CODE]

Reconcile expired attempt and run leases.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --pipeline_code PIPELINE_CODE
                        limit reconciliation to this pipeline
  --task_code TASK_CODE
                        limit reconciliation to this pipeline's task

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`rehash`¶

usage: etl-craft rehash [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                        [--log-format {text,json}] --target TARGET [--dry-run]

Upgrade an existing merge target to hash version 2.

options:
  -h, --help            show this help message and exit
  --target TARGET       schema.table (or database.schema.table)
  --dry-run             validate and show the UPDATE without writing

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`upgrade-targets`¶

usage: etl-craft upgrade-targets [-h] [--config PATH]
                                 [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                                 [--log-format {text,json}]
                                 [--action {CREATE_TABLE,SETUP_TABLE,OVERWRITE_TABLE,APPEND_TABLE,SCD1_MERGE,SCD2_MERGE,DELETE_ROWS}]
                                 [--target TARGET] [--dry-run]

Add pipeline and task-run identities to existing targets.

options:
  -h, --help            show this help message and exit
  --action {CREATE_TABLE,SETUP_TABLE,OVERWRITE_TABLE,APPEND_TABLE,SCD1_MERGE,SCD2_MERGE,DELETE_ROWS}
                        filter by SQL action; defaults to configured SQL and ingestion targets
  --target TARGET       one schema.table (or database.schema.table); defaults to all selected
                        targets
  --dry-run             validate targets and show additions without writing warehouse rows

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`pause`¶

usage: etl-craft pause [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                       [--log-format {text,json}] [--format {text,json}] --pipeline_code
                       PIPELINE_CODE --reason REASON

Stop a pipeline from running until it is resumed; local mode.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --pipeline_code PIPELINE_CODE
                        the pipeline to pause
  --reason REASON       why; recorded with the pause

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`resume`¶

usage: etl-craft resume [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                        [--log-format {text,json}] [--format {text,json}] --pipeline_code
                        PIPELINE_CODE --reason REASON

Let a paused pipeline run again; local mode.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --pipeline_code PIPELINE_CODE
                        the pipeline to resume
  --reason REASON       why; recorded with the resume

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`list`¶

usage: etl-craft list [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                      [--log-format {text,json}] [--format {text,json}]

List the active pipelines.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`graph`¶

usage: etl-craft graph [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                       [--log-format {text,json}] [--format {text,json}] --pipeline_code
                       PIPELINE_CODE

Show a pipeline's task waves and dependencies.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --pipeline_code PIPELINE_CODE
                        the pipeline to show

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`steps`¶

usage: etl-craft steps [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                       [--log-format {text,json}] [--format {text,json}]
                       [--run-id RUN_ID | --run-key RUN_KEY] --pipeline_code PIPELINE_CODE

List a pipeline's active tasks and their parameters.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --run-id RUN_ID       the exact pipeline run id
  --run-key RUN_KEY     the exact run key within this pipeline
  --pipeline_code PIPELINE_CODE
                        the pipeline to show

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`status`¶

usage: etl-craft status [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                        [--log-format {text,json}] [--format {text,json}]
                        [--run-id RUN_ID | --run-key RUN_KEY] --pipeline_code PIPELINE_CODE

Show the selected run and every configured task.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --run-id RUN_ID       the exact pipeline run id
  --run-key RUN_KEY     the exact run key within this pipeline
  --pipeline_code PIPELINE_CODE

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`explain`¶

usage: etl-craft explain [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                         [--log-format {text,json}] [--format {text,json}]
                         [--run-id RUN_ID | --run-key RUN_KEY] --pipeline_code PIPELINE_CODE
                         --task_code TASK_CODE

Explain the selected task's state and what would let it run.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --run-id RUN_ID       the exact pipeline run id
  --run-key RUN_KEY     the exact run key within this pipeline
  --pipeline_code PIPELINE_CODE
  --task_code TASK_CODE

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`history`¶

usage: etl-craft history [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                         [--log-format {text,json}] [--format {text,json}]
                         [--run-id RUN_ID | --run-key RUN_KEY] --pipeline_code PIPELINE_CODE
                         [--task_code TASK_CODE] [--all] [--limit LIMIT]

Show a pipeline's recent runs, or one task's.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --run-id RUN_ID       the exact pipeline run id
  --run-key RUN_KEY     the exact run key within this pipeline
  --pipeline_code PIPELINE_CODE
                        the pipeline to show
  --task_code TASK_CODE
                        show this task's runs instead of the pipeline's
  --all                 list runs instead of selecting one
  --limit LIMIT         how many runs, newest first

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`audit`¶

usage: etl-craft audit [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                       [--log-format {text,json}] [--format {text,json}]
                       [--pipeline_code PIPELINE_CODE] [--since SINCE]

Show command requests and metadata changes with their actors.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --pipeline_code PIPELINE_CODE
                        filter requests and related metadata rows
  --since SINCE         changes since an ISO date or timestamp

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`lineage`¶

usage: etl-craft lineage [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                         [--log-format {text,json}] [--format {text,json}] [--table TABLE]
                         [--column COLUMN] [--upstream | --downstream] [--depth DEPTH] [--refresh]
                         [--strict]

Show where tables and columns come from and where they go.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --table TABLE         schema.table (or database.schema.table) to trace
  --column COLUMN       one column of --table to trace
  --upstream            only where it comes from
  --downstream          only where it goes
  --depth DEPTH         stop after this many steps
  --refresh             work every task's lineage out again
  --strict              exit 1 when any SQL task cannot be traced

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`docs-version`¶

usage: etl-craft docs-version [-h] [--config PATH]
                              [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                              [--log-format {text,json}] [--check]

Record a new version of each task's DOCUMENTATION that changed.

options:
  -h, --help            show this help message and exit
  --check               report what would change, and record nothing

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`generate-docs`¶

usage: etl-craft generate-docs [-h] [--config PATH]
                               [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                               [--log-format {text,json}] [--output DIR] [--strict]
                               [--with-warehouse]

Write the catalog site: every pipeline, task and table, lineage graphs and search.

options:
  -h, --help            show this help message and exit
  --output DIR          the folder to write (default: Docs_site.Output, else catalog/ in the
                        project directory)
  --strict              write nothing and exit 1 when any SQL task's columns cannot be traced
  --with-warehouse      add column types and comments from the warehouse

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`publish-docs`¶

usage: etl-craft publish-docs [-h] [--config PATH]
                              [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                              [--log-format {text,json}] [--output DIR] [--local-only]
                              [--host HOST] [--port PORT] [--accept-new-url]

Serve the catalog site until stopped, at an ngrok link or on this machine.

options:
  -h, --help            show this help message and exit
  --output DIR          the site's folder (default: Docs_site.Output, else catalog/ in the project
                        directory)
  --local-only          serve on this machine or network only, without ngrok
  --host HOST           with --local-only, the address to listen on (default: 127.0.0.1)
  --port PORT           the local port (default: any free one)
  --accept-new-url      publish even though ngrok gives another URL than the one recorded, and
                        record it

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`generate-yml`¶

usage: etl-craft generate-yml [-h] [--config PATH]
                              [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                              [--log-format {text,json}]
                              (--pipeline_code PIPELINE_CODE | --global | --docs)
                              [--airflow-version RANGE] [--output PATH]

Write a pipeline's DAG, the global trigger DAG or the docs DAG, as YAML.

options:
  -h, --help            show this help message and exit
  --pipeline_code PIPELINE_CODE
                        the pipeline whose DAG to write
  --global              write the DAG that triggers pipelines in dependency order (needs
                        Global_dag: true)
  --docs                write the DAG that writes the catalog site again on Docs_site.Schedule
  --airflow-version RANGE
                        target Airflow version or PEP 440 range; remote DAGs require a floor of
                        2.2.0
  --output PATH         write here, not to stdout

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`clone`¶

usage: etl-craft clone [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                       [--log-format {text,json}]

Copy the Engine DB tables Cloning.Scope names into the warehouse schema now.

options:
  -h, --help            show this help message and exit

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`validate`¶

usage: etl-craft validate [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                          [--log-format {text,json}] [--format {text,json}]
                          [--pipeline_code PIPELINE_CODE]

Check every active pipeline's tasks, parameters and dependencies; exit 1 on a problem.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)
  --pipeline_code PIPELINE_CODE
                        check only this pipeline

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`doctor`¶

usage: etl-craft doctor [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                        [--log-format {text,json}] [--format {text,json}]

Check the configuration, secrets, connections and Engine DB; exit 1 if any check fails.

options:
  -h, --help            show this help message and exit
  --format {text,json}  result format (default: text)

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`setup`¶

usage: etl-craft setup [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                       [--log-format {text,json}] [--print-grants]

Check the configuration, then create or bring up to date the Engine DB tables.

options:
  -h, --help            show this help message and exit
  --print-grants        print PostgreSQL role grants without executing them

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`init-db`¶

usage: etl-craft init-db [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                         [--log-format {text,json}] [--force]

Create the Engine DB schema in an empty database.

options:
  -h, --help            show this help message and exit
  --force               apply the schema even though Engine DB tables already exist

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

`migrate`¶

usage: etl-craft migrate [-h] [--config PATH] [--log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
                         [--log-format {text,json}] [--migrations-dir PATH]

Apply the pending packaged and project migrations to the Engine DB.

options:
  -h, --help            show this help message and exit
  --migrations-dir PATH
                        the project's own *.sql migrations (default: $ETL_CRAFT_MIGRATIONS_DIR,
                        then migrations/ in the project directory if it exists)

global options:
  --config PATH         craft-connector.yml to read
  --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        lowest level of log record written to stderr (default: INFO)
  --log-format {text,json}
                        log records as text lines or JSON objects (default: text)

Command line¶

run¶

server¶

token¶

mark¶

cancel¶

reconcile¶

rehash¶

upgrade-targets¶

pause¶

resume¶

list¶

graph¶

steps¶

status¶

explain¶

history¶

audit¶

lineage¶

docs-version¶

generate-docs¶

publish-docs¶

generate-yml¶

clone¶

validate¶

doctor¶

setup¶

init-db¶

migrate¶