catalog_site

etl_craft.services.catalog_site ¶

The catalog as a static site: a page per asset, a search index, and the lineage graphs.

Pages are plain HTML that read well without JavaScript; catalog.js adds search as you type, the full results with filters by kind on the home page, and the lineage graph's direction and depth filters and column tracing. Everything the site needs is in its folder: no page loads anything from elsewhere, so it can be opened from disk, served from any web server, or published with publish-docs.

The site is written to a folder of its own, marked with .etl-craft-catalog: a folder that holds other files is refused, and a folder written before is replaced. The new site is written beside it first and swapped in once complete, so a server publishing the folder never serves a site half written. Every page says when it was generated, and warns once its run details are more than a day old.

MARKER `module-attribute` ¶

MARKER = '.etl-craft-catalog'

The file that marks a folder as a catalog site generate-docs may write over.

TABS `module-attribute` ¶

TABS = (('search', 'index.html', 'Search'), ('dags', 'dags.html', 'DAGs'), ('warehouse', 'warehouse.html', 'Warehouse'))

The tabs at the top of every page: key, page, label.

DEFAULT_DEPTH `module-attribute` ¶

DEFAULT_DEPTH = 3

Levels shown each way when a page opens, if the graph is deeper; the rest are a click away.

Site `dataclass` ¶

What was written: the folder and the number of pages.

write_site ¶

write_site(catalog: Catalog, config: ConnectorConfig, folder: Path) -> Site

Write the catalog site into folder; UsageError if it holds anything else.

The site is written into a hidden folder beside folder, then swapped in.

slug ¶

slug(key: str) -> str

Return a file name for key: itself when safe, else made safe with a short hash.

table_url ¶

table_url(name: str) -> str

Return the URL of a table's page, from the site's root.

task_url ¶

task_url(label: str) -> str

Return the URL of a task's page, from the site's root.

pipeline_url ¶

pipeline_url(code: str) -> str

Return the URL of a pipeline's page, from the site's root.

rule_url ¶

rule_url(rule_id: int) -> str

Return the URL of a business rule's page, from the site's root.

script_url ¶

script_url(name: str) -> str

Return the URL of an ingestion script's page, from the site's root.

run_url ¶

run_url(pipeline_run_id: int) -> str

Return the URL of a pipeline run's page, from the site's root.