Naming

Setup: the demo’s Orders schema, which every example on this page uses

import dagster as dg
import dataframely as dy
import polars as pl

import dagster_dataframely as dd
from dagster_dataframely_demo.schema import Orders

This page covers four things this package sets: the asset’s description, each check’s description, the name of the asset’s op, and the namespaces of the names and keys it generates.

The description comes from the schema

dd.asset takes the asset’s description from the first of these you set: the description= argument, the schema’s docstring, and the decorated function’s docstring, which is Dagster’s own default.

class Orders(dy.Schema):
    """A customer order line: one row per product on one order."""


@dd.asset(Orders)
def orders() -> pl.DataFrame:
    """Joins the two extracts and drops the test accounts."""
    ...

The catalog shows A customer order line: one row per product on one order. as the description. The schema’s docstring takes precedence over the function’s, because it describes the table, not the code. dd.asset reads the schema’s own docstring only, and ignores one inherited from a parent class, including dy.Schema’s.

The spec from quarantine_spec has its own description: Invalid rows from <asset>, with one column per rule.

Where a check’s text comes from

The Dagster UI shows a schema’s rules in three places, and each has its own fallback order.

place	first choice	then	then
Columns tab	the rendered column constraint	the rule name	never the docstring
check name	`dy_rule__<rule>` at `rule` granularity, `dy_col__<column>` or `dy_schema__rules` at the others	never the docstring
check description	the rule’s docstring	`<column> <rendered constraint>`	the whole name Dataframely reports, column part included
collapsed check description	each rule’s rendered constraint	the rule name	never the docstring

The package renders a rule’s value, such as the bound of min, in the column constraint and in each check result’s dy_rule__expr metadata, but never in the check name. So changing a bound does not rename the check or start a new check history.

The package renders column constraints as operators: >= 0.0, length <= 64 bytes, matches ^[a-z]+$, in (a, b), aligned to 1h. It omits nullability and unique, because Dagster has separate fields for them on the column. It omits inf and nan too, because Dataframely adds them to every float column by default. All four still appear in the check name and the check description.

A check= with a bare lambda has no name, so the package renders it as custom check everywhere. Name your checks with a dict, such as check={"lowercase": ...}, and the package shows the key instead.

Dagster names the op after the whole asset key

@dd.asset(Orders, key_prefix="sales", name="orders") creates an op named sales__orders, the same way @dg.asset names its op. An op name must be unique within a code location, and an asset name need not be. With the asset name alone, two assets named orders under different prefixes would have two ops with the same name.

The op name is also the step key and the key under ops: in run config, so both use the whole asset key:

ops:
  sales__orders:
    config:
      threshold: 4

The reserved namespaces

There are three.

dy_ starts every check name, every rule column in the quarantine, and every check metadata key except dataframely/quarantine_address. These are the column-schema check dy_schema__columns, each rule’s check dy_rule__<rule>, the collapsed checks dy_col__<column> and dy_schema__rules, and the metadata keys dy_rule, dy_rule__expr, dy_rules, dy_failed_count, dy_failed_sample and dy_schema__errors. A check name becomes an op output name, which Dagster checks against ^[A-Za-z0-9_]+$. So a check name cannot contain a slash.

A rule’s check name is dy_rule__ plus the name Dataframely gives the rule, with | replaced by __: amount|min becomes dy_rule__amount__min. dd.wiring.check_name("amount|min") returns that check name, so a test does not have to build it by hand.

dataframely/ starts every materialization metadata key this package names. A metadata key can contain a slash, so this namespace has the same form as Dagster’s own dagster/. This package’s keys then sort together, apart from Dagster’s keys and your IO manager’s.

dataframely/quarantine_address is the one dataframely/ key that also appears on check results. When every row fails validation, the run raises NoValidRowsError and has no materialization. So each check result has the quarantine address under that key.

Every public function that takes a schema raises three errors for names this package cannot use (ADR-0008). It raises ReservedColumnError for a column name that starts with dy_. It raises InvalidColumnNameError for a column name with characters Dagster does not allow in a check name, which is anything outside A-Za-z0-9_. It raises CheckNameCollisionError for two rules that produce the same check name. So dd.asset raises them where you declare the asset, and a hand-wired asset raises them in the first of these functions it calls.

A column name with such characters almost always comes from dy.Column(alias=...), which Dataframely provides for a name that is not a Python identifier. A | is one of them, and Dataframely also uses it to separate a column from its rule name.

<name>_quarantine is the third: the asset key a writer writes the invalid rows under. You also choose asset keys in this key space, so another asset can already have this key. Before the decorated function runs, a run of an asset with quarantine=True checks that no other asset in the code location materializes this key. If one does, the run raises QuarantineKeyCollisionError.

No setting changes any of the three, so they are the same in every project.