Rule severity

import dataframely as dy
import polars as pl

import dagster_dataframely as dd

Every rule has a severity, "error" or "warn", as a dbt test does. A rule is "error" unless the schema declares "warn".

severity	a row that fails the rule	the rule’s check
"error"	is not written to the table: the run fails, or with filter=True the package writes the row under <name>__error_rows	fails at ERROR
"warn"	is written to the table, unless it also fails an "error" rule	fails at WARN

Validation shows what each kind of run writes, and Filtering error rows shows the error rows.

Declaring a severity

The schema declares each severity, so every asset that uses the schema has the same severities:

class LenientOrdersSchema(dy.Schema):
    order_id = dy.String(
        primary_key=True,
        regex=r"^ORD-\d{4}$",
        metadata={"dagster_dataframely": {"severity": {"regex": "warn"}}},
    )
    amount = dy.Float64(nullable=False, min=0.0)
    note = dy.String(
        nullable=True,
        max_length=100,
        metadata={"dagster_dataframely": {"severity": "warn"}},
    )

    __dagster_dataframely__ = {"severity": {"amount_is_rounded": "warn"}}

    @dy.rule()
    def amount_is_rounded(cls) -> pl.Expr:
        """Require whole cents."""
        return (cls.amount.col * 100).round() == cls.amount.col * 100

A column rule’s severity goes in the column’s metadata, under the key dagster_dataframely. Its severity is one value for every rule on the column, as on note, or a mapping from rule name to value, as on order_id. Dataframely does not read metadata, so the key does not change how Dataframely validates.

A schema-level rule’s severity goes in the class attribute __dagster_dataframely__. A schema-level rule is primary_key or a @dy.rule(), and neither has a column to declare it on. The attribute takes the same two forms: one value for every schema-level rule, or a mapping.

A subclass has its parent’s declarations. The Columns tab does not show the dagster_dataframely key as a tag.

Rule names

A severity mapping names Dataframely’s rule name: the part after the | in the check’s dataframely/rule metadata, such as regex in order_id|regex. Four column arguments generate a rule under a different name, and the package reads the argument as that rule:

you can write	the rule it names
nullable	nullability
allow_inf	inf
allow_nan	nan
check, when check= is a list or a dict	every check__* rule on the column

A dy.List or dy.Array adds inner_ to the rule names of its inner column, such as inner_nullability. A dy.Struct adds inner_<field>_ to the rule names of each field, such as inner_email_nullability. The arguments in the table do not name those rules, so dd.asset raises on inner_nullable.

dd.asset raises on a declaration it cannot read

dd.asset checks the declarations when you declare the asset, and raises InvalidSeverityError on one it cannot read:

class MisspelledSchema(dy.Schema):
    amount = dy.Float64(
        min=0.0, metadata={"dagster_dataframely": {"severity": {"minimum": "warn"}}}
    )


try:
    dd.asset(MisspelledSchema)
except dd.errors.InvalidSeverityError as error:
    print(error)

The `dagster_dataframely` metadata of column 'amount' in MisspelledSchema names 'minimum', which is not a rule of column 'amount'. The rules of column 'amount' are 'nullability', 'min', 'inf', 'nan'.

It raises for these mistakes:

• a dagster_dataframely entry or a __dagster_dataframely__ that is not a mapping;
• a key other than severity under dagster_dataframely or in __dagster_dataframely__;
• a value other than "error", "warn" or a mapping of rule names to those, including dg.AssetCheckSeverity.WARN;
• a name that is not one of the column’s rules;
• one value for every rule, on a column that has no rules or in __dagster_dataframely__ on a schema with no schema-level rules;
• a column rule, such as amount|min, in __dagster_dataframely__;
• a schema-level rule that does not exist.

Blocking checks

With filter=False, the default, a check whose rules include an "error" rule is blocking, like the column-schema check. With filter=True, no rule’s check is blocking.

The check specs of orders show which checks are blocking:

@dd.asset(LenientOrdersSchema)
def orders() -> pl.DataFrame:
    return pl.DataFrame()


{spec.name: spec.blocking for spec in orders.check_specs}

{'dy_schema__columns': True,
 'dy_rule__amount_is_rounded': False,
 'dy_rule__primary_key': True,
 'dy_rule__order_id__nullability': True,
 'dy_rule__order_id__regex': False,
 'dy_rule__amount__nullability': True,
 'dy_rule__amount__min': True,
 'dy_rule__amount__inf': True,
 'dy_rule__amount__nan': True,
 'dy_rule__note__max_length': False}

The Checks tab marks each blocking check. Dagster does not execute the assets downstream in the same run when a blocking check fails at ERROR. A failing check also sets the asset’s health: “Degraded” for a check at ERROR, and “Warning” for a check at WARN.

Collapsed checks

With check_granularity="column", one check reports for all of a column’s rules (Settings):

• The check is blocking when filter=False and any of its rules is "error".
• It fails at ERROR when a failing rule is "error", and at WARN when only "warn" rules failed.

So a collapsed check whose "warn" rule failed and whose "error" rule passed fails at WARN, and Dagster does not fail the step. Its metadata still lists the failures of each rule.