Errors

import dagster as dg
import dataframely as dy
import polars as pl

import dagster_dataframely as dd
from dagster_dataframely_demo.schema import Orders

Every error this package raises is in dd.errors and subclasses dd.errors.DagsterDataframelyError. Catch one by name, or catch DagsterDataframelyError to catch every one. Every message names what failed and how to fix it.

error	raised	what to do
CollectionNotSupportedError	at decoration	pass a dy.Schema; declare one asset per Collection member, each with the member’s own schema
ReservedColumnError	at decoration, and by every public function that takes a schema	rename the column that starts with dy_
InvalidColumnNameError	the same	rename the column, or change the alias= that sets its name, so the name uses only A-Za-z0-9_, the characters Dagster allows in a check name
CheckNameCollisionError	the same	rename one of the two rules that produce the same check name
InvalidSettingError	when the package resolves the setting	fix the value at the source the message names
MaterializeResultValueError	before the column-schema check	set value= to the frame; or return the frame and call context.add_asset_metadata; or write a plain @dg.asset and call dd.wiring.schema_metadata
MaterializeResultFieldError	the same	remove asset_key= or check_results=, which the decorator sets itself
ColumnSchemaError	after the column-schema check fails	fix the function that produced the frame, or cast with Schema.cast in the asset body
ValidationAbortError	after Schema.filter, when rows failed and the asset declares no quarantine	fix the rows upstream, write them to a quarantine with quarantine=True, or drop them in the asset body
NoValidRowsError	after Schema.filter, when every row failed validation and the asset declares a quarantine	read the invalid rows at the quarantine address in the message
QuarantineKeyCollisionError	before the decorated function runs, on every run of an asset with quarantine=True	rename the other asset, or remove quarantine=True; if the other asset is your quarantine table, delete it and use quarantine_spec
QuarantineDirError	under direct invocation only, when there are invalid rows to write	set DAGSTER_DATAFRAMELY_QUARANTINE_DIR, or run the asset

ColumnSchemaError names every mismatched column at once, with its expected and actual dtype. The failing check’s dy_schema__errors table lists the same columns.

A run log where the quantity column is Int64. The failing dy_schema__columns check has dy_schema__errors, which shows the expected and actual dtype. The step failure below it names the same column in the ColumnSchemaError message.

ValidationAbortError and NoValidRowsError both give the failure count per rule. The counts can add up to more than the number of invalid rows, because one row can fail several rules.

Three ways to get this wrong

Do not declare an asset keyed <name>_quarantine if <name> has quarantine=True. orders with quarantine=True writes its invalid rows to the asset key orders_quarantine, through the same IO manager that writes orders. An asset of your own keyed orders_quarantine has the same address. Both assets would write to the same table or file, and the second write would replace the first. So every run of orders checks for such an asset before the decorated function runs, and raises:

QuarantineKeyCollisionError: 'orders' declares `quarantine=True`, so its invalid rows are written
to 'orders_quarantine', which another asset in this code location already materializes. Rename
that asset, or remove `quarantine=True` from 'orders'. If that asset is your own quarantine table,
delete it and use `quarantine_spec` instead, which adds the quarantine itself to the asset graph.

A from __future__ import annotations in your own module breaks an annotated context parameter. Under PEP 563, Dagster receives every annotation as a string. Dagster compares the context annotation with the real classes, so it raises this error for both context: dg.AssetExecutionContext and context: AssetExecutionContext:

DagsterInvalidDefinitionError: Cannot annotate `context` parameter with type dg.AssetExecutionContext.
`context` must be annotated with AssetExecutionContext, AssetCheckExecutionContext, OpExecutionContext, or left blank.

The restriction is Dagster’s, not this package’s. @dg.asset raises the same error for the same annotation. Both accept an unannotated context, which is the only option in that message that still works under PEP 563:

@dd.asset(Orders)
def orders(context) -> pl.DataFrame:
    context.log.info("run %s", context.run_id)
    return pl.read_parquet("raw/orders.parquet")

A @dy.rule() body needs its class parameter. Without it, Python still defines the schema class and the asset, and raises no error. The run then fails when this package reads the rule’s expression for the check metadata:

TypeError: Orders.amount_is_positive() takes 0 positional arguments but 1 was given

@dy.rule() is a classmethod-style decorator, so the body takes cls:

class Orders(dy.Schema):
    status = dy.Enum(["new", "paid", "shipped", "cancelled"], nullable=False)
    amount = dy.Decimal(10, 2, nullable=False)

    @dy.rule()
    def paid_orders_have_amount(cls) -> pl.Expr:
        """Require a positive amount on a paid line."""
        return (cls.status.col != "paid") | (cls.amount.col > 0)

The docstring becomes that check’s description in the catalog.