API

This page lists every public name with its signature, generated from the source. The guide explains what each one does.

Core

dd.asset builds an asset from a schema, and quarantine_spec adds that asset’s quarantine to the asset graph.

asset(): Turn the decorated function into an asset that validates the frame it returns against schema.
quarantine_spec(): Return an asset spec that adds an asset’s quarantine to the asset graph.
Granularity: How many asset checks report a schema’s rules: one per rule at rule, one per column with rules at column, and one for the schema at schema.
SchemaRules: Which checks report the schema-level rules at column granularity.

Hand-wiring

dd.asset assembles these parts. Use them to build a @dg.asset yourself, where dd.asset does not fit.

wiring.schema_metadata(): Return the definition metadata that fills an asset’s Columns tab from the schema.
wiring.table_schema(): Return the schema as the dg.TableSchema that Dagster’s Columns tab shows.
wiring.check_specs(): Return the asset check specs for the schema’s rules, plus the column-schema check.
wiring.check_results(): Yield a result for each check check_specs declares, without writing any rows.
wiring.check_name(): Return the asset check name for the Dataframely rule named rule_name.
wiring.validation_results(): Validate a frame and yield the asset’s materialization and check results.
wiring.AssetYield: What validation_results yields: the asset’s materialization, if the run writes its table, then one result per check. Use it as the return annotation of a hand-wired asset’s function.
wiring.QuarantineWriter: A writer: a function that writes the invalid rows and returns the quarantine address, such as an asset key or a file path, as a string.
wiring.delegating_writer(): Return a writer that passes the invalid rows to the asset’s own IO manager.
wiring.file_writer(): Return a writer that writes the invalid rows to a parquet file under quarantine_dir.
wiring.quarantine_frame(): Return the invalid rows as a writer receives them, with one rule column per rule.
wiring.quarantine_path(): Return the path of the parquet file file_writer writes an asset’s invalid rows to.
wiring.validate_quarantine_key(): Raise if another asset already materializes the quarantine’s asset key.

Errors

This package raises these errors. Each one subclasses DagsterDataframelyError.

errors.DagsterDataframelyError: Base class for every error this package raises.
errors.CollectionNotSupportedError: schema= received a dy.Collection.
errors.ReservedColumnError: A column name is in the reserved dy_ namespace.
errors.InvalidColumnNameError: A column name has a character Dagster does not allow in an asset check name.
errors.CheckNameCollisionError: Two rules produce the same asset check name.
errors.InvalidSettingError: A setting has a value it does not allow.
errors.MaterializeResultValueError: A returned dg.MaterializeResult has no frame in value.
errors.MaterializeResultFieldError: A returned dg.MaterializeResult sets asset_key or check_results.
errors.ColumnSchemaError: A frame’s columns or dtypes do not match the schema.
errors.ValidationAbortError: Rows failed validation and the asset declares no quarantine, so it writes nothing.
errors.NoValidRowsError: Every row failed validation, so the run wrote only the quarantine.
errors.QuarantineKeyCollisionError: Another asset already materializes the quarantine’s asset key.
errors.QuarantineDirError: Invalid rows need writing, and nothing sets quarantine_dir.