Upgrading
Each breaking release has a section here, with the migration for it. Before 1.0, only a minor release, such as 0.11.0, has breaking changes. The changelog lists each release’s changes. Pin to one minor version if that matters to you: >= the version you installed, < the next minor.
0.12
Every dd.wiring function that reads the error-rows key requires it as a positional dg.AssetKey. Call dd.wiring.get_error_rows_key(context.asset_key) once and pass the result to each function, so the writer, the collision check and the path cannot use different keys. build_file_writer and get_error_rows_path no longer take the asset key. An asset built with dd.asset needs no change.
| 0.11 | 0.12 |
|---|---|
validate_error_rows_key(context) |
validate_error_rows_key(context, error_rows_key) |
build_io_manager_writer(context) |
build_io_manager_writer(context, error_rows_key) |
build_file_writer(key, error_rows_dir, partition_key) |
build_file_writer(context, error_rows_key, error_rows_dir) |
get_error_rows_path(key, error_rows_dir, partition_key) |
get_error_rows_path(error_rows_key, error_rows_dir, partition_key) |
A str or Sequence[str] passed as error_rows_key= becomes dg.AssetKey(...), such as dg.AssetKey(["audit", "orders"]), passed as the positional argument in the table.
build_file_writer takes the asset’s context in place of partition_key. It reads the partition from the context, so the file goes under the partition the call was invoked with. A context with a partition range raises dg.DagsterInvariantViolationError, because the error rows do not record which partition each row came from. Invoke the asset once for each partition instead.
stream_results names its second parameter returned, not frame. A call that passes the frame by keyword changes frame= to returned=.
stream_results takes the asset’s context in place of asset_key= and partitioned=. It reads the asset key and the run’s partition from the context, so the table and the error rows have their row counts under the same key.
| 0.11 | 0.12 |
|---|---|
stream_results(schema, returned, asset_key=context.asset_key, partitioned=True) |
stream_results(schema, returned, context=context) |
A test that called stream_results outside an asset calls it inside a @dg.asset, and calls that asset with dg.build_asset_context(). An unbound context raises DagsterInvalidPropertyError when stream_results reads its asset key.
Every asset built with dd.asset takes a context parameter, so a direct invocation passes one first. orders(frame) becomes orders(dg.build_asset_context(), frame). An asset with filter=True already took one.
A partitioned asset that runs without a partition key records its row count under dagster/row_count. That covers dg.materialize without partition_key and a direct invocation with dg.build_asset_context(). Before, dd.asset(filter=True) put the error rows’ row count under dagster/row_count and the table’s under dagster/partition_row_count in such a run.
0.11
Every function that returns something is named with a verb. stream_results takes its verb from dagster-dbt’s .stream(), which also yields Dagster results. Nothing behaves differently, and every check keeps its name and its history.
| 0.10 | 0.11 |
|---|---|
dd.collection_checks |
dd.build_collection_checks |
dd.error_rows_spec |
dd.build_error_rows_spec |
dd.testing.direct_invocation_result(called) |
dd.testing.DirectInvocationResult.from_invocation(called) |
dd.wiring.validation_results(schema, frame, valid_key=...) |
dd.wiring.stream_results(schema, frame, asset_key=...) |
dd.wiring.AssetYield |
gone: write Iterator[dg.MaterializeResult[pl.DataFrame] | dg.AssetCheckResult] |
dd.wiring.asset_check_specs |
dd.wiring.build_asset_check_specs |
dd.wiring.asset_check_results |
dd.wiring.evaluate_checks |
dd.wiring.asset_check_name |
dd.wiring.get_check_name_for_rule |
dd.wiring.schema_metadata |
dd.wiring.get_column_schema_metadata |
dd.wiring.table_schema |
dd.wiring.get_table_schema |
dd.wiring.error_rows_frame |
dd.wiring.get_error_rows |
dd.wiring.error_rows_path |
dd.wiring.get_error_rows_path |
dd.wiring.delegating_writer |
dd.wiring.build_io_manager_writer |
dd.wiring.file_writer |
dd.wiring.build_file_writer |
The schema check granularity is gone. check_granularity="schema", from the argument or from DAGSTER_DATAFRAMELY_CHECK_GRANULARITY, raises InvalidSettingError. Change the argument or the variable to rule or column, and the asset starts a new check history. docs/out-of-scope/schema-granularity.md has the reasons.
schema_rules is gone, and a schema-level rule keeps its own check at column granularity. column collapses the rules Dataframely generates on a column into dy_col__<column>. Each @dy.rule() and the primary key now has its own dy_rule__<rule> check, as each dbt test has in dagster-dbt. With schema_rules="per_rule", 0.10 gave the same checks, so their history continues. dy_schema__rules is no longer declared, so its history ends with the last run before the upgrade. Remove schema_rules=, dd.SchemaRules and DAGSTER_DATAFRAMELY_SCHEMA_RULES; nothing replaces them. docs/out-of-scope/schema-rules.md has the reasons.
Every check-result metadata key is under dataframely/. dy_ now covers only check names, rule columns and the dy_rule column in a collapsed check’s failure examples. failed_row_count follows dagster-dbt’s dagster_dbt/failed_row_count.
| 0.10 | 0.11 |
|---|---|
dy_rule |
dataframely/rule |
dy_rule__expr |
dataframely/rule_expr |
dy_failed_count |
dataframely/failed_row_count |
dy_failed_sample |
dataframely/failed_examples |
dy_rules |
dataframely/rules |
dy_schema__errors |
dataframely/column_schema_errors |
dy_filter |
dataframely/filter |
The failure rows a check shows are failure examples, Dataframely’s word. max_failure_samples= is max_failure_examples=, and the variable is DAGSTER_DATAFRAMELY_MAX_FAILURE_EXAMPLES. row_sample keeps its name.
A partitioned asset’s row count is under dagster/partition_row_count. Dagster defines dagster/row_count as the whole table’s count, so a partitioned materialization no longer writes it. stream_results takes partitioned=True for a hand-wired partitioned asset, and dd.asset passes it when it has a partitions_def. Dagster’s Overview tab reads only dagster/row_count, so its row-count stat is empty for a partitioned asset unless you or the IO manager write that key.
A run with filter=True logs a materialization for the error rows after each write. The spec from build_error_rows_spec then has a history and a partition status, and an asset downstream of it can automate on it. A run launched from a code location without the spec logs no materialization for the error rows. A run of dg.materialize or execute_in_process has no code location to read, so it logs the materialization with or without the spec. Code that expected one materialization from such a run now receives two.
Every asset has a code version derived from its schema. With code_version=None, dd.asset sets the sha1 hexdigest of each column’s name and dtype, then each rule’s name, severity and serialized expression. Every existing asset reports “has a new code version” once after the upgrade. A Polars upgrade can do the same, because Polars documents its expression serialization as unstable between versions. dd.asset now calls every @dy.rule() function when the asset is defined, so dd.asset raises if one of them raises. An is_in over a Python set of strings gives a different code version in each process, so write a list.
0.10
A schema declares each rule’s severity, and filter=True replaces quarantine=True. A rule is "error" unless the schema declares "warn", so a schema that declares nothing validates as before. A row that fails only "warn" rules is written to the table, and its check fails at WARN. With filter=True, the package writes the rows that failed an "error" rule under <name>__error_rows. Rule severity shows how a schema declares a severity.
The error rows have a new asset key. A run no longer writes to <name>_quarantine, and nothing removes the rows an earlier run wrote there. To keep the old key, pass it as error_rows_key, such as error_rows_key=["sales", "orders_quarantine"].
A check for an "error" rule fails at ERROR with filter=True. In 0.9 the same check failed at WARN with quarantine=True. Downstream assets still run, because with filter=True no rule’s check is blocking. With filter=False, a check for a "warn" rule is not blocking, where in 0.9 every check was.
dd.wiring.asset_check_results no longer takes severity. It reads each rule’s severity from the schema.
The check names are unchanged, so Dagster keeps each check’s history. These names change, most of them from quarantine to error_rows:
| 0.9 | 0.10 |
|---|---|
@dd.asset(OrdersSchema, quarantine=True) |
@dd.asset(OrdersSchema, filter=True) |
<name>_quarantine |
<name>__error_rows |
dd.quarantine_spec |
dd.error_rows_spec |
DAGSTER_DATAFRAMELY_QUARANTINE_DIR |
DAGSTER_DATAFRAMELY_ERROR_ROWS_DIR |
dd.errors.QuarantineDirError |
dd.errors.ErrorRowsDirError |
dd.errors.QuarantineKeyCollisionError |
dd.errors.ErrorRowsKeyCollisionError |
dd.wiring.QuarantineWriter |
dd.wiring.ErrorRowsWriter |
dd.wiring.quarantine_frame |
dd.wiring.error_rows_frame |
dd.wiring.quarantine_path(..., quarantine_dir=...) |
dd.wiring.error_rows_path(..., error_rows_dir=...) |
dd.wiring.file_writer(..., quarantine_dir=...) |
dd.wiring.file_writer(..., error_rows_dir=...) |
dd.wiring.validation_results(..., quarantine_writer=...) |
dd.wiring.validation_results(..., error_rows_writer=...) |
dd.wiring.validate_quarantine_key |
dd.wiring.validate_error_rows_key |
dd.wiring.check_specs |
dd.wiring.asset_check_specs |
dd.wiring.check_results |
dd.wiring.asset_check_results |
dd.wiring.check_name |
dd.wiring.asset_check_name |
dataframely/quarantine_address |
dataframely/error_rows_address |
dataframely/invalid_count |
dataframely/error_rows_count |
dataframely/invalid_sample |
dataframely/error_rows_sample |
dataframely/invalid_by_rules |
dataframely/error_rows_by_rules |
dataframely/valid_sample |
dataframely/table_sample |
dataframely/valid_statistics/<group> |
dataframely/table_statistics/<group> |
0.9
The floors move to dagster>=1.13.24, dataframely>=3.1.2 and polars>=1.44.2. The tests and every measurement in this repo run against those versions.
Two error classes are renamed. Each is raised in the same cases as before.
| 0.8 | 0.9 |
|---|---|
dd.errors.NothingSurvivedError |
dd.errors.NoValidRowsError |
dd.errors.UnnameableColumnError |
dd.errors.InvalidColumnNameError |
Most error messages are reworded, so code that matches on message text needs updating.
0.8
temp_dir and DAGSTER_DATAFRAMELY_TEMP_DIR are gone. Polars collects a returned LazyFrame in its streaming engine, so the package writes no staging file: drop the argument and the variable, and nothing else changes. The measurements are in docs/pre-1.0.md.
Two functions are renamed, and nothing about either changes but the name.
| 0.7 | 0.8 |
|---|---|
dd.build_quarantine_spec |
dd.quarantine_spec |
dd.wiring.process |
dd.wiring.validation_results |
The decorator is dd.asset. dy_asset put a name you type inside dy_, which in 0.8 named only what this package generated: the check names, the rule columns and the check metadata keys. Dataframely reuses 28 Polars names, such as DataFrame and Int64, under its own dy alias, as dd.asset reuses Dagster’s asset.
| 0.7 | 0.8 |
|---|---|
@dd.dy_asset(OrdersSchema) |
@dd.asset(OrdersSchema) |
from dagster_dataframely import dy_asset |
import dagster_dataframely as dd |
Import the module, not the name. A bare asset reads as Dagster’s asset.
A terminology pass renamed what this package had invented where Dagster, Dataframely or Polars already owned the word. Nothing behaves differently, and every check keeps its history.
| 0.7 | 0.8 | reason |
|---|---|---|
multi_column_rules= |
schema_rules= |
Dataframely calls these schema-level rules, and a single-column primary_key is one, so “multi-column” was false |
dd.MultiColumnRules |
dd.SchemaRules |
|
DAGSTER_DATAFRAMELY_MULTI_COLUMN_RULES |
DAGSTER_DATAFRAMELY_SCHEMA_RULES |
|
schema_rules="schema" |
schema_rules="collapsed" |
the old value repeated the setting’s own name |
dataframely/valid_stats/<family> |
dataframely/valid_statistics/<group> |
Polars’ DataTypeGroup |
0.7
The quarantine is no longer an asset: the asset’s own IO manager writes it, and this package ships no IO manager. ADR-0004 and ADR-0006 record why.
| 0.6 | 0.7 | note |
|---|---|---|
dd.dataframely_asset(schema=OrdersSchema) |
dd.dy_asset(OrdersSchema) |
the schema is positional |
quarantine=dg.AssetOut() |
quarantine=True |
it is a bool, and True needs no configuration |
dd.DataframelyParquetIOManager |
dagster_polars.PolarsParquetIOManager |
this package ships no production IO manager |
dd.DataframelyCSVIOManager |
none | 0.7 removes the CSV codecs too; use parquet or a warehouse |
| the quarantine was a second out | dd.build_quarantine_spec(OrdersSchema, orders) |
and only if you want it in the graph |
dd.errors.SchemaShapeError |
dd.errors.ColumnSchemaError |
“shape” was Polars’ word for something else |
dd.errors.QuarantineSettingError |
none | the quarantine has no dg.AssetOut to configure |
dd.errors.UnwritableDtypeError |
none | it belonged to the CSV writer |
dd.wiring.quarantine_table_schema |
none | a quarantine spec has its own Columns tab |
dd.wiring.process(..., quarantine_key=...) |
dd.wiring.process(..., quarantine_writer=...) |
see Hand-wiring |
dy_schema__dtypes |
dy_schema__columns |
this ends that check’s history |
sample |
dataframely/valid_sample |
|
stats/<family> |
dataframely/valid_stats/<family> |
|
dy_quarantine_path |
dataframely/quarantine_address |
it can be an asset key now, not just a file path |
dy_rejected_count |
dataframely/invalid_count |
|
dy_rejected_sample |
dataframely/invalid_sample |
|
dy_rejected_rules |
dataframely/invalid_by_rules |
dy_failed_count and dy_failed_sample are unchanged. So are check_granularity, multi_column_rules, statistics, max_failure_samples and row_sample, and every asset key and rule column, so Dagster keeps every check history except the column-schema check’s.