An IO manager that writes each output to a Parquet file under base_dir.
It loads an input as a pl.LazyFrame when the input’s annotation names one, and as a pl.DataFrame otherwise. It writes the frame’s column schema to the materialization, so a plain @dg.asset has columns in the catalog. The examples use it, so they need no IO manager library. In your own project, use a library IO manager such as dagster-polars’ PolarsParquetIOManager.
Attributes
base_dir: str-
The directory the files are written in.
Examples
import tempfile
import dagster as dg
import polars as pl
import dagster_dataframely as dd
from dagster_dataframely.data import (
OrdersSchema,
ParquetIOManager,
storefront_orders,
)
@dd.asset(OrdersSchema)
def orders() -> pl.DataFrame:
return storefront_orders
with tempfile.TemporaryDirectory() as base_dir:
result = dg.materialize(
[orders], resources={"io_manager": ParquetIOManager(base_dir=base_dir)}
)
result.success
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - RUN_START - Started execution of run for "__ephemeral_asset_job__".
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - ENGINE_EVENT - Executing steps in process (pid: 3014)
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - RESOURCE_INIT_STARTED - Starting initialization of resources [io_manager].
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - RESOURCE_INIT_SUCCESS - Finished initialization of resources [io_manager].
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - LOGS_CAPTURED - Capturing logs for process (pid: 3014).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_START - Started execution of step "orders".
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "result" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - orders - Writing file at: /tmp/tmpcr4307c4/orders.parquet using ParquetIOManager...
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_MATERIALIZATION - Materialized value orders.
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - HANDLED_OUTPUT - Handled output "result" using IO manager "io_manager"
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_schema__columns" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_schema__columns' on 'orders' passed. Description: 'Every column matches.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__paid_orders_have_amount" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__paid_orders_have_amount' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__line_numbers_are_dense" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__line_numbers_are_dense' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__primary_key" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__primary_key' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__order_id__nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__order_id__nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__order_id__regex" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__order_id__regex' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__line_no__nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__line_no__nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__line_no__min" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__line_no__min' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__email__nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__email__nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__email__check__lowercase" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__email__check__lowercase' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__email__max_length" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__email__max_length' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__amount__nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__amount__nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__amount__min" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__amount__min' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__quantity__nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__quantity__nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__quantity__min" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__quantity__min' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__quantity__max" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__quantity__max' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__priority__is_in" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__priority__is_in' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__status__nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__status__nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__tracking_id__unique" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__tracking_id__unique' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__is_gift__nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__is_gift__nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__ordered_at__nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__ordered_at__nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__tags__max_length" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__tags__max_length' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__tags__inner_nullability" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__tags__inner_nullability' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_OUTPUT - Yielded output "orders_dy_rule__note__check" of type "Any". (Type check passed).
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - ASSET_CHECK_EVALUATION - Asset check 'dy_rule__note__check' on 'orders' passed. Description: 'No row failed.'
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - orders - STEP_SUCCESS - Finished execution of step "orders" in 151ms.
/home/runner/work/dagster-dataframely/dagster-dataframely/.venv/lib/python3.12/site-packages/dagster/_core/execution/context_creation_job.py:273: RuntimeWarning: coroutine 'BaseEventLoop.shutdown_asyncgens' was never awaited
pass
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - ENGINE_EVENT - Finished steps in process (pid: 3014) in 177ms
2026-10-22 02:10:04 +0000 - dagster - DEBUG - __ephemeral_asset_job__ - 0710baf1-7196-4ca7-9e67-c36cfe49f488 - 3014 - RUN_SUCCESS - Finished execution of run for "__ephemeral_asset_job__".