You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "timsaucer (via GitHub)" <gi...@apache.org> on 2026/05/11 13:27:22 UTC
[PR] Feat/prepare df 54 [datafusion-python]
timsaucer opened a new pull request, #1532:
URL: https://github.com/apache/datafusion-python/pull/1532
# Which issue does this PR close?
We do not have an issue for this.
# Rationale for this change
We are updating the upstream DataFusion dependency so that we can reduce the time to release 54 once the new version is released.
# What changes are included in this PR?
**Dependency bump:**
- `[workspace.package].version`: `53.0.0` → `54.0.0`.
- All `datafusion*` workspace deps switched from `version = "53"` to `git = "https://github.com/apache/datafusion", rev = "3d06bedcc9afbd65781ac1de28741c36140d2cbb"`.
- `Cargo.lock` refreshed for the datafusion family only.
**Rust compile fixes (28 errors):**
- Drop `as_any` impls — upstream traits (`AggregateUDFImpl`, `ScalarUDFImpl`, `WindowUDFImpl`, `SchemaProvider`, `CatalogProvider`, `CatalogProviderList`, `TableProvider`, `TableSource`, `ExecutionPlan`) now have `Any` as a supertrait. Call sites switch from `arc.as_any().downcast_ref::<T>()` to the upstream-provided `arc.downcast_ref::<T>()` helper.
- FFI provider conversions (`Arc<dyn X + Send>` → `Arc<dyn X>`): upstream `From<&FFI_*>` no longer carries the redundant `+ Send` bound now that the traits require Send/Sync as supertraits.
- `Cast` / `TryCast`: `data_type: DataType` → `field: FieldRef`. Python `PyCast.data_type()` accessor preserved.
- Stub match arms for new `Expr::HigherOrderFunction` / `Lambda` / `LambdaVariable` variants returning `Unsupported`. Upstream HOFs are not shipped yet ([apache/datafusion#14205](https://github.com/apache/datafusion/issues/14205)).
- Stub match arms for new `ScalarValue::ListView` / `LargeListView`. No 53.1.0 scalar functions produce these directly.
- `DatasetExec::partition_statistics` returns `Arc<Statistics>`; add new required `apply_expressions` trait method (leaf returns `Continue`).
- `#[allow(deprecated)]` on `TableFunctionImpl::call` pending a `call_with_args` migration that needs `SessionState` plumbing.
**Python test fixes (23 expectations) for upstream behavior changes:**
- `median` / `approx_median` / `approx_percentile_cont` return `Float64` (was matching input type).
- String functions (`concat_ws`, `lower`, `upper`, `repeat`, `reverse`, `split_part`, `translate`) return `StringView` for `StringView` input (was `String`).
- `overlay` appends past end-of-string rather than replacing.
- `arrays_zip` / `list_zip` struct field names changed from `c0`/`c1` to `"1"`/`"2"`.
- Filter on mismatched cast types now errors (was 0 matches).
**`check-upstream` audit trivial wins:**
- New `DataFrame.alias(name)` — wraps the logical plan in a `SubqueryAlias` for self-joins and qualifier-style references.
- `functions.__all__`: add `instr` and `position` (both already defined as public defs but missing from `__all__`).
- Top-level `datafusion.__all__`: re-export `TableProviderFactory` and `TableProviderFactoryExportable` (previously reachable only via the `datafusion.catalog` submodule).
# Are there any user-facing changes?
Yes — several behavior changes inherited from upstream DataFusion 54 (warrants `api change` label):
- `median` / `approx_median` / `approx_percentile_cont` now return `Float64` rather than matching the input type.
- String functions return `StringView` when fed `StringView` input (`concat_ws`, `lower`, `upper`, `repeat`, `reverse`, `split_part`, `translate`).
- `overlay` semantics: passing a start position past the end of a string now appends the replacement, e.g. `overlay("!", "--", 2) → "!--"` (was `"--"`).
- `arrays_zip` / `list_zip` field names changed: `c0`/`c1` → `"1"`/`"2"`.
- Comparing a numeric column against an incompatible string literal in a filter now raises a `Cannot cast string` error, where previously it silently produced zero matches.
- New: `DataFrame.alias(name)`, `instr` and `position` now appear under `from datafusion.functions import *`, `TableProviderFactory` and `TableProviderFactoryExportable` are now reachable from the top-level `datafusion` namespace.
## Follow-ups (not in this PR)
The `check-upstream` audit surfaced additional non-trivial gaps that each warrant their own design and PR:
- `DataFrame.registry` / `into_optimized_plan` / `into_unoptimized_plan` / `into_parts` / `task_ctx` — each needs a new wrapper class (e.g. `FunctionRegistry`, `SessionState`, `TaskContext`).
- `SessionContext` extensibility surface — I/O helpers (`read_batch`/`read_batches`), planner/rule extension (`add_optimizer_rule`, `add_analyzer_rule`, `register_expr_planner`, `register_relation_planner`, `with_function_factory`), state access (`state`, `runtime_env`, `task_ctx`, `new_with_state`), UDF introspection (`udf`/`udaf`/`udwf` lookup + listing), and misc helpers (`create_physical_expr`, `table_function`, `table_factory`, `parse_capacity_limit`). Tracked under EPIC [#24](https://github.com/apache/datafusion-python/issues/24).
- Distinct-aware aggregates: `count_distinct`, `sum_distinct`, `avg_distinct`. Upstream design at [apache/arrow-datafusion#2407](https://github.com/apache/arrow-datafusion/issues/2407).
- `TableFunctionImpl::call_with_args` migration — needs `SessionState` plumbing through `PyTableFunction`. Will be a user-facing API change.
- FFI Protocol pipeline completions for `FFI_TableFunction` (`from_pycapsule`, `TableFunctionExportable`, ABC), `FFI_LogicalExtensionCodec`, `FFI_ExtensionOptions`.
- Scalar `get_field_path` (variant of `get_field` taking a path expression).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Update datafusion dependency to latest in preparation for DF54 [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1532:
URL: https://github.com/apache/datafusion-python/pull/1532#issuecomment-4441330977
Currently blocked on a TPC-H test failing. Upstream correction is in https://github.com/apache/datafusion/pull/22146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Update datafusion dependency to latest in preparation for DF54 [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #1532:
URL: https://github.com/apache/datafusion-python/pull/1532#issuecomment-4477981225
@pgwhalen the agent skill update in this one may be relevant to you. I caught a gap in my previous check-upstream skill
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Update datafusion dependency to latest in preparation for DF54 [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on code in PR #1532:
URL: https://github.com/apache/datafusion-python/pull/1532#discussion_r3258626334
##########
python/datafusion/functions.py:
##########
@@ -184,6 +184,7 @@
"ifnull",
"in_list",
"initcap",
+ "instr",
Review Comment:
This goes a step too far imo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Update datafusion dependency to latest in preparation for DF54 [datafusion-python]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1532:
URL: https://github.com/apache/datafusion-python/pull/1532#discussion_r3254795490
##########
python/datafusion/expr.py:
##########
@@ -150,6 +150,9 @@
TransactionStart = expr_internal.TransactionStart
TryCast = expr_internal.TryCast
Union = expr_internal.Union
+HigherOrderFunction = expr_internal.HigherOrderFunction
+Lambda = expr_internal.Lambda
+LambdaVariable = expr_internal.LambdaVariable
Review Comment:
These new public expression wrapper exports are not added to the existing import/module coverage in `python/tests/test_imports.py`, which explicitly imports and checks the module for the other `datafusion.expr` classes. Adding them there would catch regressions in the wrapper export path and `__module__` metadata.
##########
Cargo.toml:
##########
@@ -71,3 +71,12 @@ codegen-units = 2
# We cannot publish to crates.io with any patches in the below section. Developers
# must remove any entries in this section before creating a release candidate.
[patch.crates-io]
+datafusion = { git = "https://github.com/apache/datafusion", rev = "47655fd6c9ef060d73497987e6ccb98e57196508" }
Review Comment:
The PR description says these patches pin DataFusion to commit `3d06bedcc9afbd65781ac1de28741c36140d2cbb`, but the manifest and lockfile pin `47655fd6c9ef060d73497987e6ccb98e57196508`. Please update the PR description or the pinned revision so reviewers and release notes identify the actual upstream commit being tested.
##########
python/datafusion/functions.py:
##########
@@ -273,6 +274,7 @@
"percent_rank",
"percentile_cont",
"pi",
+ "position",
Review Comment:
Since this change is meant to expose `position` through `from datafusion.functions import *`, please add a regression test for the star-export/`__all__` behavior. Existing tests only verify direct submodule imports for a couple of functions, so this exact export path could regress unnoticed.
##########
python/datafusion/dataframe.py:
##########
@@ -523,6 +523,32 @@ def select_exprs(self, *args: str) -> DataFrame:
"""
return self.df.select_exprs(*args)
+ def alias(self, alias: str) -> DataFrame:
Review Comment:
This adds a new public `DataFrame.alias()` API, but there is no corresponding Python test exercising qualifier-based selection/self-join behavior. The repository already has dataframe API coverage under `python/tests/test_dataframe.py`; adding a regression test would guard the intended alias semantics and the Python/Rust binding path.
##########
python/datafusion/functions.py:
##########
@@ -184,6 +184,7 @@
"ifnull",
"in_list",
"initcap",
+ "instr",
Review Comment:
Since this change is meant to expose `instr` through `from datafusion.functions import *`, please add a regression test for the star-export/`__all__` behavior. Existing tests only verify direct submodule imports for a couple of functions, so this exact export path could regress unnoticed.
##########
python/datafusion/__init__.py:
##########
@@ -133,6 +138,8 @@
"SessionContext",
"Table",
"TableFunction",
+ "TableProviderFactory",
+ "TableProviderFactoryExportable",
Review Comment:
The new top-level exports are not covered by the existing import tests, even though this change is specifically about making these classes reachable from `datafusion`. Please add an import/star-export regression test so a future `__all__` or re-export change does not silently drop them again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Update datafusion dependency to latest in preparation for DF54 [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer merged PR #1532:
URL: https://github.com/apache/datafusion-python/pull/1532
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org