You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2025/08/13 12:46:49 UTC
(datafusion) branch dependabot/cargo/main/substrait-0.59.0 updated (18a7db4c6e -> 8414462720)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/cargo/main/substrait-0.59.0
in repository https://gitbox.apache.org/repos/asf/datafusion.git
discard 18a7db4c6e chore(deps): bump substrait from 0.58.0 to 0.59.0
add 6793af5829 minor: remove unused import in docstring of datafusion_common::record_batch (#17106)
add 999b44ba6f Make macros in common::test_util hygenic (#17102)
add d987d2dc80 minor: remove unnecessary clippy:large_enum_variant allows (#17108)
add 3d6541d119 minor: Improve equivalence handling of joins (#16893)
add 9463cf6792 Fix incorrect `NULL IN ()` optimization (#17092)
add 876b504cb0 Simplify comparisons and binary operations involving NULL (#17088)
add c7d5904f01 Add `prettier` to the devcontainer (GitHub codespaces) (#17019)
add c4f0375686 Set a lower threshold for clippy to flag large error variants (#17109)
add ff9db020d7 chore(deps): bump rustyline from 16.0.0 to 17.0.0 (#17116)
add 14fb4a3c7d feat: Use Cached Metadata for ListingTable Statistics (#17022)
add ff77b70200 Add dynamic filter (bounds) pushdown to HashJoinExec (#16445)
add d375bfed45 Remove the "extended workflow" github action (#17119)
add f0e9334848 feat(spark): implement Spark math function mod/pmod (#16829)
add c5cb6f4ad7 feat(spark): implement Spark math function bit_get/bit_count (#16942)
add ceab1e1653 feat: add `isodow` (ISO day-of-week) support to date_part (Monday = 0) (#17112)
add 28e042dd47 feat(spark): implement spark datetime function date_add/date_sub (#17024)
add 8494a3967a Derive `WindowUDFImpl` equality, hash from `Eq`, `Hash` traits (#17081)
add 80d105840d chore(deps): bump sysinfo from 0.36.1 to 0.37.0 (#17124)
add a1431e2371 chore(deps): bump libc from 0.2.174 to 0.2.175 (#17121)
add 7d52145127 Preserve equivalence properties during projection pushdown (#17129)
add 9d9cde499b chore: Enforce checks for RC branches (#17132)
add 0db14c8dd3 chore(deps): bump actions/checkout from 4.2.2 to 5.0.0 (#17149)
add 00c2191bc2 fix: Add missing member to visitor for ConfigFileEncryptionProperties (#17103)
add 44daa9a7b8 minor: enhance comment in SortPreservingMergeStream.abort (#17115)
add f603d34e3b Update workspace to use Rust 1.89 (#17100)
add 25ad99dc6e feat: Add the ability to review the contents of the Metadata Cache (#17126)
add b8bf7c5bf1 Derive `AggregateUDFImpl` equality, hash from `Eq`, `Hash` traits (#17130)
add b786b9a78f Eliminate all redundant aggregations (#17139)
add e5b76f7ec6 chore: Clarify `EmptyRelation` description (#17157)
add c6304f687c chore(deps): bump on-headers and compression (#16812)
add 22665472e0 chore(deps): bump slab from 0.4.10 to 0.4.11 (#17161)
add e58f86c60a refactor `character_length` impl by unifying null handling logic (#16877)
add 1e3152ee1e Update dev env documentation to reflect pinned rust version (#17107)
add 43718d5fff Differentiate 0-row and 1-row EmptyRelation in EXPLAIN (#17145)
add dd5e0928d7 chore(deps): bump clap from 4.5.43 to 4.5.44 (#17148)
add eaf614d38e (Re)Support old syntax for `approx_percentile_cont` and `approx_percentile_cont_with_weight` (#16999)
add 0b7186ebaf Pass the input schema to stats_projection for ProjectionExpr (#17123)
add 8414462720 chore(deps): bump substrait from 0.58.0 to 0.59.0
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (18a7db4c6e)
\
N -- N -- N refs/heads/dependabot/cargo/main/substrait-0.59.0 (8414462720)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 69 +++
.devcontainer/Dockerfile | 8 +-
.github/workflows/audit.yml | 2 +-
.github/workflows/dependencies.yml | 2 +-
.github/workflows/dev.yml | 4 +-
.github/workflows/docs.yaml | 4 +-
.github/workflows/docs_pr.yaml | 2 +-
.github/workflows/extended.yml | 50 +-
.github/workflows/labeler.yml | 2 +-
.github/workflows/large_files.yml | 2 +-
.github/workflows/pr_comment_commands.yml | 89 ---
.github/workflows/rust.yml | 46 +-
Cargo.lock | 35 +-
clippy.toml | 12 +-
datafusion-cli/Cargo.toml | 4 +-
datafusion-cli/src/functions.rs | 123 +++-
datafusion-cli/src/main.rs | 105 +++-
.../cli_quick_test@can_see_indent_format.snap | 3 +-
datafusion-examples/examples/advanced_udaf.rs | 4 +-
datafusion-examples/examples/advanced_udwf.rs | 4 +-
datafusion-examples/examples/parquet_index.rs | 2 +-
datafusion/catalog/src/async.rs | 2 +-
datafusion/catalog/src/cte_worktable.rs | 2 +-
datafusion/catalog/src/default_table_source.rs | 2 +-
datafusion/catalog/src/table.rs | 4 +-
datafusion/catalog/src/view.rs | 2 +-
datafusion/common/Cargo.toml | 2 +-
datafusion/common/src/config.rs | 60 +-
datafusion/common/src/error.rs | 2 +-
datafusion/common/src/test_util.rs | 34 +-
datafusion/common/src/utils/mod.rs | 2 +-
datafusion/core/Cargo.toml | 2 +-
datafusion/core/src/dataframe/mod.rs | 2 +-
.../core/src/datasource/file_format/parquet.rs | 287 +++++++---
datafusion/core/tests/dataframe/mod.rs | 100 ++--
datafusion/core/tests/execution/logical_plan.rs | 2 +-
datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs | 2 +-
datafusion/core/tests/fuzz_cases/join_fuzz.rs | 18 +-
datafusion/core/tests/macro_hygiene/mod.rs | 1 +
datafusion/core/tests/optimizer/mod.rs | 2 +-
datafusion/core/tests/parquet/custom_reader.rs | 7 +-
datafusion/core/tests/parquet/file_statistics.rs | 2 +-
datafusion/core/tests/parquet/page_pruning.rs | 2 +-
.../physical_optimizer/filter_pushdown/mod.rs | 382 ++++++++++++-
.../tests/physical_optimizer/join_selection.rs | 7 +-
.../physical_optimizer/partition_statistics.rs | 7 +-
.../tests/user_defined/user_defined_aggregates.rs | 6 +-
.../user_defined/user_defined_window_functions.rs | 61 +-
.../src/avro_to_arrow/arrow_array_reader.rs | 2 +-
.../datasource-avro/src/avro_to_arrow/reader.rs | 2 +-
datafusion/datasource-parquet/src/file_format.rs | 91 ++-
datafusion/datasource-parquet/src/mod.rs | 2 +-
datafusion/datasource-parquet/src/opener.rs | 4 +-
datafusion/datasource-parquet/src/reader.rs | 71 ++-
datafusion/datasource/src/file_format.rs | 2 +-
datafusion/datasource/src/file_scan_config.rs | 2 +-
datafusion/datasource/src/memory.rs | 2 +-
datafusion/datasource/src/sink.rs | 2 +-
datafusion/datasource/src/source.rs | 35 +-
datafusion/datasource/src/statistics.rs | 4 +-
datafusion/execution/src/cache/cache_manager.rs | 24 +-
datafusion/execution/src/cache/cache_unit.rs | 263 ++++++++-
datafusion/execution/src/cache/lru_queue.rs | 7 +-
datafusion/expr-common/src/dyn_eq.rs | 64 +++
datafusion/expr-common/src/lib.rs | 1 +
datafusion/expr-common/src/operator.rs | 54 ++
datafusion/expr/src/expr_fn.rs | 4 -
datafusion/expr/src/logical_plan/plan.rs | 14 +-
datafusion/expr/src/planner.rs | 1 -
datafusion/expr/src/simplify.rs | 1 -
datafusion/expr/src/table_source.rs | 2 +-
datafusion/expr/src/test/function_stub.rs | 7 +-
datafusion/expr/src/udaf.rs | 66 +--
datafusion/expr/src/udf.rs | 1 +
datafusion/expr/src/udf_eq.rs | 28 +-
datafusion/expr/src/udwf.rs | 71 +--
datafusion/ffi/src/udaf/mod.rs | 3 -
datafusion/ffi/src/udwf/mod.rs | 3 -
.../functions-aggregate/benches/array_agg.rs | 2 +-
datafusion/functions-aggregate/benches/count.rs | 8 +-
datafusion/functions-aggregate/benches/sum.rs | 2 +-
.../functions-aggregate/src/approx_distinct.rs | 1 +
.../functions-aggregate/src/approx_median.rs | 1 +
.../src/approx_percentile_cont.rs | 25 +-
.../src/approx_percentile_cont_with_weight.rs | 15 +-
datafusion/functions-aggregate/src/array_agg.rs | 4 +-
datafusion/functions-aggregate/src/average.rs | 2 +-
.../functions-aggregate/src/bit_and_or_xor.rs | 6 +-
datafusion/functions-aggregate/src/bool_and_or.rs | 4 +-
datafusion/functions-aggregate/src/correlation.rs | 2 +-
datafusion/functions-aggregate/src/count.rs | 1 +
datafusion/functions-aggregate/src/covariance.rs | 2 +
datafusion/functions-aggregate/src/first_last.rs | 8 +-
datafusion/functions-aggregate/src/grouping.rs | 1 +
datafusion/functions-aggregate/src/median.rs | 1 +
datafusion/functions-aggregate/src/min_max.rs | 4 +-
datafusion/functions-aggregate/src/nth_value.rs | 2 +-
datafusion/functions-aggregate/src/regr.rs | 4 +-
datafusion/functions-aggregate/src/stddev.rs | 7 +-
datafusion/functions-aggregate/src/string_agg.rs | 5 +-
datafusion/functions-aggregate/src/sum.rs | 2 +-
datafusion/functions-aggregate/src/variance.rs | 2 +
datafusion/functions-window/src/cume_dist.rs | 2 +-
datafusion/functions-window/src/lead_lag.rs | 6 +-
datafusion/functions-window/src/macros.rs | 14 +-
datafusion/functions-window/src/nth_value.rs | 6 +-
datafusion/functions-window/src/ntile.rs | 2 +-
datafusion/functions-window/src/rank.rs | 5 +-
datafusion/functions-window/src/row_number.rs | 2 +-
datafusion/functions/src/datetime/common.rs | 4 +-
datafusion/functions/src/datetime/date_part.rs | 4 +-
datafusion/functions/src/datetime/to_local_time.rs | 2 +-
datafusion/functions/src/macros.rs | 1 +
datafusion/functions/src/math/log.rs | 2 +-
.../functions/src/unicode/character_length.rs | 53 +-
datafusion/functions/src/unicode/lpad.rs | 36 +-
datafusion/functions/src/utils.rs | 121 ++--
datafusion/optimizer/src/analyzer/type_coercion.rs | 108 ++--
datafusion/optimizer/src/eliminate_filter.rs | 8 +-
datafusion/optimizer/src/eliminate_join.rs | 2 +-
datafusion/optimizer/src/eliminate_limit.rs | 6 +-
.../optimizer/src/optimize_projections/mod.rs | 37 +-
.../optimizer/src/propagate_empty_relation.rs | 10 +-
datafusion/optimizer/src/push_down_filter.rs | 4 +-
.../src/simplify_expressions/expr_simplifier.rs | 298 ++++------
.../src/simplify_expressions/unwrap_cast.rs | 14 +-
.../optimizer/tests/optimizer_integration.rs | 4 +-
.../physical-expr-common/src/physical_expr.rs | 39 +-
datafusion/physical-expr/src/aggregate.rs | 2 +-
.../src/equivalence/properties/joins.rs | 8 +-
datafusion/physical-expr/src/intervals/utils.rs | 4 +-
datafusion/physical-plan/Cargo.toml | 1 +
.../aggregates/group_values/multi_group_by/mod.rs | 15 +-
datafusion/physical-plan/src/coalesce/mod.rs | 2 +-
datafusion/physical-plan/src/filter.rs | 4 +-
datafusion/physical-plan/src/joins/hash_join.rs | 149 ++++-
datafusion/physical-plan/src/joins/test_utils.rs | 25 +
datafusion/physical-plan/src/joins/utils.rs | 4 +-
datafusion/physical-plan/src/projection.rs | 88 ++-
datafusion/physical-plan/src/sorts/merge.rs | 15 +-
datafusion/proto/tests/cases/mod.rs | 4 -
.../proto/tests/cases/roundtrip_logical_plan.rs | 3 +-
datafusion/spark/src/function/bitwise/bit_count.rs | 319 +++++++++++
datafusion/spark/src/function/bitwise/bit_get.rs | 287 ++++++++++
datafusion/spark/src/function/bitwise/mod.rs | 20 +-
datafusion/spark/src/function/datetime/date_add.rs | 129 +++++
datafusion/spark/src/function/datetime/date_sub.rs | 123 ++++
datafusion/spark/src/function/datetime/mod.rs | 16 +-
datafusion/spark/src/function/math/mod.rs | 7 +-
datafusion/spark/src/function/math/modulus.rs | 617 +++++++++++++++++++++
datafusion/spark/src/function/utils.rs | 87 +--
datafusion/sql/src/expr/function.rs | 7 +-
datafusion/sql/src/unparser/utils.rs | 2 +-
datafusion/sql/tests/cases/params.rs | 12 +-
datafusion/sql/tests/sql_integration.rs | 264 ++++-----
datafusion/sqllogictest/Cargo.toml | 2 +-
datafusion/sqllogictest/data/1.parquet | Bin 0 -> 1258 bytes
datafusion/sqllogictest/data/2.parquet | Bin 0 -> 1273 bytes
.../src/engines/datafusion_engine/error.rs | 2 +-
datafusion/sqllogictest/test_files/aggregate.slt | 61 +-
datafusion/sqllogictest/test_files/array.slt | 4 +-
.../sqllogictest/test_files/count_star_rule.slt | 2 +-
datafusion/sqllogictest/test_files/cte.slt | 22 +-
datafusion/sqllogictest/test_files/errors.slt | 3 +-
datafusion/sqllogictest/test_files/explain.slt | 13 +-
.../sqllogictest/test_files/explain_tree.slt | 7 +-
.../sqllogictest/test_files/expr/date_part.slt | 20 +
datafusion/sqllogictest/test_files/issue_17138.slt | 36 ++
datafusion/sqllogictest/test_files/join.slt.part | 2 +-
datafusion/sqllogictest/test_files/joins.slt | 69 ++-
datafusion/sqllogictest/test_files/order.slt | 4 +-
datafusion/sqllogictest/test_files/parquet.slt | 3 +-
.../test_files/parquet_filter_pushdown.slt | 32 ++
datafusion/sqllogictest/test_files/predicates.slt | 71 ++-
.../test_files/simplify_predicates.slt | 4 +-
.../test_files/spark/bitwise/bit_count.slt | 204 ++++++-
.../test_files/spark/bitwise/bit_get.slt | 51 +-
.../test_files/spark/bitwise/getbit.slt | 51 +-
.../test_files/spark/datetime/date_add.slt | 45 +-
.../test_files/spark/datetime/date_sub.slt | 50 +-
.../test_files/spark/datetime/dateadd.slt | 36 +-
.../sqllogictest/test_files/spark/math/mod.slt | 221 +++++++-
.../sqllogictest/test_files/spark/math/pmod.slt | 320 ++++++++++-
.../test_files/spark/predicate/ilike.slt | 2 +-
datafusion/sqllogictest/test_files/subquery.slt | 12 +-
.../sqllogictest/test_files/type_coercion.slt | 32 +-
datafusion/sqllogictest/test_files/union.slt | 14 +-
datafusion/sqllogictest/test_files/unnest.slt | 6 +-
datafusion/sqllogictest/test_files/window.slt | 16 +-
.../tests/cases/roundtrip_logical_plan.rs | 2 +-
.../wasmtest/datafusion-wasm-app/package-lock.json | 131 ++---
dev/release/README.md | 13 +-
.../contributor-guide/development_environment.md | 10 +-
docs/source/contributor-guide/testing.md | 18 +-
docs/source/library-user-guide/upgrading.md | 10 +
.../library-user-guide/working-with-exprs.md | 2 +-
.../user-guide/cli/{usage.md => functions.md} | 282 +++-------
docs/source/user-guide/cli/index.rst | 1 +
docs/source/user-guide/cli/usage.md | 63 +--
docs/source/user-guide/sql/aggregate_functions.md | 29 +
docs/source/user-guide/sql/scalar_functions.md | 3 +-
rust-toolchain.toml | 2 +-
202 files changed, 5656 insertions(+), 1849 deletions(-)
delete mode 100644 .github/workflows/pr_comment_commands.yml
create mode 100644 datafusion/expr-common/src/dyn_eq.rs
create mode 100644 datafusion/spark/src/function/bitwise/bit_count.rs
create mode 100644 datafusion/spark/src/function/bitwise/bit_get.rs
create mode 100644 datafusion/spark/src/function/datetime/date_add.rs
create mode 100644 datafusion/spark/src/function/datetime/date_sub.rs
create mode 100644 datafusion/spark/src/function/math/modulus.rs
create mode 100644 datafusion/sqllogictest/data/1.parquet
create mode 100644 datafusion/sqllogictest/data/2.parquet
create mode 100644 datafusion/sqllogictest/test_files/issue_17138.slt
copy docs/source/user-guide/cli/{usage.md => functions.md} (51%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org