You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by gi...@apache.org on 2022/11/30 09:54:57 UTC
[arrow-datafusion] branch dependabot/cargo/master/cranelift-module-0.90.0 updated (4dfa9bb79 -> ce10c35f9)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch dependabot/cargo/master/cranelift-module-0.90.0
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
discard 4dfa9bb79 Update cranelift-module requirement from 0.89.0 to 0.90.0
add eac254c0e Reduce Expr copies in parquet exec (#4283)
add d7a7fb61a Support parquet page filtering on min_max for `decimal128` and `string` columns (#4255)
add 2cc9a07df Fix DESCRIBE statement qualified table issue (#4304)
add 209c26670 add ambiguous check for projection (#4260)
add 7c07e4d77 Add a checker to confirm physical optimizer rules will keep the physical plan schema immutable (#4316)
add 94cd98208 Refine Err propagation and avoid unwrap in transform closures (#4318)
add 92325bf16 fix bug: right semi join will ignore filter with left side, and add test for test (#4327)
add bfce07652 Refactor downcasting functions with downcastvalue macro and improve error handling of `ListArray` downcasting (#4313)
add d355f69aa Add rule to reimplement `Eliminate cross join` and remove it in planner (#4185)
add ad5a50870 refactor: isolate common memory accounting utils (#4341)
add e817d75bb Update cli.md (#4329)
add 502b7e3ca reimplement `eliminate_outer_join` (#4272)
add 55bf8e9c2 Support types other than String for partition columns on ListingTables (#4221)
add 03f85f1db add another test case to cover join ambiguous check (#4305)
add 72a44874d make `plan_from_tables` return one plan instead of `Vec` (#4336)
add 07f65bff3 Support `SubqueryAlias` in optimizer-executor. (#4293)
add df8aa7a2e Add ability to disable DiskManager (#4330)
add 22fdbcfd0 minor: fix typos in test names (#4344)
add 561be4fda [CBO] JoinSelection Rule, select HashJoin Partition Mode based on the Join Type and available statistics, option for SortMergeJoin (#4219)
add e1204a5bf Minor: Add docs to EliminateOuterJoins (#4343)
add e0150439e add `{TDigest,ScalarValue,Accumulator}::size` (#4342)
add 58b43f5c0 Update env_logger requirement from 0.9 to 0.10 (#4367)
add 010aded5d Support to use Schedular in tpch benchmark (#4361)
add e34c6c33a add support for xz file compression and `compression` feature (#3993)
add 0d65394f2 Improve error messages when memory is exhausted while sorting (#4348)
add 63c81fe0c Add check to avoid underflow in memory manager (#4351)
add ba73c8180 code clean (#4354)
add a61615b29 Do not error in optimizer if resulting schema has different metadata (#4347)
add 12b5a1bfe refine the code of build schema for ambiguous check, factor this out into a function (#4379)
add 7310385e8 Minor: Fix typos in the documentation (#4376)
add 24453f988 enhancement: when fetch == 0, pushdown limit 0 instead skip+fetch. (#4334)
add 120689c22 Do not log error if page index can not be evaluatied (#4358)
add 3b7f76714 minor: correct name and typo (#4359)
add 27ae14aee HashJoin should return Err when the right side input stream produce Err, add more join UTs to cover different join types (#4373)
add 323fbb43a minor: improve optimizer logging and do not repeat rule name (#4345)
add 3bfdbaa31 Include byte size estimates in the filter statistics (#4375)
add 049690412 Use limit() function instead of show_limit() in the first example (#4369)
add da54fa584 improve error handling and add some more types (#4352)
add ad3df7dd4 API-break: Support `SubqueryAlias` and remove `Alias in Projection` (#4333)
add fe8aee673 Update rstest requirement from 0.15.0 to 0.16.0 (#4399)
add be1d37671 feat: `ResourceExhausted` for memory limit in `GroupedHashAggregateStream` (#4371)
add da55f93b2 Clean the `expr_fn` - use `scalar_expr` to create unary scalar expr functions, remove macro `unary_scalar_functions` (#4357)
add 9139183b1 Fix page index pruning fail on complex_expr (#4387)
add dafd95728 add `with_new_inputs` (#4393)
add a31b44eae Teach optimizer that `CoalesceBatchesExec` does not destroy sort order (#4332)
add 52e198ea4 refactor (#4391)
add 0d334cf7d Use tournament loser tree for k-way sort-merging (#4301)
add dd3f72ad1 feat: `ResourceExhausted` for memory limit in `AggregateStream` (#4405)
add 02da32e2e reimplment eliminate_limit to remove `global-state`. (#4324)
add 1438bc4ca Remove TaskProperties / KV structure (#4382)
add e4d790d49 [minor] use arrow kernel concat_batches instead combine_batches (#4423)
add fa4bea871 Adding more dataframe example to read csv files (#4360)
add 66c95e70a improve hashjoin execution metrics (#4394)
add 49166ea55 Add integration test for erroring when memory limits are hit (#4406)
add fdc83e852 Update to arrow 28 (#4400)
add ce10c35f9 Update cranelift-module requirement from 0.89.0 to 0.90.0
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4dfa9bb79)
\
N -- N -- N refs/heads/dependabot/cargo/master/cranelift-module-0.90.0 (ce10c35f9)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
benchmarks/Cargo.toml | 8 +-
benchmarks/expected-plans/q11.txt | 17 +-
benchmarks/expected-plans/q13.txt | 17 +-
benchmarks/expected-plans/q15.txt | 25 +-
benchmarks/expected-plans/q16.txt | 7 +-
benchmarks/expected-plans/q18.txt | 9 +-
benchmarks/expected-plans/q2.txt | 44 +-
benchmarks/expected-plans/q20.txt | 27 +-
benchmarks/expected-plans/q22.txt | 11 +-
benchmarks/expected-plans/q7.txt | 4 +-
benchmarks/expected-plans/q8.txt | 43 +-
benchmarks/expected-plans/q9.txt | 27 +-
benchmarks/src/bin/tpch.rs | 27 +-
datafusion-cli/Cargo.lock | 174 +-
datafusion-cli/Cargo.toml | 2 +-
datafusion-examples/Cargo.toml | 4 +-
datafusion-examples/examples/dataframe.rs | 57 +
datafusion-examples/examples/simple_udaf.rs | 4 +
datafusion/common/Cargo.toml | 4 +-
datafusion/common/src/cast.rs | 142 +-
datafusion/common/src/dfschema.rs | 149 +-
datafusion/common/src/scalar.rs | 174 +-
datafusion/core/Cargo.toml | 20 +-
.../core/src/avro_to_arrow/arrow_array_reader.rs | 10 +-
datafusion/core/src/avro_to_arrow/schema.rs | 30 +-
datafusion/core/src/config.rs | 27 +
datafusion/core/src/dataframe.rs | 6 +-
datafusion/core/src/datasource/file_format/csv.rs | 2 +-
.../core/src/datasource/file_format/file_type.rs | 88 +-
datafusion/core/src/datasource/file_format/json.rs | 4 +-
.../core/src/datasource/file_format/parquet.rs | 2 +-
datafusion/core/src/datasource/listing/helpers.rs | 77 +-
datafusion/core/src/datasource/listing/table.rs | 60 +-
.../core/src/datasource/listing_table_factory.rs | 38 +-
datafusion/core/src/datasource/memory.rs | 1 +
datafusion/core/src/datasource/view.rs | 5 +-
datafusion/core/src/execution/context.rs | 119 +-
datafusion/core/src/execution/disk_manager.rs | 67 +-
.../{memory_manager.rs => memory_manager/mod.rs} | 14 +-
.../core/src/execution/memory_manager/proxy.rs | 181 ++
datafusion/core/src/execution/options.rs | 34 +-
.../src/physical_optimizer/aggregate_statistics.rs | 8 +-
.../src/physical_optimizer/coalesce_batches.rs | 10 +-
.../core/src/physical_optimizer/enforcement.rs | 308 ++--
.../physical_optimizer/hash_build_probe_order.rs | 558 -------
.../core/src/physical_optimizer/join_selection.rs | 979 +++++++++++
datafusion/core/src/physical_optimizer/mod.rs | 2 +-
.../core/src/physical_optimizer/optimizer.rs | 6 +
datafusion/core/src/physical_optimizer/pruning.rs | 2 +-
.../core/src/physical_optimizer/repartition.rs | 4 +
.../core/src/physical_plan/aggregates/hash.rs | 219 ++-
.../core/src/physical_plan/aggregates/mod.rs | 195 ++-
.../src/physical_plan/aggregates/no_grouping.rs | 160 +-
.../core/src/physical_plan/aggregates/row_hash.rs | 150 +-
.../core/src/physical_plan/coalesce_batches.rs | 3 +-
datafusion/core/src/physical_plan/common.rs | 67 -
.../core/src/physical_plan/file_format/avro.rs | 6 +-
.../core/src/physical_plan/file_format/csv.rs | 23 +-
.../src/physical_plan/file_format/file_stream.rs | 6 +-
.../core/src/physical_plan/file_format/json.rs | 16 +-
.../core/src/physical_plan/file_format/mod.rs | 41 +-
.../core/src/physical_plan/file_format/parquet.rs | 71 +-
.../file_format/parquet/page_filter.rs | 162 +-
.../file_format/parquet/row_filter.rs | 8 +-
.../file_format/parquet/row_groups.rs | 68 +-
datafusion/core/src/physical_plan/filter.rs | 6 +
.../core/src/physical_plan/joins/hash_join.rs | 245 ++-
datafusion/core/src/physical_plan/joins/mod.rs | 3 +
.../src/physical_plan/joins/sort_merge_join.rs | 70 +-
datafusion/core/src/physical_plan/joins/utils.rs | 6 +-
datafusion/core/src/physical_plan/limit.rs | 66 +-
datafusion/core/src/physical_plan/planner.rs | 61 +-
datafusion/core/src/physical_plan/projection.rs | 12 +-
datafusion/core/src/physical_plan/rewrite.rs | 10 +-
datafusion/core/src/physical_plan/sorts/cursor.rs | 12 +-
datafusion/core/src/physical_plan/sorts/sort.rs | 44 +-
.../physical_plan/sorts/sort_preserving_merge.rs | 162 +-
datafusion/core/src/physical_plan/windows/mod.rs | 7 +-
.../src/physical_plan/windows/window_agg_exec.rs | 30 +-
datafusion/core/src/test/mod.rs | 16 +
datafusion/core/src/test_util.rs | 10 +-
datafusion/core/tests/custom_sources.rs | 15 +-
datafusion/core/tests/join_fuzz.rs | 19 +-
datafusion/core/tests/memory_limit.rs | 112 ++
datafusion/core/tests/merge_fuzz.rs | 18 +-
datafusion/core/tests/order_spill_fuzz.rs | 21 +-
datafusion/core/tests/parquet/filter_pushdown.rs | 53 +-
datafusion/core/tests/parquet/mod.rs | 538 ++++++
datafusion/core/tests/parquet/page_pruning.rs | 504 +++++-
datafusion/core/tests/parquet/row_group_pruning.rs | 500 +-----
datafusion/core/tests/path_partition.rs | 123 +-
datafusion/core/tests/provider_filter_pushdown.rs | 7 +-
datafusion/core/tests/sql/create_drop.rs | 68 +
datafusion/core/tests/sql/explain_analyze.rs | 24 +-
datafusion/core/tests/sql/expr.rs | 46 +-
datafusion/core/tests/sql/information_schema.rs | 55 +-
datafusion/core/tests/sql/joins.rs | 1739 +++++++++++++-------
datafusion/core/tests/sql/mod.rs | 263 ++-
datafusion/core/tests/sql/parquet.rs | 32 +-
datafusion/core/tests/sql/parquet_schema.rs | 21 +-
datafusion/core/tests/sql/references.rs | 2 +-
datafusion/core/tests/sql/subqueries.rs | 206 +--
datafusion/core/tests/sql/wildcard.rs | 4 +-
datafusion/core/tests/sql/window.rs | 52 +-
datafusion/core/tests/user_defined_aggregates.rs | 11 +-
datafusion/expr/Cargo.toml | 2 +-
datafusion/expr/src/accumulator.rs | 5 +
datafusion/expr/src/expr_fn.rs | 297 ++--
datafusion/expr/src/logical_plan/builder.rs | 59 +-
datafusion/expr/src/logical_plan/plan.rs | 38 +-
datafusion/expr/src/type_coercion/binary.rs | 22 +-
datafusion/expr/src/utils.rs | 9 +-
datafusion/jit/Cargo.toml | 2 +-
datafusion/optimizer/Cargo.toml | 4 +-
datafusion/optimizer/README.md | 2 +-
.../optimizer/src/common_subexpr_eliminate.rs | 14 +-
datafusion/optimizer/src/decorrelate_where_in.rs | 131 +-
...educe_cross_join.rs => eliminate_cross_join.rs} | 443 +++--
datafusion/optimizer/src/eliminate_limit.rs | 307 ++--
...educe_outer_join.rs => eliminate_outer_join.rs} | 380 ++---
datafusion/optimizer/src/filter_push_down.rs | 42 +-
datafusion/optimizer/src/inline_table_scan.rs | 10 +-
datafusion/optimizer/src/lib.rs | 4 +-
datafusion/optimizer/src/limit_push_down.rs | 26 +-
datafusion/optimizer/src/optimizer.rs | 73 +-
datafusion/optimizer/src/projection_push_down.rs | 221 ++-
.../optimizer/src/rewrite_disjunctive_predicate.rs | 4 +-
.../optimizer/src/scalar_subquery_to_join.rs | 178 +-
.../optimizer/src/simplify_expressions/utils.rs | 10 +-
.../optimizer/src/subquery_filter_to_join.rs | 14 +-
.../optimizer/src/unwrap_cast_in_comparison.rs | 4 +-
datafusion/optimizer/tests/integration-test.rs | 56 +-
datafusion/physical-expr/Cargo.toml | 6 +-
.../physical-expr/src/aggregate/approx_distinct.rs | 5 +
.../src/aggregate/approx_percentile_cont.rs | 6 +
.../approx_percentile_cont_with_weight.rs | 6 +
.../physical-expr/src/aggregate/array_agg.rs | 6 +
.../src/aggregate/array_agg_distinct.rs | 6 +
datafusion/physical-expr/src/aggregate/average.rs | 4 +
.../physical-expr/src/aggregate/correlation.rs | 9 +
datafusion/physical-expr/src/aggregate/count.rs | 4 +
.../physical-expr/src/aggregate/count_distinct.rs | 20 +-
.../physical-expr/src/aggregate/covariance.rs | 4 +
datafusion/physical-expr/src/aggregate/median.rs | 33 +-
datafusion/physical-expr/src/aggregate/min_max.rs | 8 +
datafusion/physical-expr/src/aggregate/stddev.rs | 5 +
datafusion/physical-expr/src/aggregate/sum.rs | 6 +-
.../physical-expr/src/aggregate/sum_distinct.rs | 6 +
datafusion/physical-expr/src/aggregate/tdigest.rs | 14 +
datafusion/physical-expr/src/aggregate/variance.rs | 4 +
datafusion/physical-expr/src/crypto_expressions.rs | 14 +-
datafusion/physical-expr/src/expressions/binary.rs | 2 +-
.../src/expressions/binary/kernels_arrow.rs | 2 +-
datafusion/physical-expr/src/expressions/case.rs | 12 +-
.../src/expressions/get_indexed_field.rs | 11 +-
.../physical-expr/src/expressions/in_list.rs | 12 +-
.../physical-expr/src/expressions/try_cast.rs | 2 +-
datafusion/physical-expr/src/functions.rs | 10 +-
datafusion/physical-expr/src/hash_utils.rs | 12 +-
datafusion/physical-expr/src/rewrite.rs | 10 +-
datafusion/physical-expr/src/string_expressions.rs | 60 +-
datafusion/physical-expr/src/utils.rs | 6 +-
datafusion/proto/Cargo.toml | 2 +-
datafusion/proto/proto/datafusion.proto | 5 +-
datafusion/proto/src/from_proto.rs | 8 +-
datafusion/proto/src/generated/pbjson.rs | 48 +-
datafusion/proto/src/generated/prost.rs | 8 +-
datafusion/proto/src/lib.rs | 6 +-
datafusion/proto/src/logical_plan.rs | 63 +-
datafusion/proto/src/to_proto.rs | 6 +-
datafusion/row/Cargo.toml | 2 +-
datafusion/row/src/lib.rs | 7 +-
datafusion/sql/Cargo.toml | 2 +-
datafusion/sql/src/parser.rs | 18 +-
datafusion/sql/src/planner.rs | 673 ++++----
datafusion/sql/src/utils.rs | 9 +-
docs/source/user-guide/cli.md | 22 +-
docs/source/user-guide/configs.md | 37 +-
docs/source/user-guide/dataframe.md | 2 +-
docs/source/user-guide/example-usage.md | 10 +-
docs/source/user-guide/sql/explain.md | 2 +-
docs/source/user-guide/sql/information_schema.md | 2 +-
parquet-test-utils/Cargo.toml | 2 +-
test-utils/Cargo.toml | 4 +-
test-utils/src/data_gen.rs | 13 +-
test-utils/src/lib.rs | 32 +-
186 files changed, 8516 insertions(+), 5155 deletions(-)
rename datafusion/core/src/execution/{memory_manager.rs => memory_manager/mod.rs} (98%)
create mode 100644 datafusion/core/src/execution/memory_manager/proxy.rs
delete mode 100644 datafusion/core/src/physical_optimizer/hash_build_probe_order.rs
create mode 100644 datafusion/core/src/physical_optimizer/join_selection.rs
create mode 100644 datafusion/core/tests/memory_limit.rs
rename datafusion/optimizer/src/{reduce_cross_join.rs => eliminate_cross_join.rs} (73%)
rename datafusion/optimizer/src/{reduce_outer_join.rs => eliminate_outer_join.rs} (50%)