You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by mg...@apache.org on 2026/07/29 08:57:09 UTC
(datafusion-ballista) branch redirect-root-to-nightlies updated (cac0d4015 -> cf5dab3e1)
This is an automated email from the ASF dual-hosted git repository.
martin-g pushed a change to branch redirect-root-to-nightlies
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
from cac0d4015 Add unit tests for the get_webtui() Axum handler
add bb2fdfee2 ci(tpch): add AQE-on smoke pass with multi-partition tasks (#2120)
add b16270801 see through Shared wrapper when classifying task IO errors (#2119)
add 62228ecc2 feat(core): RuntimeStatsExec — passthrough tap for row counts + quantile sketches (#2094)
add b80947575 feat(core): add BufferExec — generic flow-control op with a memory-pool Dam mode (#2095)
add bd17e435c feat: Improve TPC-H performance with AQE on (#2084)
add 6540d0851 Remove the hash-partitioning shuffle writer (#2106)
add 8be1e8986 docs: correct scan partitioning description in tuning guide (#2125)
add 65dc2dfb4 feat(benchmarks): add --skip to omit queries from a tpch suite run (#2129)
add 9b587954a chore: upgrade DataFusion to 54.1.0 (#2126)
add 2c9367033 fix: Various fixes for AQE join selection (#2124)
add af7db0be4 feat(core): report the actual sort shuffle spill trigger in logs (#2143)
add b55d1e99e Generate the config reference documentation from the config registry (#2146)
add ce115795f fix: preserve null_equality so distributed INTERSECT/EXCEPT match NULLs (#2064)
add 2bb0e8eb9 feat(core): add UnorderedRangeRepartitionExec — value-range router over unordered inputs (#2123)
add 7e2e4d95c chore(deps): bump tokio-stream from 0.1.18 to 0.1.19 in /python (#2157)
add ba6ae5c71 chore(deps): bump tokio-stream from 0.1.18 to 0.1.19 (#2156)
add 696ca29b3 fix(executor): keep heartbeating while all task slots are busy (#2159)
add d738ca9bd docs: refresh benchmarking.md with current SF1000 results (Ballista main @ 696ca29b, Spark 3.4 baseline) (#2161)
add 90f8096ed feat(executor): default to a bounded, auto-sized memory pool (#2160)
add 9efb1f46e docs: refresh benchmarking.md - dedicated shuffle PVC, r6i node, updated numbers (#2171)
add 67ae6263f chore(ci): bump taiki-e/install-action from 2.84.0 to 2.85.0 (#2164)
add d9e189b3e chore(deps): bump ctor from 1.0.9 to 1.0.10 (#2137)
add 561037c7b ci: allow the docs site deploy to be triggered manually (#2177)
add a9bf1168c chore(deps): bump libc from 0.2.187 to 0.2.189 (#2142)
add 9af0a28ca chore(deps): bump tokio-util from 0.7.18 to 0.7.19 (#2135)
add 5e6201e96 feat(core): add OrderedRangeRepartitionExec — N sorted overlapping → K sorted disjoint (#2169)
add b9711402b kll: add native Rust KLL quantile sketch (#2180)
add be65da313 chore(deps): bump arrow from 58.3.0 to 58.4.0 (#2178)
add 1517c94f2 docs: add 54.0.0 changelog (#2012)
add bdc21e641 chore(deps): bump quinn-proto from 0.11.14 to 0.11.16 (#2176)
add 5dc768b68 fix: unskip five TPC-DS queries by rewriting generator column names (#2183)
add 8b3ec30c8 chore(ci): bump taiki-e/install-action from 2.85.0 to 2.85.2 (#2191)
add 0deac135d chore(deps): bump aws-config from 1.9.0 to 1.10.1 (#2190)
add 99897a051 chore(deps): bump ctor from 1.0.10 to 1.0.11 (#2189)
add da96bff15 chore(ci): bump the github-codeql group across 1 directory with 2 updates (#2133)
add 33d9bac62 chore(ci): bump astral-sh/setup-uv from 8.3.2 to 9.0.0 (#2134)
add dc6b38097 feat(core,scheduler): transport RuntimeStatsExec reports to scheduler; log merged quantile cuts per stage (#2175)
add 828d6f007 fix: stop every task running every union branch at low partition counts (#2186)
add ba46698f3 slice 1b3a: PerPartitionFilterExec operator + codec (#2195)
add 2b04f83ad ci: run TPC-DS SF1 with 1 and 4 partitions per task (#2197)
add bd5563ade fix(scheduler): execute null-aware anti joins in one task (#2188)
add 7eeff3beb fix(scheduler): keep GROUP BY-less aggregates when propagating empty stages (#2194)
add 6bbcc077a chore: use taplo for Cargo.toml formatting (#2179)
add 86fc9a766 fix(scheduler): harden PropagateEmptyExecRule against grouping sets, embedded projections, and dropped exchange partitioning (#2200)
add d32877b85 feat(core): rewrite NOT IN subqueries into distributable anti join plans (#2199)
add b34144e5e test: add HA chaos harness that drives the scheduler's fault-tolerance paths on a real cluster (#2026)
add 0c7b707f4 chore(ci): bump taiki-e/install-action from 2.85.2 to 2.85.3 (#2203)
add cf5dab3e1 Merge branch 'main' into redirect-root-to-nightlies
No new revisions were added by this update.
Summary of changes:
.github/workflows/build.yml | 14 +-
.github/workflows/codeql.yml | 4 +-
.github/workflows/docs.yaml | 3 +
.github/workflows/rust.yml | 35 +-
.github/workflows/tpcds.yml | 73 +-
.github/workflows/tpch.yml | 68 +-
.github/workflows/web-tui.yml | 4 +-
CONTRIBUTING.md | 16 +
Cargo.lock | 339 ++---
Cargo.toml | 38 +-
ballista-cli/Cargo.toml | 62 +-
ballista/client/Cargo.toml | 6 +-
ballista/client/tests/context_checks.rs | 2 +-
ballista/client/tests/null_aware.rs | 152 +++
ballista/client/tests/sort_shuffle.rs | 86 --
ballista/core/Cargo.toml | 1 +
ballista/core/proto/ballista.proto | 113 ++
ballista/core/proto/datafusion.proto | 1 +
ballista/core/src/bin/update_config_docs.rs | 835 ++++++++++++
ballista/core/src/config.rs | 179 ++-
ballista/core/src/error.rs | 75 +-
ballista/core/src/execution_plans/buffer.rs | 515 ++++++++
ballista/core/src/execution_plans/mod.rs | 15 +
.../execution_plans/ordered_range_repartition.rs | 931 +++++++++++++
.../src/execution_plans/per_partition_filter.rs | 426 ++++++
.../execution_plans/range_repartition_common.rs | 547 ++++++++
ballista/core/src/execution_plans/runtime_stats.rs | 1395 ++++++++++++++++++++
.../core/src/execution_plans/shuffle_reader.rs | 23 +-
.../core/src/execution_plans/shuffle_writer.rs | 570 ++------
.../core/src/execution_plans/sort_shuffle/spill.rs | 6 +-
.../src/execution_plans/sort_shuffle/writer.rs | 151 ++-
.../execution_plans/unordered_range_repartition.rs | 931 +++++++++++++
ballista/core/src/extension.rs | 47 +-
ballista/core/src/kll.rs | 670 ++++++++++
ballista/core/src/lib.rs | 4 +
ballista/core/src/optimizer.rs | 475 +++++++
ballista/core/src/serde/generated/ballista.rs | 166 ++-
ballista/core/src/serde/mod.rs | 762 ++++++++++-
ballista/core/src/utils.rs | 1 +
ballista/executor/Cargo.toml | 10 +-
ballista/executor/src/config.rs | 45 +-
ballista/executor/src/execution_engine.rs | 59 +-
ballista/executor/src/execution_loop.rs | 46 +-
ballista/executor/src/executor.rs | 3 +-
ballista/executor/src/executor_process.rs | 342 ++++-
ballista/executor/src/executor_server.rs | 10 +-
ballista/executor/src/flight_service.rs | 2 +-
ballista/executor/src/lib.rs | 32 +-
ballista/scheduler/Cargo.toml | 2 +-
.../src/physical_optimizer/join_selection.rs | 130 +-
ballista/scheduler/src/planner.rs | 513 +++++--
ballista/scheduler/src/scheduler_server/mod.rs | 1 +
.../src/state/aqe/execution_plan/dynamic_join.rs | 754 ++++++++++-
ballista/scheduler/src/state/aqe/mod.rs | 26 +-
.../aqe/optimizer_rule/distributed_exchange.rs | 73 +
.../src/state/aqe/optimizer_rule/join_selection.rs | 152 ++-
.../state/aqe/optimizer_rule/propagate_empty.rs | 287 +++-
.../src/state/aqe/test/broadcast_thresholds.rs | 299 +++++
.../scheduler/src/state/aqe/test/join_selection.rs | 33 +-
ballista/scheduler/src/state/aqe/test/mod.rs | 20 +-
.../scheduler/src/state/aqe/test/plan_to_stages.rs | 33 +-
.../scheduler/src/state/aqe/test/stats_table.rs | 232 ++++
ballista/scheduler/src/state/execution_graph.rs | 24 +-
ballista/scheduler/src/state/execution_stage.rs | 139 +-
ballista/scheduler/src/state/task_builder.rs | 108 +-
ballista/scheduler/src/state/task_manager.rs | 44 +
ballista/scheduler/src/test_utils.rs | 29 +
benchmarks/src/bin/shuffle_bench.rs | 82 +-
benchmarks/src/bin/tpcds.rs | 134 +-
benchmarks/src/bin/tpch.rs | 123 +-
chaos-testing/Cargo.toml | 55 +
chaos-testing/README.md | 350 +++++
chaos-testing/src/bin/chaos-executor.rs | 62 +
.../src/bin/chaos-scheduler.rs | 54 +-
chaos-testing/src/budget.rs | 126 ++
chaos-testing/src/cluster.rs | 673 ++++++++++
chaos-testing/src/fixture.rs | 373 ++++++
.../mod.rs => chaos-testing/src/lib.rs | 18 +-
chaos-testing/src/registry.rs | 103 ++
chaos-testing/src/udf.rs | 328 +++++
chaos-testing/tests/common/mod.rs | 131 ++
chaos-testing/tests/ha.rs | 417 ++++++
.../{rust_docs.sh => rust_config_docs_check.sh} | 5 +-
ci/scripts/rust_toml_fmt.sh | 23 +-
dev/rust_lint.sh | 6 +-
.../rust_docs.sh => dev/update_config_docs.sh | 7 +-
docs/source/changelog/54.0.0.md | 150 +++
docs/source/changelog/index.md | 1 +
docs/source/contributors-guide/benchmarking.md | 493 +++----
docs/source/upgrading/55.0.0.md | 44 +
docs/source/user-guide/configs.md | 93 +-
docs/source/user-guide/tuning-guide.md | 159 ++-
examples/Cargo.toml | 2 +-
python/Cargo.lock | 79 +-
python/python/tests/test_context.py | 4 +-
taplo.toml | 63 +
96 files changed, 15603 insertions(+), 1744 deletions(-)
create mode 100644 ballista/client/tests/null_aware.rs
create mode 100644 ballista/core/src/bin/update_config_docs.rs
create mode 100644 ballista/core/src/execution_plans/buffer.rs
create mode 100644 ballista/core/src/execution_plans/ordered_range_repartition.rs
create mode 100644 ballista/core/src/execution_plans/per_partition_filter.rs
create mode 100644 ballista/core/src/execution_plans/range_repartition_common.rs
create mode 100644 ballista/core/src/execution_plans/runtime_stats.rs
create mode 100644 ballista/core/src/execution_plans/unordered_range_repartition.rs
create mode 100644 ballista/core/src/kll.rs
create mode 100644 ballista/core/src/optimizer.rs
create mode 100644 ballista/scheduler/src/state/aqe/test/broadcast_thresholds.rs
create mode 100644 ballista/scheduler/src/state/aqe/test/stats_table.rs
create mode 100644 chaos-testing/Cargo.toml
create mode 100644 chaos-testing/README.md
create mode 100644 chaos-testing/src/bin/chaos-executor.rs
copy examples/examples/custom-scheduler.rs => chaos-testing/src/bin/chaos-scheduler.rs (51%)
create mode 100644 chaos-testing/src/budget.rs
create mode 100644 chaos-testing/src/cluster.rs
create mode 100644 chaos-testing/src/fixture.rs
copy ballista/scheduler/src/state/aqe/optimizer_rule/mod.rs => chaos-testing/src/lib.rs (75%)
create mode 100644 chaos-testing/src/registry.rs
create mode 100644 chaos-testing/src/udf.rs
create mode 100644 chaos-testing/tests/common/mod.rs
create mode 100644 chaos-testing/tests/ha.rs
copy ci/scripts/{rust_docs.sh => rust_config_docs_check.sh} (81%)
copy ci/scripts/rust_docs.sh => dev/update_config_docs.sh (77%)
create mode 100644 docs/source/changelog/54.0.0.md
create mode 100644 taplo.toml
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org