You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@auron.apache.org by "squanch-c (via GitHub)" <gi...@apache.org> on 2026/10/13 05:36:30 UTC
[PR] [AURON #2579] Fix zero-column round-robin shuffle presort [auron]
squanch-c opened a new pull request, #2580:
URL: https://github.com/apache/auron/pull/2580
# Which issue does this PR close?
Closes apache/auron#2579
# Rationale for this change
`COUNT(*)` above an unkeyed `REPARTITION` can leave zero-column input batches
with positive row counts. Native round-robin presort then passes no columns
to `RowConverter`, which produces zero key rows and causes division by zero
in `ExternalSorter`.
Minimal reproduction with Auron native shuffle conversion enabled:
```sql
SELECT count(*) AS row_count
FROM (
SELECT /*+ REPARTITION(2) */ *
FROM (
SELECT explode(array(
CAST(0 AS BIGINT), CAST(1 AS BIGINT),
CAST(2 AS BIGINT), CAST(3 AS BIGINT)
)) AS id
) seed
) inputs;
```
Expected result: `4`, as returned by the JVM engine and by the same query
without the `REPARTITION` hint. The original Native execution fails with:
```text
output_with_sender[Shuffle] error:
output_with_sender[Sort] error:
attempt to divide by zero
```
The captured failing plan includes `Project (Output: [])` followed by
`NativeShuffleExchange (Input: [], RoundRobinPartitioning(2))`. This example
reproduced the failure on Spark 3.5.9 using the RSS path; SQL execution with
a rebuilt patched JAR has not yet been verified.
An empty-key guard existed in the former sorter implementation in #958 and
was removed with that implementation in #1078. This restores the behavior
in the shared presort helper, which has no limit or offset.
# What changes are included in this PR?
- Return the input from `create_default_ascending_sort_exec` when no sort keys
are present. Both local and RSS round-robin shuffle use this helper.
- Add two regression tests covering zero-column batches, multiple batches and
partitions, empty input, and values with and without sort keys.
# Are there any user-facing changes?
`COUNT(*)` over native round-robin repartitioning no longer fails when column
pruning leaves zero input columns. No configuration change is required.
# How was this patch tested?
On macOS arm64 with nightly-2025-05-09, upstream vendor setup and unchanged
`Cargo.lock`:
```text
cargo test --locked -p datafusion-ext-plans test_default_sort_ -- --nocapture
2 passed; 0 failed; 189 filtered out
```
Rust workspace formatting also passed:
```text
cargo fmt --all -- --check
Exit code: 0
```
Pinned rustfmt and `git diff --check` passed. An earlier negative-control run
of the zero-column regression test reproduced division by zero with only the
helper guard removed.
The [Rust Test workflow](https://github.com/squanch-c/auron/actions/runs/38246720520)
also passed on Ubuntu 24.04:
```text
cargo fmt --all -- --check
cargo clippy --all-targets --workspace -- -D warnings
cargo test --workspace --all-features
```
All technical checks in [fork PR #1](https://github.com/squanch-c/auron/pull/1/checks)
passed: Rust tests and lint, repository style and license checks, the Spark
3.0-3.5 and 4.0-4.2 TPC-DS build/test matrix, and the Celeborn, Uniffle, Hudi,
Paimon, Iceberg and Flink workflows. The TPC-DS jobs build Linux Auron JARs
from the PR code and run result and plan checks.
The minimal zero-column SQL above has not yet been rerun with a patched JAR;
the added regression coverage for this case is in Rust.
# Was this patch authored or co-authored using generative AI tooling?
- [x] Yes
- [ ] No
Generated-by: OpenAI Codex (GPT-6; exact client build unavailable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2579] Fix zero-column round-robin shuffle presort [auron]
Posted by "squanch-c (via GitHub)" <gi...@apache.org>.
squanch-c commented on PR #2580:
URL: https://github.com/apache/auron/pull/2580#issuecomment-6141456789
Runtime validation has now been completed on Spark 3.5.9 with Celeborn (RSS), using a rebuilt Auron 9.0.0-SNAPSHOT JAR with this fix backported.
The previously failing zero-column round-robin SQL now completes successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2579] Fix zero-column round-robin shuffle presort [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 commented on PR #2580:
URL: https://github.com/apache/auron/pull/2580#issuecomment-6256493102
@squanch-c Thanks for the contribution! @lyne7-sc Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2579] Fix zero-column round-robin shuffle presort [auron]
Posted by "slfan1989 (via GitHub)" <gi...@apache.org>.
slfan1989 merged PR #2580:
URL: https://github.com/apache/auron/pull/2580
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2579] Fix zero-column round-robin shuffle presort [auron]
Posted by "squanch-c (via GitHub)" <gi...@apache.org>.
squanch-c commented on PR #2580:
URL: https://github.com/apache/auron/pull/2580#issuecomment-6183952133
> LGTM.
>
> The fix preserves zero-column row counts, and the regression tests cover the failure case. Please update the PR description to reflect the completed SQL validation.
Thank you for the feedback. I have updated the PR description accordingly. Could you please review it again and let me know if it does not follow the project’s conventions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AURON #2579] Fix zero-column round-robin shuffle presort [auron]
Posted by "squanch-c (via GitHub)" <gi...@apache.org>.
squanch-c commented on PR #2580:
URL: https://github.com/apache/auron/pull/2580#issuecomment-6168630581
> The fix makes sense to me. could we also add the reproduction sql as a Spark regression test?
Thanks for the review and the suggestion! I’ve added the SQL reproduction as a Spark regression test and verified that it passes with Spark 3.5.9 and Scala 2.12 using:
./build/mvn \
-Ppre,spark-3.5,scala-2.12,spark-tests \
-DsparkVersion=3.5.9 \
-pl auron-spark-tests/spark35 \
test -DskipTests=false \
-Dsuites=org.apache.spark.sql.AuronDataFrameSuite \
-Dtests=AURON-2579
Could you please take another look when you have a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@auron.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org