You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "nuno-faria (via GitHub)" <gi...@apache.org> on 2025/07/01 09:15:50 UTC
[I] TopK Sort incorrectly pushed down past Join [datafusion]
nuno-faria opened a new issue, #16638:
URL: https://github.com/apache/datafusion/issues/16638
### Describe the bug
The `SortExec` physical operator is being incorrectly pushed down past a `LEFT ANTI JOIN`, returning incorrect results.
Disabling the `EnforceSorting` physical optimizer rule results in a correct plan.
### To Reproduce
```rust
use datafusion::error::Result;
use datafusion::prelude::SessionContext;
#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
ctx.sql("create table t1 (k int, v int);")
.await?
.collect()
.await?;
ctx.sql("create table t2 (k int, v int);")
.await?
.collect()
.await?;
ctx.sql("insert into t1 select i as k, i as v from generate_series(1, 10000) t(i);")
.await?
.collect()
.await?;
ctx.sql("insert into t2 values (1, 1);")
.await?
.collect()
.await?;
let df = ctx
.sql("select * from t1 left anti join t2 on t1.k = t2.k order by t1.k limit 2;")
.await?;
df.clone().explain(false, false)?.show().await?;
df.show().await?;
Ok(())
}
```
Which returns:
```txt
+---------------+----------------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+----------------------------------------------------------------------------------------+
| logical_plan | Sort: t1.k ASC NULLS LAST, fetch=2 |
| | LeftAnti Join: t1.k = t2.k |
| | TableScan: t1 projection=[k, v] |
| | TableScan: t2 projection=[k] |
| physical_plan | SortPreservingMergeExec: [k@0 ASC NULLS LAST], fetch=2 |
| | CoalesceBatchesExec: target_batch_size=8192, fetch=2 |
| | HashJoinExec: mode=CollectLeft, join_type=RightAnti, on=[(k@0, k@0)] |
| | DataSourceExec: partitions=1, partition_sizes=[1] |
| | SortExec: TopK(fetch=2), expr=[k@0 ASC NULLS LAST], preserve_partitioning=[true] |
| | RepartitionExec: partitioning=RoundRobinBatch(12), input_partitions=1 |
| | DataSourceExec: partitions=1, partition_sizes=[2] |
| | |
+---------------+----------------------------------------------------------------------------------------+
+------+------+
| k | v |
+------+------+
| 2 | 2 |
| 8193 | 8193 |
+------+------+
```
After disabling the `EnforceSorting` rule:
```txt
+---------------+-----------------------------------------------------------------------------------+
| plan_type | plan |
+---------------+-----------------------------------------------------------------------------------+
| logical_plan | Sort: t1.k ASC NULLS LAST, fetch=2 |
| | LeftAnti Join: t1.k = t2.k |
| | TableScan: t1 projection=[k, v] |
| | TableScan: t2 projection=[k] |
| physical_plan | SortExec: TopK(fetch=2), expr=[k@0 ASC NULLS LAST], preserve_partitioning=[false] |
| | CoalescePartitionsExec |
| | CoalesceBatchesExec: target_batch_size=8192 |
| | HashJoinExec: mode=CollectLeft, join_type=RightAnti, on=[(k@0, k@0)] |
| | DataSourceExec: partitions=1, partition_sizes=[1] |
| | RepartitionExec: partitioning=RoundRobinBatch(12), input_partitions=1 |
| | DataSourceExec: partitions=1, partition_sizes=[2] |
| | |
+---------------+-----------------------------------------------------------------------------------+
+---+---+
| k | v |
+---+---+
| 2 | 2 |
| 3 | 3 |
+---+---+
```
### Expected behavior
Not push down the sort in this case.
### Additional context
- Tested from main (7cdac33dabbca2c06fac11cf6d2995dbc490c4bd)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TopK Sort incorrectly pushed down past Join [datafusion]
Posted by "nuno-faria (via GitHub)" <gi...@apache.org>.
nuno-faria closed issue #16638: TopK Sort incorrectly pushed down past Join
URL: https://github.com/apache/datafusion/issues/16638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TopK Sort incorrectly pushed down past Join [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on issue #16638:
URL: https://github.com/apache/datafusion/issues/16638#issuecomment-3023878360
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] TopK Sort incorrectly pushed down past Join [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on issue #16638:
URL: https://github.com/apache/datafusion/issues/16638#issuecomment-3024090674
Thank you @nuno-faria for reporting this bug, submitted the PR try to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org