You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/09/26 19:01:35 UTC
[I] Parquet scan claims equivalences from a pruning-only predicate, giving wrong ORDER BY and LIMIT results [datafusion]
adriangb opened a new issue, #25779:
URL: https://github.com/apache/datafusion/issues/25779
### Describe the bug
A Parquet scan with `datafusion.execution.parquet.pushdown_filters = false` (the default) can return rows in the wrong order, and `ORDER BY ... LIMIT` can return wrong rows.
With `pushdown_filters = false`, `ParquetSource` keeps a pushed-down predicate only for row group and page pruning, and it replies `PushedDown::No`, so a `FilterExec` stays above the scan. But `FileScanConfig::eq_properties` reads the same predicate through `FileSource::filter()` and adds its equalities to the scan's equivalence properties. The scan then claims that `a` is constant (for `a = 5`) or that `a` and `b` are equal (for `a = b`), while it emits rows that violate this.
An operator placed directly on the scan, below the `FilterExec`, trusts the false property. An order-preserving `RepartitionExec` takes its merge keys from the scan's normalized output ordering. Normalization removes the "constant" `a`, so the repartition merges streams that are sorted by `(a, b)` using `b` only. The `FilterExec` then removes the `a != 5` rows, and the rows that stay are out of order.
### To Reproduce
```sql
COPY (SELECT * FROM (VALUES (3, 9), (5, 1)) v(a, b)) TO 'mre/f1.parquet' STORED AS PARQUET;
COPY (SELECT * FROM (VALUES (5, 2)) v(a, b)) TO 'mre/f2.parquet' STORED AS PARQUET;
CREATE EXTERNAL TABLE t (a INT, b INT) STORED AS PARQUET LOCATION 'mre/' WITH ORDER (a ASC, b ASC);
SET datafusion.execution.target_partitions = 4;
SET datafusion.optimizer.repartition_file_scans = false;
-- Expected (5,1,1), (5,2,1). Actual: (5,2,1), (5,1,1)
SELECT a, b, count(*) FROM t WHERE a = 5 GROUP BY a, b ORDER BY b;
SET datafusion.optimizer.prefer_existing_sort = true;
-- Expected 1. Actual: 2
SELECT b FROM t WHERE a = 5 ORDER BY b LIMIT 1;
```
The plan of the `LIMIT` query shows the mismatch. The scan declares `output_ordering=[a, b]`, but the repartition merges on `b` only:
```
SortPreservingMergeExec: [b@0 ASC NULLS LAST], fetch=1
FilterExec: a@0 = 5, projection=[b@1], fetch=1
RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2, preserve_order=true, sort_exprs=b@1 ASC NULLS LAST
DataSourceExec: file_groups={2 groups: [...]}, projection=[a, b], output_ordering=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST], file_type=parquet, predicate=a@0 = 5, ...
```
The equality form fails in the same way. With files sorted by `a`, `WHERE a = b ORDER BY a` returns `5, 3` instead of `3, 5`, because the repartition merges on `b`.
### Expected behavior
The results are the same as with `pushdown_filters = true`, which is correct in all of these cases. DuckDB gives the same results.
### Additional context
- Reproduced on `main` at e8e41ae958.
- With the default configuration, the scan is split into `target_partitions` file groups, so no `RoundRobin` repartition is added below the `FilterExec`. `repartition_file_scans = false` is sufficient to trigger the bug. Declared file partitioning (`preserve_file_partitions`) should also trigger it, but I did not test that.
- The equivalence derivation was added in https://github.com/apache/datafusion/pull/16686 (for https://github.com/apache/datafusion/issues/16563). It is correct when `pushdown_filters = true`, because then the scan applies the filter to every row and the `FilterExec` is removed.
- Operators that filter pushdown moves through (aggregate, sort, projection, join, union) are not affected. They commute with the filter, and a property that holds on the filtered rows is sufficient for them. The failure needs a node that is inserted directly on the scan after pushdown.
- Dynamic filters (hash join, TopK) are not affected. They are not plain `=` expressions, and the equivalences are computed at planning time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Parquet scan claims equivalences from a pruning-only predicate, giving wrong ORDER BY and LIMIT results [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed issue #25779: Parquet scan claims equivalences from a pruning-only predicate, giving wrong ORDER BY and LIMIT results
URL: https://github.com/apache/datafusion/issues/25779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org