You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2026/06/27 10:28:13 UTC
[PR] [core] Support OR/nested partition predicate pruning in format table scan [paimon]
Zouxxyy opened a new pull request, #8367:
URL: https://github.com/apache/paimon/pull/8367
### Purpose
Support generic partition-directory pruning for format table scans when partition predicates contain OR/nested expressions.
Previously pruning used a per-field \`Map<String, Predicate>\` built from \`splitAnd\`, so any predicate referencing more than one partition field (for example \`(dt='20260625' AND hour<'16') OR (dt='20260624' AND hour>='16')\`) was dropped and the scan fell back to listing every partition directory.
This change:
- replaces the per-field pruning model with incremental partial evaluation of the whole partition predicate during directory descent, so AND/OR/nested/cross-field partition predicates can prune format-table directories correctly;
- keeps scan-path prefix optimization unchanged for leading equality conjuncts;
- fixes \`format-table.partition-path-only-value=true\` with default/null partitions by allowing the configured default partition directory name through hidden-path filtering and by passing \`defaultPartName\` into \`listPartitionEntries\`.
### Tests
- \`mvn -pl paimon-core -Pfast-build -Dtest='FormatTableScanTest,PartitionPathUtilsTest' -DfailIfNoTests=false test\`
- \`mvn -pl paimon-spark/paimon-spark-ut -Pspark3,fast-build -DskipTests install\`
- \`mvn -pl paimon-spark/paimon-spark-3.5 -Pspark3,fast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.FormatTableTest -Dtest=none test\`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support OR/nested partition predicate pruning in format table scan [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8367:
URL: https://github.com/apache/paimon/pull/8367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Support OR/nested partition predicate pruning in format table scan [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8367:
URL: https://github.com/apache/paimon/pull/8367#issuecomment-4829310482
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org