You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "plusplusjiajia (via GitHub)" <gi...@apache.org> on 2026/07/03 09:01:20 UTC
[PR] [fix] Apply row filter auth during scan planning [paimon]
plusplusjiajia opened a new pull request, #8447:
URL: https://github.com/apache/paimon/pull/8447
### Purpose
Apply row filter auth predicates during scan planning, not only at read time. This lets auth filters participate in partition pruning, non-partition filter pushdown, and limit pushdown decisions, while keeping
reader-side auth filtering as final enforcement.
This fixes cases where a partition row filter allows data in some partitions, but queries without explicit partition predicates may plan splits from unauthorized partitions first and return empty results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix] Apply row filter auth during scan planning [paimon]
Posted by "plusplusjiajia (via GitHub)" <gi...@apache.org>.
plusplusjiajia commented on code in PR #8447:
URL: https://github.com/apache/paimon/pull/8447#discussion_r3520091253
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -237,7 +237,7 @@ public SnapshotReader withFilter(Predicate predicate) {
splitPartitionPredicatesAndDataPredicates(
predicate, tableSchema.logicalRowType(), tableSchema.partitionKeys());
if (pair.getLeft().isPresent()) {
- scan.withPartitionFilter(pair.getLeft().get());
+ scan.withPartitionFilterAnd(pair.getLeft().get());
Review Comment:
@JingsongLi Agree. withPartitionFilterAnd is no longer needed for this PR now that authPartitionFilter is separate. I'll revert that part and keep the existing withPartitionFilter(...) behavior for normal query filters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix] Apply row filter auth during scan planning [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8447:
URL: https://github.com/apache/paimon/pull/8447#discussion_r3520066489
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -237,7 +237,7 @@ public SnapshotReader withFilter(Predicate predicate) {
splitPartitionPredicatesAndDataPredicates(
predicate, tableSchema.logicalRowType(), tableSchema.partitionKeys());
if (pair.getLeft().isPresent()) {
- scan.withPartitionFilter(pair.getLeft().get());
+ scan.withPartitionFilterAnd(pair.getLeft().get());
Review Comment:
Do we still need to introduce `withPartitionFilterAnd` here? Since auth-derived partition predicates are now kept in a separate `authPartitionFilter` slot and combined with the normal `partitionFilter` only when computing the effective manifest filter, this PR could keep the existing `withPartitionFilter(...)` behavior for normal query filters. The repeated `withFilter(...)` partition-merge semantics feels like a separate behavior change that can be discussed independently if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix] Apply row filter auth during scan planning [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8447:
URL: https://github.com/apache/paimon/pull/8447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix] Apply row filter auth during scan planning [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8447:
URL: https://github.com/apache/paimon/pull/8447#discussion_r3520104485
##########
paimon-core/src/main/java/org/apache/paimon/table/source/AbstractDataTableScan.java:
##########
@@ -111,6 +117,37 @@ public final TableScan.Plan plan() {
protected abstract TableScan.Plan planWithoutAuth();
+ private void applyAuthFilter(@Nullable Predicate authPredicate) {
+ if (Objects.equals(authPredicate, appliedAuthPredicate)) {
+ return;
+ }
+ appliedAuthPredicate = authPredicate;
+
+ PartitionPredicate authPartitionFilter = null;
+ boolean hasNonPartitionPart = false;
+ if (authPredicate != null) {
+ // Remap field-id FieldRefs to positional indices by name (as doAuth does on read), so
+ // pruning stays correct across schema evolution.
+ Predicate remappedAuth =
+ TableQueryAuthResult.remapPredicate(authPredicate, schema.logicalRowType());
+ if (remappedAuth != null) {
+ Pair<Optional<PartitionPredicate>, List<Predicate>> split =
+ PartitionPredicate.splitPartitionPredicatesAndDataPredicates(
+ remappedAuth, schema.logicalRowType(), schema.partitionKeys());
+ authPartitionFilter = split.getLeft().orElse(null);
+ hasNonPartitionPart = !split.getRight().isEmpty();
+ }
+ }
+
+ // Push only the partition part, to a dedicated slot overwritten/cleared each plan() so a
+ // changed/removed auth leaves no stale pruning. The full filter is enforced at read time.
+ snapshotReader.manifestsReader().withAuthPartitionFilter(authPartitionFilter);
+ if (hasNonPartitionPart) {
+ // Non-partition auth removes rows at read time, so limit push down is unsafe.
+ snapshotReader.markHasNonPartitionFilter();
Review Comment:
If this marker is only used to make `DataTableBatchScan.applyPushDownLimit()` return empty, can we keep this state local to `DataTableBatchScan` instead? For example, when the auth predicate has a non-partition part, set `pushDownLimit = null` before planning. `null` already means no limit pushdown there, and this avoids adding `markHasNonPartitionFilter()` to `SnapshotReader` for a read-time auth filter that was not actually pushed through `withFilter(...)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix] Apply row filter auth during scan planning [paimon]
Posted by "plusplusjiajia (via GitHub)" <gi...@apache.org>.
plusplusjiajia commented on code in PR #8447:
URL: https://github.com/apache/paimon/pull/8447#discussion_r3520179351
##########
paimon-core/src/main/java/org/apache/paimon/table/source/AbstractDataTableScan.java:
##########
@@ -111,6 +117,37 @@ public final TableScan.Plan plan() {
protected abstract TableScan.Plan planWithoutAuth();
+ private void applyAuthFilter(@Nullable Predicate authPredicate) {
+ if (Objects.equals(authPredicate, appliedAuthPredicate)) {
+ return;
+ }
+ appliedAuthPredicate = authPredicate;
+
+ PartitionPredicate authPartitionFilter = null;
+ boolean hasNonPartitionPart = false;
+ if (authPredicate != null) {
+ // Remap field-id FieldRefs to positional indices by name (as doAuth does on read), so
+ // pruning stays correct across schema evolution.
+ Predicate remappedAuth =
+ TableQueryAuthResult.remapPredicate(authPredicate, schema.logicalRowType());
+ if (remappedAuth != null) {
+ Pair<Optional<PartitionPredicate>, List<Predicate>> split =
+ PartitionPredicate.splitPartitionPredicatesAndDataPredicates(
+ remappedAuth, schema.logicalRowType(), schema.partitionKeys());
+ authPartitionFilter = split.getLeft().orElse(null);
+ hasNonPartitionPart = !split.getRight().isEmpty();
+ }
+ }
+
+ // Push only the partition part, to a dedicated slot overwritten/cleared each plan() so a
+ // changed/removed auth leaves no stale pruning. The full filter is enforced at read time.
+ snapshotReader.manifestsReader().withAuthPartitionFilter(authPartitionFilter);
+ if (hasNonPartitionPart) {
+ // Non-partition auth removes rows at read time, so limit push down is unsafe.
+ snapshotReader.markHasNonPartitionFilter();
Review Comment:
@JingsongLi good point! dropped it from SnapshotReader, kept the flag scan-local in DataTableBatchScan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix] Apply row filter auth during scan planning [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8447:
URL: https://github.com/apache/paimon/pull/8447#issuecomment-4876038756
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix] Apply row filter auth during scan planning [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8447:
URL: https://github.com/apache/paimon/pull/8447#discussion_r3520066489
##########
paimon-core/src/main/java/org/apache/paimon/table/source/snapshot/SnapshotReaderImpl.java:
##########
@@ -237,7 +237,7 @@ public SnapshotReader withFilter(Predicate predicate) {
splitPartitionPredicatesAndDataPredicates(
predicate, tableSchema.logicalRowType(), tableSchema.partitionKeys());
if (pair.getLeft().isPresent()) {
- scan.withPartitionFilter(pair.getLeft().get());
+ scan.withPartitionFilterAnd(pair.getLeft().get());
Review Comment:
Do we still need to introduce here? Since auth-derived partition predicates are now kept in a separate slot and combined with the normal only when computing the effective manifest filter, this PR could keep the existing behavior for normal query filters. The repeated partition-merge semantics feels like a separate behavior change that can be discussed independently if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org