You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/09/04 02:27:40 UTC

Re: [PR] [core][format][spark] Support nested field predicate pushdown [paimon]

JingsongLi commented on code in PR #9423:
URL: https://github.com/apache/paimon/pull/9423#discussion_r3930362467


##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -273,9 +278,31 @@ public FilterPredicate visitNotIn(FieldRef fieldRef, List<Object> literals) {
             throw new UnsupportedOperationException();
         }
 
+        /**
+         * A nested field carries no index into the file, only a path, so it is re-dispatched under
+         * a {@link FieldRef} naming that path. Every other transform - casts, string functions -
+         * has no column of its own to filter on and is given up here.
+         */
         @Override
         public FilterPredicate visitNonFieldLeaf(LeafPredicate predicate) {
-            throw new UnsupportedOperationException();
+            if (!(predicate.transform() instanceof NestedFieldTransform)) {
+                throw new UnsupportedOperationException();
+            }
+            NestedFieldTransform nested = (NestedFieldTransform) predicate.transform();
+            // The path reaches parquet-mr as a dot-joined string, which it splits back into
+            // components. A component that itself contains a dot does not survive that round trip:
+            // the filter would address a column the file does not hold, and a missing column reads
+            // as all-null, pruning row groups that actually match. Give up the pruning instead.
+            if (nested.fieldRef().name().indexOf('.') >= 0) {
+                throw new UnsupportedOperationException();
+            }
+            for (String component : nested.path()) {
+                if (component.indexOf('.') >= 0) {
+                    throw new UnsupportedOperationException();
+                }
+            }
+            FieldRef pathRef = new FieldRef(UNUSED_INDEX, nested.fieldName(), nested.outputType());

Review Comment:
   [P1] Reject collisions between a nested path and a dotted top-level name
   
   Joining the components here is still ambiguous even when none of the individual components contains a dot. A valid schema can contain both `s ROW<a INT>` and a top-level column named `s.a` (the existing flat dotted-name coverage confirms the latter is supported). A predicate on the nested `s.a` is re-dispatched as `FieldRef("s.a")`; `findFileColumn` checks an exact top-level name before walking the split components, so it binds this predicate to the top-level `s.a` column instead of `s -> a`.
   
   If the top-level column's row-group stats do not match while the nested column does, parquet-mr prunes the row group and Spark's residual filter never sees the matching row. Please retain the component path through column resolution/build a `ColumnPath`, or at minimum reject nested pushdown when the file schema has a top-level field equal to the dot-joined path. An end-to-end row-group test with both columns and opposing values should expose the false negative.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org