You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "xuzifu666 (via GitHub)" <gi...@apache.org> on 2024/11/04 02:49:11 UTC

[PR] System opt in [paimon]

xuzifu666 opened a new pull request, #4436:
URL: https://github.com/apache/paimon/pull/4436

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close #xxx
   
   <!-- What is the purpose of the change -->
   1. Currently IN filter cannot pushdown to snapshot/tag/schema system tables and IN would tramsform to OR(contians all Equal LeafPredicate), if user query with IN filter would cost more unnessary IO in a large number of snapshots/tag/schema;
   2. TagTable predicate specified as LeafPredicate which cannot resolve other kind of Predicate, this pr had improved it.
   3. Add APIs for get snapshots/tags/schemas which get multiple numbers for a list argument.
   
   such like sql: select snapshot_id, schema_id, commit_user from T$snapshots where snapshot_id in (1, 3);
   before the pr would query all snapshots files firstly, then filter with 1 and 3;
   after the pr  would query only snapshots file with 1 and 3, not query all snapshots file, which can reduce unnessary IO.
   note: if filter contains other fields would handle regress as before( like: select snapshot_id, schema_id, commit_user from T$snapshots where snapshot_id in (1, 3) or/and commit_user='xxx'; ), just work only snapshot_id IN filter.
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on PR #4436:
URL: https://github.com/apache/paimon/pull/4436#issuecomment-2456779894

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #4436:
URL: https://github.com/apache/paimon/pull/4436#issuecomment-2458586303

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4436:
URL: https://github.com/apache/paimon/pull/4436#discussion_r1828581456


##########
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java:
##########
@@ -417,6 +417,13 @@ public List<Path> snapshotPaths(Predicate<Long> predicate) throws IOException {
                 .collect(Collectors.toList());
     }
 
+    public Iterator<Snapshot> snapshotsWithIds(List<Long> snapshotIds) {

Review Comment:
   snapshotsWithId



##########
paimon-core/src/main/java/org/apache/paimon/schema/SchemaManager.java:
##########
@@ -128,6 +128,10 @@ public List<TableSchema> listAll() {
         return listAllIds().stream().map(this::schema).collect(Collectors.toList());
     }
 
+    public List<TableSchema> schemasWithIds(List<Long> schemaIds) {

Review Comment:
   schemasWithId



##########
paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java:
##########
@@ -217,18 +217,49 @@ public RecordReader<InternalRow> createReader(Split split) {
                 throw new IllegalArgumentException("Unsupported split: " + split.getClass());
             }
             Path location = ((TagsSplit) split).location;
-            LeafPredicate predicate = ((TagsSplit) split).tagName;
+            Predicate predicate = ((TagsSplit) split).tagPredicate;
             TagManager tagManager = new TagManager(fileIO, location, branch);
+            String leafName = "tag_name";

Review Comment:
   first : private static final TAG_NAME = "tag_name";
   second : replace all "tag_name" to TAG_NAME in this class
   



##########
paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java:
##########
@@ -217,18 +217,49 @@ public RecordReader<InternalRow> createReader(Split split) {
                 throw new IllegalArgumentException("Unsupported split: " + split.getClass());
             }
             Path location = ((TagsSplit) split).location;
-            LeafPredicate predicate = ((TagsSplit) split).tagName;
+            Predicate predicate = ((TagsSplit) split).tagPredicate;
             TagManager tagManager = new TagManager(fileIO, location, branch);
+            String leafName = "tag_name";
 
             Map<String, Tag> nameToSnapshot = new TreeMap<>();
+            Map<String, Tag> predicateMap = new TreeMap<>();
+            if (predicate != null) {
+                if (predicate instanceof LeafPredicate
+                        && ((LeafPredicate) predicate).function() instanceof Equal
+                        && ((LeafPredicate) predicate).literals().get(0) instanceof BinaryString
+                        && predicate.visit(LeafPredicateExtractor.INSTANCE).get(leafName) != null) {
+                    String equalValue = ((LeafPredicate) predicate).literals().get(0).toString();
+                    if (tagManager.tagExists(equalValue)) {
+                        predicateMap.put(equalValue, tagManager.tag(equalValue));
+                    }
+                }
 
-            if (predicate != null
-                    && predicate.function() instanceof Equal
-                    && predicate.literals().get(0) instanceof BinaryString) {
-                String equalValue = predicate.literals().get(0).toString();
-                if (tagManager.tagExists(equalValue)) {
-                    nameToSnapshot.put(equalValue, tagManager.tag(equalValue));
+                if (predicate instanceof CompoundPredicate) {
+                    CompoundPredicate compoundPredicate = (CompoundPredicate) predicate;
+                    // optimize for IN filter
+                    if ((compoundPredicate.function()) instanceof Or) {
+                        List<Predicate> children = compoundPredicate.children();
+                        for (Predicate leaf : children) {
+                            if (leaf instanceof LeafPredicate
+                                    && (((LeafPredicate) leaf).function() instanceof Equal)

Review Comment:
   Why here not judge:
   ((LeafPredicate) predicate).literals().get(0) instanceof BinaryString
   ?



##########
paimon-core/src/main/java/org/apache/paimon/table/system/SnapshotsTable.java:
##########
@@ -226,6 +229,22 @@ public InnerTableRead withFilter(Predicate predicate) {
                         handleLeafPredicate(leaf, leafName);
                     }
                 }
+

Review Comment:
   List<Predicate> children = compoundPredicate.children();
   if ((compoundPredicate.function()) instanceof And) {
   xxx;
   } else if (if ((compoundPredicate.function()) instanceof And) {) {
   xxx;
   } else {
   throw xxx;
   }



##########
paimon-core/src/main/java/org/apache/paimon/table/system/SnapshotsTable.java:
##########
@@ -206,6 +208,7 @@ private class SnapshotsRead implements InnerTableRead {
         private RowType readType;
         private Optional<Long> optionalFilterSnapshotIdMax = Optional.empty();
         private Optional<Long> optionalFilterSnapshotIdMin = Optional.empty();
+        private List<Long> snapshotIds = new ArrayList<>();

Review Comment:
   final



##########
paimon-core/src/main/java/org/apache/paimon/table/system/SchemasTable.java:
##########
@@ -203,6 +205,7 @@ private class SchemasRead implements InnerTableRead {
 
         private Optional<Long> optionalFilterSchemaIdMax = Optional.empty();
         private Optional<Long> optionalFilterSchemaIdMin = Optional.empty();
+        private List<Long> schemaIds = new ArrayList<>();

Review Comment:
   final



##########
paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java:
##########
@@ -217,18 +217,49 @@ public RecordReader<InternalRow> createReader(Split split) {
                 throw new IllegalArgumentException("Unsupported split: " + split.getClass());
             }
             Path location = ((TagsSplit) split).location;
-            LeafPredicate predicate = ((TagsSplit) split).tagName;
+            Predicate predicate = ((TagsSplit) split).tagPredicate;
             TagManager tagManager = new TagManager(fileIO, location, branch);
+            String leafName = "tag_name";

Review Comment:
   first : private static final TAG_NAME = "tag_name";
   second : replace all "tag_name" to TAG_NAME in this class
   



##########
paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java:
##########
@@ -217,18 +217,49 @@ public RecordReader<InternalRow> createReader(Split split) {
                 throw new IllegalArgumentException("Unsupported split: " + split.getClass());
             }
             Path location = ((TagsSplit) split).location;
-            LeafPredicate predicate = ((TagsSplit) split).tagName;
+            Predicate predicate = ((TagsSplit) split).tagPredicate;
             TagManager tagManager = new TagManager(fileIO, location, branch);
+            String leafName = "tag_name";
 
             Map<String, Tag> nameToSnapshot = new TreeMap<>();
+            Map<String, Tag> predicateMap = new TreeMap<>();
+            if (predicate != null) {
+                if (predicate instanceof LeafPredicate
+                        && ((LeafPredicate) predicate).function() instanceof Equal
+                        && ((LeafPredicate) predicate).literals().get(0) instanceof BinaryString
+                        && predicate.visit(LeafPredicateExtractor.INSTANCE).get(leafName) != null) {
+                    String equalValue = ((LeafPredicate) predicate).literals().get(0).toString();
+                    if (tagManager.tagExists(equalValue)) {
+                        predicateMap.put(equalValue, tagManager.tag(equalValue));
+                    }
+                }
 
-            if (predicate != null
-                    && predicate.function() instanceof Equal
-                    && predicate.literals().get(0) instanceof BinaryString) {
-                String equalValue = predicate.literals().get(0).toString();
-                if (tagManager.tagExists(equalValue)) {
-                    nameToSnapshot.put(equalValue, tagManager.tag(equalValue));
+                if (predicate instanceof CompoundPredicate) {
+                    CompoundPredicate compoundPredicate = (CompoundPredicate) predicate;
+                    // optimize for IN filter
+                    if ((compoundPredicate.function()) instanceof Or) {
+                        List<Predicate> children = compoundPredicate.children();
+                        for (Predicate leaf : children) {
+                            if (leaf instanceof LeafPredicate
+                                    && (((LeafPredicate) leaf).function() instanceof Equal)
+                                    && predicate
+                                                    .visit(LeafPredicateExtractor.INSTANCE)
+                                                    .get(leafName)
+                                            != null) {
+                                String equalValue =
+                                        ((LeafPredicate) leaf).literals().get(0).toString();
+                                predicateMap.put(equalValue, tagManager.tag(equalValue));

Review Comment:
   if (tagManager.tagExists(equalValue)) {
   predicateMap.put(equalValue, tagManager.tag(equalValue));
   }



##########
paimon-core/src/main/java/org/apache/paimon/table/system/TagsTable.java:
##########
@@ -217,18 +217,49 @@ public RecordReader<InternalRow> createReader(Split split) {
                 throw new IllegalArgumentException("Unsupported split: " + split.getClass());
             }
             Path location = ((TagsSplit) split).location;
-            LeafPredicate predicate = ((TagsSplit) split).tagName;
+            Predicate predicate = ((TagsSplit) split).tagPredicate;
             TagManager tagManager = new TagManager(fileIO, location, branch);
+            String leafName = "tag_name";
 
             Map<String, Tag> nameToSnapshot = new TreeMap<>();
+            Map<String, Tag> predicateMap = new TreeMap<>();
+            if (predicate != null) {
+                if (predicate instanceof LeafPredicate
+                        && ((LeafPredicate) predicate).function() instanceof Equal
+                        && ((LeafPredicate) predicate).literals().get(0) instanceof BinaryString
+                        && predicate.visit(LeafPredicateExtractor.INSTANCE).get(leafName) != null) {
+                    String equalValue = ((LeafPredicate) predicate).literals().get(0).toString();
+                    if (tagManager.tagExists(equalValue)) {
+                        predicateMap.put(equalValue, tagManager.tag(equalValue));
+                    }
+                }
 
-            if (predicate != null
-                    && predicate.function() instanceof Equal
-                    && predicate.literals().get(0) instanceof BinaryString) {
-                String equalValue = predicate.literals().get(0).toString();
-                if (tagManager.tagExists(equalValue)) {
-                    nameToSnapshot.put(equalValue, tagManager.tag(equalValue));
+                if (predicate instanceof CompoundPredicate) {
+                    CompoundPredicate compoundPredicate = (CompoundPredicate) predicate;
+                    // optimize for IN filter
+                    if ((compoundPredicate.function()) instanceof Or) {
+                        List<Predicate> children = compoundPredicate.children();
+                        for (Predicate leaf : children) {
+                            if (leaf instanceof LeafPredicate
+                                    && (((LeafPredicate) leaf).function() instanceof Equal)
+                                    && predicate
+                                                    .visit(LeafPredicateExtractor.INSTANCE)
+                                                    .get(leafName)
+                                            != null) {
+                                String equalValue =
+                                        ((LeafPredicate) leaf).literals().get(0).toString();
+                                predicateMap.put(equalValue, tagManager.tag(equalValue));

Review Comment:
   if (tagManager.tagExists(equalValue)) {
   predicateMap.put(equalValue, tagManager.tag(equalValue));
   }



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 closed pull request #4436: [core] Optizime IN filter pushdown to snapshot/tag/schema system tables
URL: https://github.com/apache/paimon/pull/4436


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4436:
URL: https://github.com/apache/paimon/pull/4436#discussion_r1830423025


##########
paimon-core/src/main/java/org/apache/paimon/table/system/SchemasTable.java:
##########
@@ -223,6 +226,22 @@ public InnerTableRead withFilter(Predicate predicate) {
                         handleLeafPredicate(leaf, leafName);
                     }
                 }
+
+                // optimize for IN filter
+                if ((compoundPredicate.function()) instanceof Or) {

Review Comment:
   We should extract a class to reduce duplicate codes, this can be done in a new PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4436:
URL: https://github.com/apache/paimon/pull/4436


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on PR #4436:
URL: https://github.com/apache/paimon/pull/4436#issuecomment-2456060536

   @wwj6591812 Thanks review,had addressed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 commented on code in PR #4436:
URL: https://github.com/apache/paimon/pull/4436#discussion_r1830429463


##########
paimon-core/src/main/java/org/apache/paimon/table/system/SchemasTable.java:
##########
@@ -223,6 +226,22 @@ public InnerTableRead withFilter(Predicate predicate) {
                         handleLeafPredicate(leaf, leafName);
                     }
                 }
+
+                // optimize for IN filter
+                if ((compoundPredicate.function()) instanceof Or) {

Review Comment:
   OK,would refactor it as suggest,Thanks @JingsongLi 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 closed pull request #4436: [core] Optizime IN filter pushdown to snapshot/tag/schema system tables
URL: https://github.com/apache/paimon/pull/4436


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Optizime IN filter pushdown to snapshot/tag/schema system tables [paimon]

Posted by "xuzifu666 (via GitHub)" <gi...@apache.org>.
xuzifu666 closed pull request #4436: [core] Optizime IN filter pushdown to snapshot/tag/schema system tables
URL: https://github.com/apache/paimon/pull/4436


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org