[GitHub] [iceberg] aokolnychyi commented on a change in pull request #1665: Parquet: Introduce a limit to evaluate IN predicates

[GitBox <gi...@apache.org>]

aokolnychyi commented on a change in pull request #1665:
URL: https://github.com/apache/iceberg/pull/1665#discussion_r512220298



##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -84,6 +84,9 @@ private TableProperties() {
   public static final String PARQUET_BATCH_SIZE = "read.parquet.vectorization.batch-size";
   public static final int PARQUET_BATCH_SIZE_DEFAULT = 5000;
 
+  public static final String PARQUET_IN_LIMIT = "read.parquet.in-predicate-limit";
+  public static final int PARQUET_IN_LIMIT_DEFAULT = 200;

Review comment:
       Spark's default value is 10 but it also rewrites IN as OR/EQUALS. We can do better that this because of the way we evaluate IN predicates. On datasets we tested this, we saw performance improvements on IN predicates with up to 200 elements (on sorted column). We may increase the default value a bit but I am very reluctant to make it more than 300 since our jobs were 20% slower with IN predicates that contained 300 elements. That being said, the optimal value still depends on a particular query and table.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org