[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3745: Push down partition filter to Spark when Importing File Based Tables

[GitBox <gi...@apache.org>]

RussellSpitzer commented on a change in pull request #3745:
URL: https://github.com/apache/iceberg/pull/3745#discussion_r769114012



##########
File path: spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java
##########
@@ -779,10 +800,54 @@ public static TableIdentifier identifierToTableIdentifier(Identifier identifier)
             Object value = CatalystTypeConverters.convertToScala(catalystValue, field.dataType());
             values.put(field.name(), value.toString());
           });
-          return new SparkPartition(values, partition.path().toString(), format);
+          FileStatus fileStatus =
+              scala.collection.JavaConverters.seqAsJavaListConverter(partition.files()).asJava().get(0);
+
+          return new SparkPartition(values, fileStatus.getPath().getParent().toString(), format);
         }).collect(Collectors.toList());
   }
 
+  private static List getPartitionFilterExpressions(StructType schema,
+                                                    Map<String, String> partitionFilter) {
+    List<org.apache.spark.sql.catalyst.expressions.Expression> filterExpressions = new java.util.ArrayList<>();
+    for (Map.Entry<String, String> entry : partitionFilter.entrySet()) {
+      try {
+        // IllegalArgumentException is thrown if schema doesn't contain this entry,

Review comment:
       Just wondering, how does  Spark parse these. Is it just in the ParseExpressions code and then fed directly into here? I know in the RewriteDataProcedure we end up using the parser to do a parseExpression since then we don't have to deal with all of these internal transforms and such. I am wondering if a similar approach could work here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org