[GitHub] [spark] coalchan commented on a change in pull request #35549: [SPARK-38230][SQL] InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions in most cases

[GitBox <gi...@apache.org>]

coalchan commented on a change in pull request #35549:
URL: https://github.com/apache/spark/pull/35549#discussion_r822366293



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala
##########
@@ -92,16 +92,30 @@ case class InsertIntoHadoopFsRelationCommand(
 
     var initialMatchingPartitions: Seq[TablePartitionSpec] = Nil
     var customPartitionLocations: Map[TablePartitionSpec, String] = Map.empty
-    var matchingPartitions: Seq[CatalogTablePartition] = Seq.empty
 
     // When partitions are tracked by the catalog, compute all custom partition locations that
     // may be relevant to the insertion job.
     if (partitionsTrackedByCatalog) {

Review comment:
       Thanks for your response. `tracksPartitionsInCatalog` is set true in [HiveExternalCatalog.restoreHiveSerdeTable](https://github.com/apache/spark/blob/52e7602344036a649bb19ed642dbd74bc7ee9cb1/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala#L800)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org