[GitHub] [spark] jeanlyn commented on a diff in pull request #41628: [SPARK-38230][SQL] InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions

["jeanlyn (via GitHub)" <gi...@apache.org>]

jeanlyn commented on code in PR #41628:
URL: https://github.com/apache/spark/pull/41628#discussion_r1234694418


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala:
##########
@@ -261,6 +261,20 @@ trait ExternalCatalog {
       table: String,
       partialSpec: Option[TablePartitionSpec] = None): Seq[CatalogTablePartition]

Review Comment:
   Yeah, it's hard to add pattern to `listPartitions`.  `getPartitionsByNames` also had argument with `Map<String, String> partialPartSpec` like `getPartitionsByNames(Table tbl,
         Map<String, String> partialPartSpec)`. It's more efficient in Hive. However, it still pull all partitions info when using dynamic partition overwrite.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org