[GitHub] [spark] jeanlyn commented on pull request #41628: [SPARK-38230][SQL] InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions

["jeanlyn (via GitHub)" <gi...@apache.org>]

jeanlyn commented on PR #41628:
URL: https://github.com/apache/spark/pull/41628#issuecomment-1647887447

   Thanks @attilapiros , and sorry for the late reply. The solution indeed do some hacks on custom locations of partition, because i think users rarely modify individual partition paths, and I couldn't find a better way to handle custom partition locations both to ensure atomicity, but also consider reducing memory pressure on large partitions table. Do you have a better suggestion?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org