[GitHub] [spark] steveloughran commented on pull request #39185: [SPARK-41551][SQL] Dynamic/absolute path support in PathOutputCommitters

[GitBox <gi...@apache.org>]

steveloughran commented on PR #39185:
URL: https://github.com/apache/spark/pull/39185#issuecomment-1364000436

   1. I think i should add a test which verifies the abs path temp file is always under .spark-staging *and* includes the task attempt id in it. 
   1. jobs created through RDDs don't get unique IDs across processes, they just get the RDD counter. so > 1 job writing to same dest may really ruin the lives of others even when updating different partitions. I can fix that here or file a separate JIRA


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org