[GitHub] [spark] rdblue commented on pull request #30141: [SPARK-33230][SQL] Hadoop committers to get unique job ID in "spark.sql.sources.writeJobUUID"

[GitBox <gi...@apache.org>]

rdblue commented on pull request #30141:
URL: https://github.com/apache/spark/pull/30141#issuecomment-715449287


   I think what we ended up doing was to generate a UUID in `OutputCommitter.setupJob` and place it in the Hadoop `Configuration`. Now that the configuration is reliable for a given stage, wouldn't that work here instead of using the one from Spark?
   
   I'm okay adding this. It matches what we do in v2 writes. The new API also passes a UUID in, so I think there is a reasonable precedent for it, even if the committer or writer could generate one itself.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org