[GitHub] [spark] itskals commented on issue #21066: [SPARK-23977][CLOUD][WIP] Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism

[GitBox <gi...@apache.org>]

itskals commented on issue #21066: [SPARK-23977][CLOUD][WIP] Add commit protocol binding to Hadoop 3.1 PathOutputCommitter mechanism
URL: https://github.com/apache/spark/pull/21066#issuecomment-473780802
 
 
   Hi @steveloughran ,
   I had a doubt on the behaviour of `PathOutputCommitProtocol`, which throws an exception when `dynamicPartitionOverwrite` is true. I am not clear why it should throw this exception? What would happen if this code is opened. I tried to decode the comment on the parameter, " dynamic partition overwrite is not supported, so that committers for stores which do not support rename will not get confused.", but I got bit confused. There are no renames in this complete idea. 
   Can you kindly elaborate this?
   As far as I understand( please correct me if I am wrong) , when DynamicOverwrite is set, the committer(Directory/partition) will delete the existing contents(in REPLACE mode). Which is exactly what is expected. 
   Thanks.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org