[GitHub] [spark] vanzin commented on issue #24142: [SPARK-27194][core] Job failures when task attempts do not clean up spark-staging parquet files

[GitBox <gi...@apache.org>]

vanzin commented on issue #24142: [SPARK-27194][core] Job failures when task attempts do not clean up spark-staging parquet files
URL: https://github.com/apache/spark/pull/24142#issuecomment-474605554
 
 
   > The fix is not changing the behavior of writing to temp location and then moving to final location. the fix deals only with file name ( even in temp/final location, the file names must not conflict)
   
   You seem to be saying that with your fix, it's possible that the partially written file will end up in the final output.
   
   If that's true, then that's worse than the current status: right now, your app will fail. With that partial fix, the app will generate wrong data.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org