You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2019/05/25 22:04:02 UTC

conflict with multiple jobs writing to different partitions but same baseDir

lets say i have 2 dataframe jobs that write to /somedir/a=1 and
somedir/a=2. these can run at same time without issues.

but now i get excited about dynamic partitioning. so i add "a" as a column
to my 2 dataframes, set the option partitionOverwriteMode=dynamic, add
partitionBy("a": _*) to the writing, and write both to /somedir. this works
fine, and the result is the same, but i can no longer safely run both jobs
at same time, because they both try to create and delete
/somedir/_temporary, and i end up with exceptions about files not found in
/somedir/_temporary.

this makes me wonder why _temporary is hardcoded. why not _temporary_<uuid>?