[jira] [Created] (SPARK-32395) Commit taskattempt's output after succeed in dynamic partition datawriter

["Wan Kun (Jira)" <ji...@apache.org>]

Wan Kun created SPARK-32395:
-------------------------------

             Summary: Commit taskattempt's output after succeed in dynamic partition datawriter
                 Key: SPARK-32395
                 URL: https://issues.apache.org/jira/browse/SPARK-32395
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Wan Kun


Generally, distributed jobs have two stages of committing files: committing task's output files and committing job's output files.
If one attempt fails, another attempt will try to run the task again, after all tasks succeed, the job will commit the output of all tasks.
But now if we run a dynamic partition overwrite job, for example, *INSERT OVERWRITE table dst partition(part) SELECT * from src*, then if one of the final stage tasks fails, the job will fail.
The first taskattempt datawriter in final stage writes the output data directly to spark stage directory.If the first taskattempt fails, the second taskattempt datawriter will fail to setup, because the task's output file is already exists. Then the job will fail.
Therefore, I think we should write the temporary data to the taskattempt's work directory, and commit it's result files after the taskattempt succeed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org