[jira] [Updated] (SPARK-11328) Provide more informative error message when direct parquet output committer is used and there is a file already exists error.

["Yin Huai (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-11328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yin Huai updated SPARK-11328:
-----------------------------
    Summary: Provide more informative error message when direct parquet output committer is used and there is a file already exists error.  (was: Correctly propagate error message in the case of failures when writing parquet)

> Provide more informative error message when direct parquet output committer is used and there is a file already exists error.
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-11328
>                 URL: https://issues.apache.org/jira/browse/SPARK-11328
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Yin Huai
>            Assignee: Nong Li
>            Priority: Critical
>             Fix For: 1.5.3, 1.6.0
>
>
> When saving data to S3 (e.g. saving to parquet), if there is an error during the query execution, the partial file generated by the failed task will be uploaded to S3 and the retries of this task will throw file already exist error. It is very confusing to users because they may think that file already exist error is the error causing the job failure. They can only find the real error in the spark ui (in the stage page).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org