You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:33:34 UTC

[jira] [Resolved] (SPARK-17858) Provide option for Spark SQL to skip corrupt files

     [ https://issues.apache.org/jira/browse/SPARK-17858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-17858.
----------------------------------
    Resolution: Incomplete

> Provide option for Spark SQL to skip corrupt files
> --------------------------------------------------
>
>                 Key: SPARK-17858
>                 URL: https://issues.apache.org/jira/browse/SPARK-17858
>             Project: Spark
>          Issue Type: Improvement
>            Reporter: Shixiong Zhu
>            Priority: Major
>              Labels: bulk-closed
>
> In Spark 2.0, corrupt files will fail a SQL query. However, the user may just want to skip corrupt files and still run the query.
> Another painful thing is the current exception doesn't contain the paths of corrupt files, makes the user hard to fix their files. It's better to include the paths in the error message.
> Note: In Spark 1.6, Spark SQL always skip corrupt files because of SPARK-17850.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org