You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/03/29 13:14:52 UTC
[jira] [Commented] (SPARK-6593) Provide option for HadoopRDD to
skip bad data splits.
[ https://issues.apache.org/jira/browse/SPARK-6593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14385706#comment-14385706 ]
Sean Owen commented on SPARK-6593:
----------------------------------
At this level though, what's a bad split? a line of text that doesn't parse as expected? that's application-level logic. Given how little the framework knows, this would amount to ignoring a partition if there was any error in computing it, which seems too coarse to encourage people to use. You can of course handle this in the application logic -- catch the error, return nothing, log it, add to a counter, etc.
> Provide option for HadoopRDD to skip bad data splits.
> -----------------------------------------------------
>
> Key: SPARK-6593
> URL: https://issues.apache.org/jira/browse/SPARK-6593
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.3.0
> Reporter: Dale Richardson
> Priority: Minor
>
> When reading a large amount of files from HDFS eg. with sc.textFile("hdfs:///user/cloudera/logs*.gz"). If a single split is corrupted then the entire job is canceled. As default behaviour this is probably for the best, but it would be nice in some circumstances where you know it will be ok to have the option to skip the corrupted portion and continue the job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org