You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tigerquoll <gi...@git.apache.org> on 2015/03/29 13:33:27 UTC

[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

GitHub user tigerquoll opened a pull request:

    https://github.com/apache/spark/pull/5250

    [CORE] [SPARK-6593] Provide option for HadoopRDD to skip bad data splits

    When reading a large amount of files from HDFS eg. with sc.textFile("hdfs:///user/cloudera/logs*.gz"). If a single split is corrupted then the entire job is canceled. As default behaviour this is probably for the best, but it would be nice in some circumstances where you know it will be ok to have the option to skip the corrupted portion and continue the job.
    
    Ideally I'd like to be able to report the list of bad files directly back to the master, but that would involve a public API change and is a discussion for later.  For now, I propose a new option of 'spark.hadoop.ignoreInputErrors', which defaults to false to maintain the existing behaviour of terminating the job upon any exception from the hadoop input libraries.  when set to true however, this option will simply log the error to the executor's log, skip the input-split that caused the error and continue processing.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tigerquoll/spark SPARK-6593a

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/5250.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5250
    
----
commit 37d2cbde5b67c94cac26e2c55030e9bb00678d7a
Author: Dale <ti...@outlook.com>
Date:   2015-03-29T11:23:01Z

    [SPARK-6593] Added spark.hadoop.ignoreInputEerrors config option and implementation

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by tigerquoll <gi...@git.apache.org>.
Github user tigerquoll commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87825625
  
    I'd Also like to point out the converse of my second paragraph above - we are also giving a means of converting deterministic application crashing to deterministic data.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87795029
  
    I was thinking about a few use cases, e.g. textFile, and all the operations in DataFrames.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by tigerquoll <gi...@git.apache.org>.
Github user tigerquoll commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87650287
  
    Hi Sean, 
    Thanks for your input - your views have helped me refine my thinking on the matter.  I believe that If you take a purist's point of view then yes you can say the source of the problem (likely) is with the data producer and should be fixed at the data producer's end. 
    
    The point being is that this is a problem that is affecting many spark users right now, and many users are not in control of the source system of the data they are analysing and are forced to 'make do' with what they have.  You call this solution a band-aid - but many ETL solutions are a bandaid - but providing this functionality is useful and serves a purpose for the end-user.
    
    Are you concerned that swallowing an exception could leave the hadoop input libraries in an inconsistent state, causing more data corruption?  This will not happen because swallowing the exception triggers the immediate finish of the file reading task and no more data will be read by the task.
    
    Are you concerned that swallowing an exception indicates that something has potentially gone wrong earlier in the hadoop input read, and that previous data could have been corrupted?  The user already knows this is potentially the case because running the application without this option enabled has caused the application to terminate in the first place.
    
    The fact that we are being more permissive of potentially corrupt data is a show stopper for this being default behaviour - but I'm not proposing this be default behaviour, I'm proposing this be a last-ditch option that an advanced user can knowingly enable when attempting to deal with corrupted data, with the understanding that their data could be made worse, but most likely corrupt data will be omitted. 
    
    The alternative is to tell them that their data is not suitable for being loaded into spark and perhaps they should use another tool or tell the data system owner to fix their data feeds and get back to them with another data set some time in the future.  I know which option I would prefer if given the choice - don't let perfect be the enemy of good.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87653690
  
    I don't know if corrupted gzip files are such a common problem, but I'm not sure that would change the logic about where to fix things. It is a problem with the preceding ETL process, yes. Something else needs to explicitly check and/or fix the input first if this is a problem.
    
    I suppose my point too is that this change does not just address the proposed problem with gzip files. It treats any error as recoverable.
    
    It's nothing to do with inconsistent state. It's the presenting a successful result that is actually silently missing input, which might not even be deterministic. This seems way more problematic than reliably failing-fast and, yes, making you fix your upstream process.
    
    Hiding behind a flag only goes so far. It's documented (or else how many people does it help?). It becomes a code path that has to be supported for a long time. It is presented to users as a fine thing to do when I don't believe it is. It's not the good being the enemy of the perfect, but the dangerous being the enemy of the good.
    
    This is nothing to do with telling people they can't use Spark, or have to fix an unfixable upstream process. This is about appropriately dealing with bad upstream data in the right place, and this is not how to do it.
    
    Specifically: why not write a process that just opens a stream on each input file in turn and tries to read a handful of bytes? if it fails, delete the file or do what you like with it. This is maybe 10 lines of code in your driver.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87791639
  
    My opinion is that this is the kind of thing that should go into a custom `InputFormat`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by tigerquoll <gi...@git.apache.org>.
Github user tigerquoll commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5250#discussion_r27377569
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
    @@ -246,6 +249,15 @@ class HadoopRDD[K, V](
             } catch {
               case eof: EOFException =>
                 finished = true
    +          case e: Exception =>
    --- End diff --
    
    Having been on the receiving end of things I know that the gzip module throws an IOException, but unfortunately I have no knowledge over what the Hadoop input modules and what exceptions they throw, or if they propagate exceptions up from other 3rd party libraries.   Catching such a broad exception is mitigated by the fact that this particular option defaults to off, and should only be enabled when you are trying to parse files that you know are corrupt.  Given the situation, when you turn the option on, we should really try to finish processing files to the best of our ability, thus I think in this case catching 'Exception' might be appropriate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by tigerquoll <gi...@git.apache.org>.
Github user tigerquoll commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87681300
  
    If a user can write scala codes that appropriately deals with the problem, why can't they write spark code to deal with it in parallel? Isn't this what spark is about? Isn't this a problem that can be readily parallelised? Spark is being put forward as data processing framework - bad data needs to be handled in some way better then just refusing to have anything to do with it.
    
    I believe to parallelise your mentioned solution means adding to the public API, which takes time and consideration.  The option was considered as a scoped, quick fix solution to at least give users some ability to continue - the idea would be to retire the option once a new API was in place to gracefully deal with the problem.
    
    In regards to the option being "presented to the users as a fine thing to do when I don't believe it is" - how about providing the information to the user a letting the users chose themselves? A good point about an option being a public API though - what is the understanding about how stable options are? No real Experimental or DeveloperAPI tags available here.
    
    Your proposed solution was the same solution I ended up settling on when first confronted with the issue - but only after a number of frustrated attempts at getting spark to do what I wanted it to.  What you proposed and what I did In the end was to give up using spark and to bashing out some standalone code using hadoop libraries to do the job.  ie: Stopped using spark and used another tool that made my job easier.  I felt that it didn't have to be this way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by tigerquoll <gi...@git.apache.org>.
Github user tigerquoll commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87822606
  
    To sum up the debate so far there doesn't appear to be any major concerns about leaving the system in an inconsistent state.  The major concern seems to be about swallowing of the exception and not letting any sign of it propagate back up to the main flow of execution, with the additional potential risk that the fact we've done so could lead to non-deterministic results.
    
    By swallowing non-deterministic exceptions, we are explicitly converting non-deterministic application crashing to potentially non-deterministic data - no getting around this fact.
    
    I'd like to achieve an outcome here, and I'm not really wedded to the means that outcome is achieved. As a counter-proposal, how about a new HadoopRDD derivative that works with Try((K,V))?  We propagate the exception cleanly back to calling code and allow them to deal with it explicitly.  it is slightly more cumbersome to use, but you'd only use it when you are expecting corrupted data back, and this allows an easy manipulation of data that contains potential exceptions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87792254
  
    But in the case of compression and stuff, most users don't ever go into that low level of details?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87688106
  
    You can use Spark to do this too, sure. Functions can call the HDFS API to check and delete files in parallel. Roughly:
    
    ```
    sc.parallelize(fs.listStatus(...).map(_.getPath.toString)).map { pathStr => 
      val path = new Path(pathStr)
      val in = new GZIPInputStream(fs.open(path))
      try {
        in.read()
      } catch {
        case e: ZipException => fs.delete(path, false)
      } finally {
        in.close()
      }
    }
    ```
    
    I'm sure that's not 100% right but you see the idea. 
    
    I am not proposing that this become a Spark API. It seems like an application-specific piece of logic that can be written using Spark. I don't claim Scala + Spark + Hadoop is easy, but it is directly doable with these tools.
    
    I think the point stands that this change does not help solve the problem directly, as the above does. It ignores the problem, which is sometimes a fine strategy, but at the cost of significant side effects. The side effects are the non-starter, to me. But the upside is I think there is a direct solution too.
    
    Well I've said enough so it's time to let others weigh in too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87399869
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5250#discussion_r27366354
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
    @@ -246,6 +249,15 @@ class HadoopRDD[K, V](
             } catch {
               case eof: EOFException =>
                 finished = true
    +          case e: Exception =>
    --- End diff --
    
    will the corrupt files throw something more specific than general Exception? seems a bit too wide of a net to cast here.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87793634
  
    @rxin can you explain that a little more?  I was envisioning an `InputFormat` that wrapped other `InputFormat`s and ignored exceptions, not something that would require users to replicate the code that converts bytes into records. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5250#discussion_r27378646
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
    @@ -246,6 +249,15 @@ class HadoopRDD[K, V](
             } catch {
               case eof: EOFException =>
                 finished = true
    +          case e: Exception =>
    --- End diff --
    
    Yes, but it calls into question when you would turn it on. You can't actually handle _just_ the situation you describe reliably even with `IOException`. I think this is band-aiding over an input problem that just isn't properly handled two more steps down the pipeline.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [CORE] [SPARK-6593] Provide option for HadoopR...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/5250#issuecomment-87788812
  
    I'm fairly torn about this. I can see why it'd be much easier for users with the introduction of a new config since this could be a common problem, but at the same time as I said earlier, this is a much wider net that would catch any exceptions -- e.g. maybe the data node was just down for a second, and then suddenly Spark will return incorrect result.
    
    Let's leave the discussion open for others to chime in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org