You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by 10110346 <gi...@git.apache.org> on 2018/10/15 11:44:15 UTC

[GitHub] spark pull request #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small ...

GitHub user 10110346 opened a pull request:

    https://github.com/apache/spark/pull/22725

    [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files via BinaryFileRDD

    ## What changes were proposed in this pull request?
    
    This is a follow up of #21601, `StreamFileInputFormat` and `WholeTextFileInputFormat` have the same problem.
    
    `Minimum split size pernode 5123456 cannot be larger than maximum split size 4194304
    java.io.IOException: Minimum split size pernode 5123456 cannot be larger than maximum split size 4194304
            at org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat.getSplits(CombineFileInputFormat.java: 201)
    	at org.apache.spark.rdd.BinaryFileRDD.getPartitions(BinaryFileRDD.scala:52)
    	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:254)
    	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
    	at scala.Option.getOrElse(Option.scala:121)
    	at org.apache.spark.rdd.RDD.partitions(RDD.scala:252)
    	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2138)`
    
    ## How was this patch tested?
    Added a unit test


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/10110346/spark maxSplitSize_node_rack

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22725.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22725
    
----
commit 54ffcdb7a18471a7a24fe36a000ca0cc4e8d0eba
Author: liuxian <li...@...>
Date:   2018-10-15T07:28:31Z

    fix

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-25753][CORE]fix reading small files via BinaryFil...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    **[Test build #97874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97874/testReport)** for PR 22725 at commit [`54ffcdb`](https://github.com/apache/spark/commit/54ffcdb7a18471a7a24fe36a000ca0cc4e8d0eba).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-25753][CORE]fix reading small files via BinaryFil...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    **[Test build #97854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97854/testReport)** for PR 22725 at commit [`54ffcdb`](https://github.com/apache/spark/commit/54ffcdb7a18471a7a24fe36a000ca0cc4e8d0eba).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22725: [SPARK-25753][CORE]fix reading small files via Bi...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/22725


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    SPARK-24610 is the original issue, please file a new jira for StreamFileInputFormat


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-25753][[CORE]fix reading small files via BinaryFi...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    It still has `[[` before `CORE`. :)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3980/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-25753][[CORE][FOLLOW-UP]fix reading small files v...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    +1 Looks good, thanks @10110346 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-25753][[CORE][FOLLOW-UP]fix reading small files v...

Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    ok,thanks @dongjoon-hyun 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-25753][CORE]fix reading small files via BinaryFil...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    Build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    cc @dhruve @tgravescs


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by 10110346 <gi...@git.apache.org>.
Github user 10110346 commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    @tgravescs ok, I will do it ,thanks


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-25753][CORE]fix reading small files via BinaryFil...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    **[Test build #97862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97862/testReport)** for PR 22725 at commit [`54ffcdb`](https://github.com/apache/spark/commit/54ffcdb7a18471a7a24fe36a000ca0cc4e8d0eba).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    **[Test build #97378 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97378/testReport)** for PR 22725 at commit [`54ffcdb`](https://github.com/apache/spark/commit/54ffcdb7a18471a7a24fe36a000ca0cc4e8d0eba).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    **[Test build #97378 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97378/testReport)** for PR 22725 at commit [`54ffcdb`](https://github.com/apache/spark/commit/54ffcdb7a18471a7a24fe36a000ca0cc4e8d0eba).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-24610][[CORE][FOLLOW-UP]fix reading small files v...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/97378/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22725: [SPARK-25753][CORE]fix reading small files via BinaryFil...

Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:

    https://github.com/apache/spark/pull/22725
  
    merged to master


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org