You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by rdblue <gi...@git.apache.org> on 2016/02/17 22:18:06 UTC

[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

GitHub user rdblue opened a pull request:

    https://github.com/apache/spark/pull/11242

    SPARK-9926: Parallelize partition logic in UnionRDD.

    This patch has the new logic from #8512 that uses a parallel collection to compute partitions in UnionRDD. The rest of #8512 added an alternative code path for calculating splits in S3, but that isn't necessary to get the same speedup. The underlying problem wasn't that bulk listing wasn't used, it was that an extra FileStatus was retrieved for each file. The fix was just committed as [HADOOP-12810](https://issues.apache.org/jira/browse/HADOOP-12810). (I think the original commit also used a single prefix to enumerate all paths, but that isn't always helpful and it was removed in later versions so there is no need for SparkS3Utils.)
    
    I tested this using the same table that @piapiaozhexiu was using. Calculating splits for a 10-day period took 25 seconds with this change and HADOOP-12810, which is on par with the results from #8512.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/rdblue/spark SPARK-9926-parallelize-union-rdd

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11242.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11242
    
----
commit 535128ccb96ec22b95a286ae3f736abfa9ab8002
Author: Cheolsoo Park <ch...@netflix.com>
Date:   2015-08-12T20:30:24Z

    SPARK-9926: Parallelize partition logic in UnionRDD.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217230427
  
    **[Test build #57907 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57907/consoleFull)** for PR 11242 at commit [`002353a`](https://github.com/apache/spark/commit/002353af871c8f4fcddbbd704f4356bf5aa7c598).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217263203
  
    **[Test build #57907 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57907/consoleFull)** for PR 11242 at commit [`002353a`](https://github.com/apache/spark/commit/002353af871c8f4fcddbbd704f4356bf5aa7c598).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r62215656
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -56,14 +58,30 @@ private[spark] class UnionPartition[T: ClassTag](
       }
     }
     
    +object UnionRDD {
    +  private lazy val partitionEvalThreadPool = new ForkJoinPool(8)
    +}
    +
     @DeveloperApi
     class UnionRDD[T: ClassTag](
         sc: SparkContext,
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    --- End diff --
    
    I updated this to `shouldListPartitionsInParallel`. Starting with "read" makes it sound like it should do a read action. And using "list" matches the config property.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-197591766
  
    @JoshRosen, I've added the test. Sorry for taking so long, my e-mail notifications weren't getting through for a little while.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r58977765
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    Could you document this configuration somewhere?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-198034848
  
    @dbtsai, I updated the PR with your suggestion and rebased on master. Thanks for taking the time to review!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217252564
  
    **[Test build #57913 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57913/consoleFull)** for PR 11242 at commit [`9b183cb`](https://github.com/apache/spark/commit/9b183cb80496feb9da34d85e1a0bc426a4f861a7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56918938
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    If I remember correctly, no one recommends reusing task VMs for MR. Too many things assumed that they weren't reused, which is a major advantage for Spark. Also, mapper VMs aren't normally calling getSplits in MR. Split planning happens on the driver/job client, which may stick around submit multiple jobs but usually not in parallel. Caching data used for split planning would typically make a difference for iterative use cases.
    
    This commit does change the default to parallel split planning, but I'm arguing that it's a good idea to have the safety valve, not that this is a significant enough risk that we should make parallel split planning opt-in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-216710971
  
    @rdblue Looks great! Much simpler than earlier patches. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217248027
  
    **[Test build #57911 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57911/consoleFull)** for PR 11242 at commit [`9b183cb`](https://github.com/apache/spark/commit/9b183cb80496feb9da34d85e1a0bc426a4f861a7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-207184221
  
    LGTM. This PR dramatically improves our s3 performance at Netflix. 
    
    @andrewor14 @srowen @JoshRosen @davies @marmbrus @yhuai, any further feedback? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59058588
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    I think it was, or at least, that's what we were discussing at https://github.com/apache/spark/pull/11242#discussion_r56891305 Not sure what the original in this PR was because it was force-pushed. In any event, I personally agree that a minimum count has a little more motivation, even if I still don't see when this setting would be used. Where do we disagree ... are you saying that a problematic `InputFormat` of this form would actually work everywhere else in Spark?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-211700015
  
    **[Test build #56161 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56161/consoleFull)** for PR 11242 at commit [`96b188d`](https://github.com/apache/spark/commit/96b188df03c9c10ee73270943ac47cb6aa36125c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217283166
  
    **[Test build #57913 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57913/consoleFull)** for PR 11242 at commit [`9b183cb`](https://github.com/apache/spark/commit/9b183cb80496feb9da34d85e1a0bc426a4f861a7).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56892490
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    Such an implementation would break all over in, say, MapReduce. Can it be likely, like is there a known instance of this? I must say that fixing and redeploying the code strikes me as a great recommendation, since it is going to break a bunch of other places in Spark that actually read data and maybe rely on the correctness of this shared state, and where a failure could be silent corruption. Making a flag signals that it's a valid design choice.
    
    I wouldn't normally object to a hidden undocumented flag as a safety valve but I can't see a case where it's safe to recommend?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56432618
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,21 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private lazy val evaluatePartitions: Unit = {
    +    val threshold = conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +    if (threshold > 0 && rdds.length > threshold) {
    +      val parArray = rdds.toParArray
    +      parArray.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(threshold))
    +      parArray.foreach(_.partitions)
    +    } else {
    --- End diff --
    
    `false` and remove `rdds.foreach(_.partitions)` to make it evaluate in the original path. Thus, you can check if we go through this path in unit test.
    
    Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59053387
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) rdds.par else rdds
    --- End diff --
    
    No the problem is that this could be blocked entirely, if some random user writes some code to block the global threadpool, or even a third party library that does that. Spark should not use the global thread pool anywhere.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217263232
  
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/11242


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217237871
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57902/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-211700178
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56161/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56878058
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    I'm reluctant to make `rdds.zipWithIndex` parallel by default with no way to disable it. It's a common case for RDDs in the union to wrap a Hadoop InputFormat. I don't think it's normally an expectation that getSplits could be called concurrently and custom InputFormats are fairly common. I know Parquet/Hive had a bug where Parquet's InputFormat was caching results, so I think users should be able to disable this to avoid similar issues.
    
    How about:
    1. Separate the flag from the partition eval method so it is clear that the flag indicates whether partitions are calculated in parallel (for testing)
    2. Use the default thread pool instead of custom parallelism
    
    Would that work?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-198572978
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217263457
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57907/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r62228103
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +64,21 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionListingParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
    +  private lazy val partitionEvalTaskSupport = new ForkJoinTaskSupport(new ForkJoinPool(8))
    --- End diff --
    
    Fixed. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56885555
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    You're right that there are several instances of InputFormat and those aren't shared across threads. The problem is that Hadoop uses reflection to get InputFormat instances, which means the way for an InputFormat to cache intermediate results is to use a static cache available across instances. I don't think this is a huge risk, but I think it warrants being able to disable UnionRDD concurrency.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r57203120
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    I think there's a big difference between the two scenarios. While we may have lurking concurrency bugs in getSplits calls today, this patch kicks off a bunch of tasks that call getSplits at the same time. That's much more likely to hit concurrency problems if they exist and I don't think we can conclude that it is safe from your example, where a driver could be doing split calculations in multiple threads.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r62237486
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +64,22 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionListingParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
    +  @transient private lazy val partitionEvalTaskSupport =
    +      new ForkJoinTaskSupport(new ForkJoinPool(8))
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionListingParallel) {
    +      val parArray = rdds.par
    +      parArray.tasksupport = partitionEvalTaskSupport
    +      parArray
    +    } else {
    +      rdds
    +    }
    +    val array = new Array[Partition](parRDDs.map(_.partitions.length).sum)
    --- End diff --
    
    I think the problem is this line: after you call a map Scala will use the original global execution context for the sum. But here it's just a list of numbers so we can sum that ourselves synchronously, so it'll work if you do
    ```
    parRDDs.map(_.partitions.length).seq.sum
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r61978567
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -56,14 +58,30 @@ private[spark] class UnionPartition[T: ClassTag](
       }
     }
     
    +object UnionRDD {
    --- End diff --
    
    please make this private


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-207062268
  
    I've updated this PR and rebased on the current master. As Sean suggested, I used the default thread pool and a parallel collection to do the work, so the commit is considerably smaller and uses the default thread pool. I left the property in as a safety valve, along with a val to check during tests from DB's review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59058248
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) rdds.par else rdds
    --- End diff --
    
    Right, I suppose I'm saying much more than just this fails in that case. I support your suggestion. The original version always made a brand-new pool on each call; we should avoid that at least, and also not keep a bunch of threads around when not used if that's possible in ForkJoinSupport.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59061019
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    One JVM can definitely have many instances of one `InputFormat` and get partitions from them at the same time. If this is where we disagree, let's talk about that.
    
    You're arguing that it becomes more likely to trigger with this change -- yes. That argues against making this change at all with the default to 'on'. If this is likely to be a real problem, then we're opting in to possibly subtle correctness bugs by default. A safety valve only helps if someone even knows there's a problem and knows this is the source, which is unlikely for most callers.
    
    I'm further asserting that even if this behavior can be disabled, this is a bug that's going to be triggered by other Spark usage already. Current usage is already going to cause this type of problem; it's not even that rare. Although I can't be sure we aren't seeing bugs due to this, I can't think of any evidence of this in the wild.
    
    Examples are good, but, these aren't actual bugs at this point right? they're instances where the format implementation had a bug (at least, made a much too aggressive assumption) and was fixed. That's the right outcome, since it's the only thing that actually fixes the problem.
    
    I myself will not merge this change for these reasons. It's hard to feel too too strongly about an undocumented flag, so I would not veto it if someone else wants to merge it. My concern is that it does a little harm: this may help someone think they've avoided a particular type of problem when they have not, in Spark.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-211629761
  
    **[Test build #56155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56155/consoleFull)** for PR 11242 at commit [`4fc9be4`](https://github.com/apache/spark/commit/4fc9be4630d45a80b5662605371dde3e1614a3b7).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r57210970
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    I'm all for parallelizing the loop, just not commiting this without the safety valve.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56746051
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    This took me a minute to understand. It's a boolean val but really that's a dummy. This is evaluated only for side effects. It's only evaluated one place the partitions are used and not the other. If it's only really helpful in `getPartitions`, how about just making the `rdds.zipWithIndex` parallel and let it process that way? no dummy lazy vals.
    
    I also think a custom parallelism and fork join pool are overkill. Just use the default pool. I'd like to avoid yet more knobs to tune unless it's strongly needed.
    
    Is there any downside to parallel execution? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56968395
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    Reuse wouldn't cause concurrent access to an `InputFormat` instance by itself. I thought the issue here was use for reading/writing data rather than `getSplits`. I could be wrong that MR would have two map tasks reading/writing in one JVM at the same time, but in any event, it's actually on Spark that matters here. And RDDs' `InputFormat` can definitely be accessed concurrently in Spark, and map tasks from the same RDD can run in the same JVM.
    
    I guess the point is: _if_ this is a problem, this doesn't resolve it. More immediately, there are cases where the same _instance_ of `InputFormat` is accessed concurrently. I don't think we've heard of any problems to date though?
    
    I can't see working around this kind of implementation problem. It's not just that the implementation assumes no concurrent access to an instance, but it would have to assume no concurrent access to any two instances at the same time. There are a hundred lesser sins I could see working around before this.
    
    Is there a specific known issue this is working around? I'm not following why this would ever be the right thing to do. Turning off this flag still leaves you with more serious silent errors elsewhere. I think this can be as simple as parallelizing the outer loop with no additional problems.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-197086147
  
    **[Test build #53234 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53234/consoleFull)** for PR 11242 at commit [`535128c`](https://github.com/apache/spark/commit/535128ccb96ec22b95a286ae3f736abfa9ab8002).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56891305
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    Then I think have `spark.rdd.enableParallelListing` default to `true`, and use the default thread pool is a good idea. This requires the manual tuning compared with  `spark.rdd.parallelListingThreshold`, and it's flexible enough to disable it. My 0.02 cents. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217288532
  
    Thank you @andrewor14 and all the reviewers!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217237543
  
    **[Test build #57902 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57902/consoleFull)** for PR 11242 at commit [`2e90c5a`](https://github.com/apache/spark/commit/2e90c5a8423970cde7d4ca432e07c38aa285fc2a).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r57014360
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    > I thought the issue here was use for reading/writing data rather than getSplits
    
    The issue is introducing concurrent calls to getSplits. [PARQUET-16|https://issues.apache.org/jira/browse/PARQUET-16] is a suggestion to cache the results of file system operations during split planning, which would have to be done by a cache shared across instances. (Instead, we fixed the underlying problem with HADOOP-12810.)
    
    > if this is a problem, this doesn't resolve it. More immediately, there are cases where the same instance of InputFormat is accessed concurrently.
    
    If this is true, then your line of reasoning makes sense. Where are InputFormat instances accessed concurrently?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56890135
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    I agree that it's a flawed implementation, but it's reasonable to say they exist without causing problems today because split planning isn't done in parallel. I think your point about how someone would find this setting is valid, but I'm more concerned that someone that hits the problem should have a way around it. Without a way to turn this off, a user would be forced to either roll back their Spark version or fix and redeploy their InputFormat code. Both are expensive compared to a config option.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-211652005
  
    **[Test build #56161 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56161/consoleFull)** for PR 11242 at commit [`96b188d`](https://github.com/apache/spark/commit/96b188df03c9c10ee73270943ac47cb6aa36125c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r61978625
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -56,14 +58,30 @@ private[spark] class UnionPartition[T: ClassTag](
       }
     }
     
    +object UnionRDD {
    +  private lazy val partitionEvalThreadPool = new ForkJoinPool(8)
    +}
    +
     @DeveloperApi
     class UnionRDD[T: ClassTag](
         sc: SparkContext,
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    --- End diff --
    
    can you at least add a comment on what this means?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r61978707
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -56,14 +58,30 @@ private[spark] class UnionPartition[T: ClassTag](
       }
     }
     
    +object UnionRDD {
    +  private lazy val partitionEvalThreadPool = new ForkJoinPool(8)
    +}
    +
     @DeveloperApi
     class UnionRDD[T: ClassTag](
         sc: SparkContext,
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    --- End diff --
    
    also should this be better called `readPartitionsInParallel` or something?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59010483
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) rdds.par else rdds
    --- End diff --
    
    Yeah we had this discussion above. The downside is that you make a pool of threads that sits around and/or has to spin up just for this block. I agree with the potential problem, though lots of things fall apart if user code jams the shared thread pool. Since this isn't that resource or time sensitive, seemed good to do the simple thing here, but I don't feel strongly about it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r62238592
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +64,22 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionListingParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
    +  @transient private lazy val partitionEvalTaskSupport =
    +      new ForkJoinTaskSupport(new ForkJoinPool(8))
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionListingParallel) {
    +      val parArray = rdds.par
    +      parArray.tasksupport = partitionEvalTaskSupport
    +      parArray
    +    } else {
    +      rdds
    +    }
    +    val array = new Array[Partition](parRDDs.map(_.partitions.length).sum)
    --- End diff --
    
    I just pushed a fix with what you suggested. It's really strange to me that the thread pool changes. Seems like the new parallel collection should use its parent's task support. Good catch, and thanks for reviewing!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-211629524
  
    **[Test build #56155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56155/consoleFull)** for PR 11242 at commit [`4fc9be4`](https://github.com/apache/spark/commit/4fc9be4630d45a80b5662605371dde3e1614a3b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-207141392
  
    **[Test build #55278 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55278/consoleFull)** for PR 11242 at commit [`1633bd6`](https://github.com/apache/spark/commit/1633bd6e29bc75ed5978586ce65ae03b30941247).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by piaozhexiu <gi...@git.apache.org>.
Github user piaozhexiu commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-186731750
  
    @rdblue thanks for pick up this. LGTM of course!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56882466
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    One `HadoopRDD` has one `InputFormat` to generate its splits/partitions, and one `UnionRDD` has many `HadoopRDD`s here. If each child `HadoopRDD`'s partitions are generated in parallel, individually they still compute their partitions serially as before. I think its access to its single instance of `InputFormat` doesn't become (more) concurrent. And in `getPartitions` it's only being used to compute splits rather than read data.
    
    Either that's good news and there's no problem of this form or I have missed a point here? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217251208
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r62215696
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -56,14 +58,30 @@ private[spark] class UnionPartition[T: ClassTag](
       }
     }
     
    +object UnionRDD {
    +  private lazy val partitionEvalThreadPool = new ForkJoinPool(8)
    +}
    +
     @DeveloperApi
     class UnionRDD[T: ClassTag](
         sc: SparkContext,
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) {
    +      val parArray = rdds.par
    +      parArray.tasksupport = new ForkJoinTaskSupport(UnionRDD.partitionEvalThreadPool)
    +      parArray
    +    } else {
    +      rdds
    +    }
    +    // evaluating parRDDs.map(_.partitions) will execute in parallel and cache partitions
    --- End diff --
    
    Removed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56432610
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,21 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private lazy val evaluatePartitions: Unit = {
    --- End diff --
    
    Why not 
    
    ```
    private[spark] val isParallellEvaluatePartitions: Boolean = {
    ...
    ...
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217251801
  
    Jenkins retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56747153
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    +    val threshold = conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +    if (threshold > 0 && rdds.length > threshold) {
    +      val parArray = rdds.toParArray
    +      parArray.tasksupport = new ForkJoinTaskSupport(
    +        new ForkJoinPool(math.min(threshold, 8)))
    --- End diff --
    
    Sounds fair. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59059521
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) rdds.par else rdds
    --- End diff --
    
    @rxin, does Spark have a default thread-pool for situations like this? I tend to agree with Sean's reasoning that we should use an existing thread-pool over the risk of contention with user code. Both concerns could be solved by having a pool for these one-off internal tasks.
    
    Wouldn't user or third-party code blocking like you suggest slow down or stop the application anyway?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59048405
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    Is it possible that listing in some data source is not thread safe?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56880928
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    The use case I'm interested in is when we load a S3-backed table from Hive. Each partition is one of the RDDs in this union and the partition's splits are planned when getPartitions is called. (We also tracked down [HADOOP-12810](https://issues.apache.org/jira/browse/HADOOP-12810) as part of this work.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-207181096
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55278/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-211629764
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r57021251
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    Yeah I was referring to the side point about caching, but I likely misunderstood. Yes getSplits is the issue. HadoopRDD has an InputFormat and it is possible the driver calls getPartitions on it simultaneously because two actions could be invoked in parallel. This is the basic scenario I am thinking of though it has a bunch of variants. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-197086307
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53234/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59062676
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    Concurrency bugs are hard to trigger and I disagree that other Spark usage would have exposed them already. I think this is the main point where we disagree.
    
    Assuming that the user has tracked the problem down, deploying the fix may be the hardest part. Many customers can't simply make changes and push a new version without sending code through QA or some other validation. A runtime config change is much easier.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r57203700
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    Agreed. The root cause would still be a badly flawed custom implementation, and the right fix to fix that, and I don't see any evidence that this exists in the wild. Do you have a specific example in mind?
    
    Otherwise I personally would not merge this change for this reason. I would support merging the simpler change to parallelize the loop.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-198535147
  
    Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r58932013
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    Hey Sean, sorry I hadn't gotten back to this for a couple of weeks. I've updated the PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-197086304
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r61979292
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -56,14 +58,30 @@ private[spark] class UnionPartition[T: ClassTag](
       }
     }
     
    +object UnionRDD {
    --- End diff --
    
    actually, it's best to avoid having a singleton fork join pool here. Just move it into `getPartitions`. It's unlikely that we'll need to share these pools across `UnionRDD`s; each RDD probably has enough partitions to saturate the pool.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217283422
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r61979094
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    I agree with @srowen. Let's keep this undocumented because
    (1) IDK how many people would actually want to set this (probably not many)
    (2) We would need to provide backward compatibility in the future, which makes the code harder to support
    
    The reason why it's even a config is only for safety. In the unlikely event that this somehow screws things up badly we can always tell the user to just set this to 1 to revert to the original behvaior. Normally, however, we wouldn't recommend the user to even touch this at all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-211629766
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/56155/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-197052153
  
    Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56727449
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    +    val threshold = conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +    if (threshold > 0 && rdds.length > threshold) {
    +      val parArray = rdds.toParArray
    +      parArray.tasksupport = new ForkJoinTaskSupport(
    +        new ForkJoinPool(math.min(threshold, 8)))
    --- End diff --
    
    The current code allows the user to turn parallelism off (thresh=0) or to control when parallel processing happens, but limits execution to a maximum of 8 threads. I think that's pretty reasonable.
    
    Your suggestion allows setting the number of threads, but not controlling when execution happens in parallel. But controlling the number of threads is not the important part. At a minimum, users should be able to keep the current behavior.
    
    If users need to set the number of threads, we can do that in a later patch when necessary. Otherwise, I think the current patch is the right way to go.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-197052654
  
    **[Test build #53234 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53234/consoleFull)** for PR 11242 at commit [`535128c`](https://github.com/apache/spark/commit/535128ccb96ec22b95a286ae3f736abfa9ab8002).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-198535461
  
    **[Test build #53562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53562/consoleFull)** for PR 11242 at commit [`79d618e`](https://github.com/apache/spark/commit/79d618efd299eda148acfd998f86a32c80d43929).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-211700176
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-185410593
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r62215672
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -56,14 +58,30 @@ private[spark] class UnionPartition[T: ClassTag](
       }
     }
     
    +object UnionRDD {
    --- End diff --
    
    Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r62225084
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +64,21 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionListingParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
    +  private lazy val partitionEvalTaskSupport = new ForkJoinTaskSupport(new ForkJoinPool(8))
    --- End diff --
    
    This needs to be transient


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r62260803
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +64,22 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionListingParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
    +  @transient private lazy val partitionEvalTaskSupport =
    +      new ForkJoinTaskSupport(new ForkJoinPool(8))
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionListingParallel) {
    +      val parArray = rdds.par
    +      parArray.tasksupport = partitionEvalTaskSupport
    +      parArray
    +    } else {
    +      rdds
    +    }
    +    val array = new Array[Partition](parRDDs.map(_.partitions.length).sum)
    --- End diff --
    
    I agree. It's a very confusing aspect of the scala `ParSeq` API.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217283424
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57913/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217212503
  
    **[Test build #57902 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57902/consoleFull)** for PR 11242 at commit [`2e90c5a`](https://github.com/apache/spark/commit/2e90c5a8423970cde7d4ca432e07c38aa285fc2a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59050309
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    Heh, yeah can you see the long collapsed conversation above? that was indeed the concern. Here, you would have to have source whose entire class was not thread-safe (i.e. you can't concurrently access two unrelated instances), which would be pretty broken. My contention is that such a source would break in other places in Spark, so trying to fix it here doesn't do anything. It also sounds like there's no known instance of this (?) so that's why I'm not sure if this is a meaningful option.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-207181095
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56719728
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    +    val threshold = conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +    if (threshold > 0 && rdds.length > threshold) {
    +      val parArray = rdds.toParArray
    +      parArray.tasksupport = new ForkJoinTaskSupport(
    +        new ForkJoinPool(math.min(threshold, 8)))
    --- End diff --
    
    In this case, it will always be 8 threads by default. How about 
    
    ```scala
      private[spark] lazy val parallelPartitionEval: Boolean = {
        val parallelism = conf.getInt("spark.rdd.listingParallelism", 16)
        val threshold = 8
        if (rdds.length > threshold) {
          val parArray = rdds.toParArray
          parArray.tasksupport = new ForkJoinTaskSupport(
            new ForkJoinPool(math.min(parallelism, rdds.length)))
          parArray.foreach(_.partitions)
          true
        } else {
          false
        }
      }
    ```
    
    Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59058645
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    I don't agree that such a source would necessarily break other places in Spark. There's a big difference between threads may happen to execute some method concurrently and kicking off a pool of threads at that method.
    
    While there is no known instance, this is based on two real-world cases:
    1. Parquet had a caching scheme that assumed no reuse at all, which broken when that assumption was violated by Hive.
    2. Parquet had a proposal to cache splits for Spark, which would require a static/class-level cache.
    
    InputFormats are one of the most common places to plug in bad code because they are used for old, custom formats. I'm fine with parallelizing this by default, but I'm not comfortable with the idea of changing how they are used  and increasing the chances of hitting a bug if one exists without a safety valve. I also don't see the down-side to having one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r61979221
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -56,14 +58,30 @@ private[spark] class UnionPartition[T: ClassTag](
       }
     }
     
    +object UnionRDD {
    +  private lazy val partitionEvalThreadPool = new ForkJoinPool(8)
    +}
    +
     @DeveloperApi
     class UnionRDD[T: ClassTag](
         sc: SparkContext,
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) {
    +      val parArray = rdds.par
    +      parArray.tasksupport = new ForkJoinTaskSupport(UnionRDD.partitionEvalThreadPool)
    +      parArray
    +    } else {
    +      rdds
    +    }
    +    // evaluating parRDDs.map(_.partitions) will execute in parallel and cache partitions
    --- End diff --
    
    I don't think this comment adds any value. First of all it's not necessarily true. Second if it's a parallel seq then that's just what Scala does, no surprises there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217237865
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59111476
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) rdds.par else rdds
    --- End diff --
    
    just create a shared pool that can be used here.
    
    no user code wouldn't necessarily slow down the operation. just imagine somebody running the following:
    
    ```scala
    (1 to 1000).par.foreach(_ => Thread.sleep(100000000))
    ```
    
    suddenly nobody can use union anymore.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-198572783
  
    **[Test build #53562 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53562/consoleFull)** for PR 11242 at commit [`79d618e`](https://github.com/apache/spark/commit/79d618efd299eda148acfd998f86a32c80d43929).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217251188
  
    **[Test build #57911 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57911/consoleFull)** for PR 11242 at commit [`9b183cb`](https://github.com/apache/spark/commit/9b183cb80496feb9da34d85e1a0bc426a4f861a7).
     * This patch **fails MiMa tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r60151772
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) rdds.par else rdds
    --- End diff --
    
    @rxin, I added a thread-pool for UnionRDDs to share. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-185481901
  
    Ping @piaozhexiu for review as well.
    
    Glad to see that this change is relatively small and straightforward. Quick question: is this case covered by any existing test? Do you mind adding an integration test which explicitly exercises this code path (e.g. by configuring the parallel listing threshold to be really low)? Just want to make sure that this path gets hit at least one in our test suite.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56893742
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    Why would this break in MR? A typical MR job will create one instance of the InputFormat and call getSplits. File listing within that can be done in parallel, but I'm not aware of other cases where frameworks like Hive or Spark call getSplits in parallel. If that's common outside of Spark, then you're right that this is broken elsewhere and we don't need the safety valve.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56894879
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    You can have multiple map tasks in one JVM right? they'd be executing simultaneously, using this shared state simultaneously. The issue is really Spark though, where that's definitely going to happen. Same for simply calling `getSplits`. Isn't your current PR already doing the same thing though since it also processes the RDDs in parallel?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56747209
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    I'm also curious to use the default thread pool, and I like the idea not adding extra knobs to tune. However, in the case, should we still have a hard coded default threshold since when the # of rdds are small, we don't want the overhead. 
    
    If we just make `rdds.zipWithIndex` parallel, it will be hard to test if this path is touched. What do you think?
    
    Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by steveloughran <gi...@git.apache.org>.
Github user steveloughran commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r60199645
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    Changing a config option is turning on a different codepath; in theory it should go through a QE process. At least with a config option, when it does play up, you can roll back by changing the switch and restarting: you've saved the cost of redeploying the old code.
    
    Undocumented options are dangerous though
    
    * nobody in support will know what to do if problems surface and the switch is set
    * someone is going to find it in the code and wonder "what does this do"... the origins will be lost in the folklore of the project.
    
    What about having a section in the docs on "unstable+unsupported options", list it with a link to the JIRA and a "use at your own risk" disclaimer?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56879711
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    Do we have a problem with concurrent access to an `InputFormat` instance here? I'd assume each partition might have an object representing a split but this change doesn't make that accessed any (more) concurrently than it is now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56432616
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,21 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private lazy val evaluatePartitions: Unit = {
    +    val threshold = conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +    if (threshold > 0 && rdds.length > threshold) {
    +      val parArray = rdds.toParArray
    +      parArray.tasksupport = new ForkJoinTaskSupport(new ForkJoinPool(threshold))
    +      parArray.foreach(_.partitions)
    --- End diff --
    
    `ture`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-207180917
  
    **[Test build #55278 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55278/consoleFull)** for PR 11242 at commit [`1633bd6`](https://github.com/apache/spark/commit/1633bd6e29bc75ed5978586ce65ae03b30941247).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r56886896
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,7 +64,23 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // Evaluate partitions in parallel. Partitions of each rdd will be cached by the `partitions`
    +  // val in `RDD`.
    +  private[spark] lazy val parallelPartitionEval: Boolean = {
    --- End diff --
    
    If an impl is doing that and it isn't thread-safe, surely that's a busted implementation? you can't expect that no two instances of the class in the same JVM would not be accessed concurrently (unless you design for it of course). At the least, no reading of data happens in this path anyway? I'm trying to avoid designing around this, since this is a flag that would be on by default, hidden, but it's not clear how someone would reasonably figure out this was the problem and this was the solution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r58991517
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    +
       override def getPartitions: Array[Partition] = {
    -    val array = new Array[Partition](rdds.map(_.partitions.length).sum)
    +    val parRDDs = if (isPartitionEvalParallel) rdds.par else rdds
    --- End diff --
    
    we should make sure we have a thread pool explicitly specified -- otherwise this might get blocked by some other random user code.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217251212
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57911/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-207140497
  
    add to whitelist


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by dbtsai <gi...@git.apache.org>.
Github user dbtsai commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-207063759
  
    Jenkins, test this please.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217245132
  
    By the way, I just tried this out in the REPL and it still blocks behind user code:
    ```
    import scala.concurrent.ExecutionContext.Implicits.global
    import scala.concurrent.Future
    Future { (1 to 1000).par.foreach(_ => Thread.sleep(100000000)) }
    sc.union((0 to 10).map { _ => sc.parallelize(1 to 1000) }).count() // blocks forever
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-198572980
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53562/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59057410
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    @srowen, I don't think `enableParallelListing` was the option. This always used `parallelListingThreshold`, from the [original version](https://github.com/apache/spark/pull/8512/files#diff-eaababfc87ea4949f97860e8b89b7586). I'm happy to change it to the boolean version if there's consensus around that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11242#discussion_r59010259
  
    --- Diff: core/src/main/scala/org/apache/spark/rdd/UnionRDD.scala ---
    @@ -62,8 +62,14 @@ class UnionRDD[T: ClassTag](
         var rdds: Seq[RDD[T]])
       extends RDD[T](sc, Nil) {  // Nil since we implement getDependencies
     
    +  // visible for testing
    +  private[spark] val isPartitionEvalParallel: Boolean =
    +    rdds.length > conf.getInt("spark.rdd.parallelListingThreshold", 10)
    --- End diff --
    
    @davies I think this is best left undocumented. But is this setting meaningful? This has more purpose than the previous `spark.rdd.enableParallelListing` config, which only let you disable it. I actually don't see a case where you wouldn't want parallel listing (see previous discussion)?  why would i want to list 9 serially but 10 all at once? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217288054
  
    Merging into master 2.0. Thanks for your work!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: SPARK-9926: Parallelize partition logic in Uni...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11242#issuecomment-217263452
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org