You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by brennonyork <gi...@git.apache.org> on 2015/03/23 19:28:10 UTC

[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

GitHub user brennonyork opened a pull request:

    https://github.com/apache/spark/pull/5142

    [SPARK-4086][GraphX]: Fold-style aggregation for VertexRDD

    Adds five new methods into the `VertexRDD` suite to allow for fold-style calling conventions. Those methods are:
    
    * `leftZipJoinWithFold`
    * `leftJoinWithFold`
    * `innerZipJoinWithFold`
    * `innerJoinWithFold`
    * `aggregateUsingIndexWithFold`
    
    Each of above has a set of tests within the `VertexRDDSuite` to ensure proper functionality.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/brennonyork/spark SPARK-4086

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/5142.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5142
    
----
commit c2ef961e1168bb2de57bc4e12d118d9d5883345b
Author: Brennon York <br...@capitalone.com>
Date:   2015-03-18T21:32:48Z

    added leftJoin*WithFold

commit 639046c6fcc0f9f82f5b1fa6fc4092efa2a6ecff
Author: Brennon York <br...@capitalone.com>
Date:   2015-03-19T22:57:51Z

    added innerJoin with folds

commit 1229f9fa3ddcadc39a17d1af0146275208f4c34e
Author: Brennon York <br...@capitalone.com>
Date:   2015-03-23T18:07:19Z

    added aggregateUsingIndexWithFold

commit 98197e743cf71d954fd45a456a88a7ae2ff47888
Author: Brennon York <br...@capitalone.com>
Date:   2015-03-23T18:10:46Z

    updated test to better demonstrate the aggregate fold-style values correctly being passed in

commit 6a6136424ab2805148e141471fb2e22d37223d05
Author: Brennon York <br...@capitalone.com>
Date:   2015-03-23T18:25:03Z

    added proper docstrings

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85595804
  
      [Test build #29094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29094/consoleFull) for   PR 5142 at commit [`d519370`](https://github.com/apache/spark/commit/d519370c8b836a4a9edb8eab4e022bb976bb7d27).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85172240
  
      [Test build #29018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29018/consoleFull) for   PR 5142 at commit [`e661a8f`](https://github.com/apache/spark/commit/e661a8f3b146eef237777aa668b2c321fecdc8fc).
     * This patch **fails MiMa tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85653726
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29096/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85596003
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29094/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85135767
  
      [Test build #29016 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29016/consoleFull) for   PR 5142 at commit [`6a61364`](https://github.com/apache/spark/commit/6a6136424ab2805148e141471fb2e22d37223d05).
     * This patch **fails Scala style tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5142#discussion_r27005726
  
    --- Diff: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala ---
    @@ -154,7 +154,30 @@ abstract class VertexRDD[VD](
        * @return a VertexRDD containing the results of `f`
        */
       def leftZipJoin[VD2: ClassTag, VD3: ClassTag]
    -      (other: VertexRDD[VD2])(f: (VertexId, VD, Option[VD2]) => VD3): VertexRDD[VD3]
    +      (other: VertexRDD[VD2])
    +      (f: (VertexId, VD, Option[VD2]) => VD3)
    +    : VertexRDD[VD3]
    +
    +  /**
    +   * Left joins this RDD with another VertexRDD with the same index. This function will fail if
    +   * both VertexRDDs do not share the same index. The resulting vertex set contains an entry for
    +   * each vertex in `this`.
    +   * If `other` is missing any vertex in this VertexRDD, `f` is passed `None`.
    +   *
    +   * @tparam VD2 the attribute type of the other VertexRDD
    +   * @tparam VD3 the attribute type of the resulting VertexRDD
    +   * @tparam A the type of the given starting value and accumulator
    +   *
    +   * @param other the other VertexRDD with which to join.
    +   * @param acc the initial value for the accumulator
    +   * @param f the function mapping a vertex id and its attributes in this and the other vertex set
    +   * to a new vertex attribute.
    +   * @return a VertexRDD containing the results of `f`
    +   */
    +  def leftZipJoinWithFold[VD2: ClassTag, VD3: ClassTag, A]
    +      (other: VertexRDD[VD2], acc: A)
    +      (f: (A, VertexId, VD, Option[VD2]) => VD3)
    --- End diff --
    
    ISTM, not 'acc' but 'initVal' would  be better for an initial value.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85182245
  
      [Test build #29029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29029/consoleFull) for   PR 5142 at commit [`170d6f9`](https://github.com/apache/spark/commit/170d6f971f29049715cb4aff919ac4e6d7855020).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85791447
  
      [Test build #29140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29140/consoleFull) for   PR 5142 at commit [`a1d758b`](https://github.com/apache/spark/commit/a1d758b415adf7c7347ea0c1cad0f8f59c4a38c9).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85740018
  
      [Test build #29122 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29122/consoleFull) for   PR 5142 at commit [`a1d758b`](https://github.com/apache/spark/commit/a1d758b415adf7c7347ea0c1cad0f8f59c4a38c9).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85247503
  
      [Test build #29033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29033/consoleFull) for   PR 5142 at commit [`c6744b8`](https://github.com/apache/spark/commit/c6744b82776263889c7a5eb7664835419834d28b).
     * This patch **passes all tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-136915436
  
    @ankurdave any thoughts on this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by ankurdave <gi...@git.apache.org>.
Github user ankurdave commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85805531
  
    Hmm, this doesn't really fulfill the role of folds at the moment. The initial values are passed to a "fold" function that otherwise does the same thing as a reduce function, so it's equivalent to the user just generating the initial values in the fold function themselves.
    
    With SPARK-4086, I think @jegonzal and I were imagining a use case where you have an Iterator[(VertexId, A)] with duplicate keys and you want a VertexRDD[Array[A]]. Currently the only way is to map each element to an array and then concatenate arrays, which creates a lot of garbage. With folds you could use an empty ArrayBuffer as the initial value and do in-place inserts as elements appear.
    
    So it only seems to make sense to have a fold version of `aggregateUsingIndex` and the `VertexRDD.apply` constructor (maybe called `VertexRDD.createWithFold`), not the joins, because joining two VertexRDDs will never involve more than two values for a key.
    
    Also, in general the fold function has to return a value of the same type as the init value, e.g. `def fold[A](initVal: A, f: (VD, A) => A): VertexRDD[A]`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85818212
  
      [Test build #29140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29140/consoleFull) for   PR 5142 at commit [`a1d758b`](https://github.com/apache/spark/commit/a1d758b415adf7c7347ea0c1cad0f8f59c4a38c9).
     * This patch **passes all tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-136969040
  
    (Is GraphX active anymore? I haven't seen activity or responses on these issues in quite a while)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85595999
  
      [Test build #29094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29094/consoleFull) for   PR 5142 at commit [`d519370`](https://github.com/apache/spark/commit/d519370c8b836a4a9edb8eab4e022bb976bb7d27).
     * This patch **fails Scala style tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85135520
  
      [Test build #29016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29016/consoleFull) for   PR 5142 at commit [`6a61364`](https://github.com/apache/spark/commit/6a6136424ab2805148e141471fb2e22d37223d05).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by jegonzal <gi...@git.apache.org>.
Github user jegonzal commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-137170613
  
    @srowen GraphX is still active we have just been pretty busy with some other changes.  Let me see what needs to be done with this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85653703
  
      [Test build #29096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29096/consoleFull) for   PR 5142 at commit [`c42cd54`](https://github.com/apache/spark/commit/c42cd548c8933838198e65de8e4e80a4bdb13401).
     * This patch **passes all tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/5142


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85135778
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29016/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5142#discussion_r27045011
  
    --- Diff: graphx/src/main/scala/org/apache/spark/graphx/impl/VertexPartitionBaseOps.scala ---
    @@ -136,6 +136,31 @@ private[graphx] abstract class VertexPartitionBaseOps
         leftJoin(createUsingIndex(other))(f)
       }
     
    +  def leftJoinWithFold[VD2: ClassTag, VD3: ClassTag, A]
    +      (other: Self[VD2], acc: A)
    --- End diff --
    
    I'm pretty sure @ankurdave was looking at the `*WithFold` style addition and not changing the original method names to keep backwards compatibility as best as possible. I agree its a bit confusing, but I figure maintaining backwards compatibility as best as best is the better option.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85759797
  
      [Test build #29122 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29122/consoleFull) for   PR 5142 at commit [`a1d758b`](https://github.com/apache/spark/commit/a1d758b415adf7c7347ea0c1cad0f8f59c4a38c9).
     * This patch **fails Spark unit tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85818242
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29140/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5142#discussion_r27005643
  
    --- Diff: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala ---
    @@ -154,7 +154,30 @@ abstract class VertexRDD[VD](
        * @return a VertexRDD containing the results of `f`
        */
       def leftZipJoin[VD2: ClassTag, VD3: ClassTag]
    -      (other: VertexRDD[VD2])(f: (VertexId, VD, Option[VD2]) => VD3): VertexRDD[VD3]
    +      (other: VertexRDD[VD2])
    +      (f: (VertexId, VD, Option[VD2]) => VD3)
    +    : VertexRDD[VD3]
    +
    +  /**
    +   * Left joins this RDD with another VertexRDD with the same index. This function will fail if
    +   * both VertexRDDs do not share the same index. The resulting vertex set contains an entry for
    --- End diff --
    
    Is "will fail" correct? In the implementation, I think the codes fail back to creating same indices. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-96769296
  
      [Test build #31044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31044/consoleFull) for   PR 5142 at commit [`a1d758b`](https://github.com/apache/spark/commit/a1d758b415adf7c7347ea0c1cad0f8f59c4a38c9).
     * This patch **does not merge cleanly**.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85228796
  
      [Test build #29033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29033/consoleFull) for   PR 5142 at commit [`c6744b8`](https://github.com/apache/spark/commit/c6744b82776263889c7a5eb7664835419834d28b).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85759824
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29122/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85220210
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29029/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5142#discussion_r27005945
  
    --- Diff: graphx/src/main/scala/org/apache/spark/graphx/impl/VertexPartitionBaseOps.scala ---
    @@ -136,6 +136,31 @@ private[graphx] abstract class VertexPartitionBaseOps
         leftJoin(createUsingIndex(other))(f)
       }
     
    +  def leftJoinWithFold[VD2: ClassTag, VD3: ClassTag, A]
    +      (other: Self[VD2], acc: A)
    --- End diff --
    
    Reduce-style aggregation is named 'leftJoin', and fold-style aggregation is 'leftJoinWithFold'.
    A rule of naming is kind of inconsistent, so would it be better to rename 'leftJoin' into 'leftJoinWithReduce'? Thought?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85787944
  
    lgtm except for the naming issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85142892
  
      [Test build #29018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29018/consoleFull) for   PR 5142 at commit [`e661a8f`](https://github.com/apache/spark/commit/e661a8f3b146eef237777aa668b2c321fecdc8fc).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85220152
  
      [Test build #29029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29029/consoleFull) for   PR 5142 at commit [`170d6f9`](https://github.com/apache/spark/commit/170d6f971f29049715cb4aff919ac4e6d7855020).
     * This patch **fails MiMa tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85172275
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29018/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85789231
  
    jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5142#discussion_r27045035
  
    --- Diff: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala ---
    @@ -154,7 +154,30 @@ abstract class VertexRDD[VD](
        * @return a VertexRDD containing the results of `f`
        */
       def leftZipJoin[VD2: ClassTag, VD3: ClassTag]
    -      (other: VertexRDD[VD2])(f: (VertexId, VD, Option[VD2]) => VD3): VertexRDD[VD3]
    +      (other: VertexRDD[VD2])
    +      (f: (VertexId, VD, Option[VD2]) => VD3)
    +    : VertexRDD[VD3]
    +
    +  /**
    +   * Left joins this RDD with another VertexRDD with the same index. This function will fail if
    +   * both VertexRDDs do not share the same index. The resulting vertex set contains an entry for
    +   * each vertex in `this`.
    +   * If `other` is missing any vertex in this VertexRDD, `f` is passed `None`.
    +   *
    +   * @tparam VD2 the attribute type of the other VertexRDD
    +   * @tparam VD3 the attribute type of the resulting VertexRDD
    +   * @tparam A the type of the given starting value and accumulator
    +   *
    +   * @param other the other VertexRDD with which to join.
    +   * @param acc the initial value for the accumulator
    +   * @param f the function mapping a vertex id and its attributes in this and the other vertex set
    +   * to a new vertex attribute.
    +   * @return a VertexRDD containing the results of `f`
    +   */
    +  def leftZipJoinWithFold[VD2: ClassTag, VD3: ClassTag, A]
    +      (other: VertexRDD[VD2], acc: A)
    +      (f: (A, VertexId, VD, Option[VD2]) => VD3)
    --- End diff --
    
    Very good point. Will update that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85621704
  
      [Test build #29096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29096/consoleFull) for   PR 5142 at commit [`c42cd54`](https://github.com/apache/spark/commit/c42cd548c8933838198e65de8e4e80a4bdb13401).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-168112617
  
    I'm going to close this pull request. If this is still relevant and you are interested in pushing it forward, please open a new pull request. Thanks!



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85247520
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29033/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-85256908
  
    @ankurdave thoughts / comments? Is this in line with what you were looking for?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5142#discussion_r27046758
  
    --- Diff: graphx/src/main/scala/org/apache/spark/graphx/VertexRDD.scala ---
    @@ -154,7 +154,30 @@ abstract class VertexRDD[VD](
        * @return a VertexRDD containing the results of `f`
        */
       def leftZipJoin[VD2: ClassTag, VD3: ClassTag]
    -      (other: VertexRDD[VD2])(f: (VertexId, VD, Option[VD2]) => VD3): VertexRDD[VD3]
    +      (other: VertexRDD[VD2])
    +      (f: (VertexId, VD, Option[VD2]) => VD3)
    +    : VertexRDD[VD3]
    +
    +  /**
    +   * Left joins this RDD with another VertexRDD with the same index. This function will fail if
    +   * both VertexRDDs do not share the same index. The resulting vertex set contains an entry for
    --- End diff --
    
    Very true. Rereading the docs it looks like they haven't been updated with verbage of a few of the more recent bug fixes. Will add that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4086][GraphX]: Fold-style aggregation f...

Posted by brennonyork <gi...@git.apache.org>.
Github user brennonyork commented on the pull request:

    https://github.com/apache/spark/pull/5142#issuecomment-86178368
  
    @ankurdave, thanks for the clarification. Let me take a second stab at this given what you stated and I should have something much more in line with the original thought!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org