You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by coderxiang <gi...@git.apache.org> on 2014/02/23 06:30:21 UTC

[GitHub] incubator-spark pull request: allCollect functions for RDD

GitHub user coderxiang opened a pull request:

    https://github.com/apache/incubator-spark/pull/635

    allCollect functions for RDD

     Two methods (`allCollect`, `allCollectBroadcast`) are added to `RDD[T]`, which output a new `RDD[Array[T]]` instance with each partition containing all of the records of the original RDD stored in a single `Array[T]` instance (the same as RDD.collect). This functionality can be useful in machine learning tasks that require sharing updated parameters across partitions.
    
    Method `allCollect` creates a new `AllCollectedRDD` while method `allCollectBroadcast` applies broadcasting. Both of them need collecting the data and therefore should deliver similar performance.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/coderxiang/incubator-spark allCollect

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-spark/pull/635.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #635
    
----
commit 74e7817d73bf635f9d27149d609b4da1c1e78f51
Author: lebesgue <le...@lebesgue.net>
Date:   2014-02-21T03:46:30Z

    add a simple implementation of allCollect using AllCollectedRDD

commit b619cbd2642502a80f53bf6f4b753301cb157956
Author: lebesgue <le...@lebesgue.net>
Date:   2014-02-21T03:51:07Z

    code reorganization

commit f727cd936dfad513238d8a127fdc15507a4025b0
Author: lebesgue <le...@lebesgue.net>
Date:   2014-02-21T06:21:09Z

    add the implementation of allCollect using a broadcast variable

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35969142
  
    @JoshRosen I didn't implement this ... AllCollect is better than directly using broadcast variables if it is implemented without putting heavy load on the driver and data is not very small. But the current implementation is no better than directly broadcasting variables. A slightly better solution would be shuffle-based, which does not put load on the driver but it might create duplicate blocks at the same physical node. The efficient broadcasting you described sounds interesting, but yes I can imagine it is difficult to implement. Thanks for sharing your thoughts!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-spark pull request: allCollect functions for RDD

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35824561
  
    ah, it's here: http://apache-spark-developers-list.1001551.n3.nabble.com/Proposal-for-JIRA-and-Pull-Request-Policy-td505.html


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

Posted by coderxiang <gi...@git.apache.org>.
Github user coderxiang commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35824621
  
    @CodingCat Updated. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35824836
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: allCollect functions for RDD

Posted by coderxiang <gi...@git.apache.org>.
Github user coderxiang commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35824486
  
    @CodingCat  Thanks for mentioning. I do create a JIRA here: https://spark-project.atlassian.net/browse/SPARK-1122


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

Posted by markhamstra <gi...@git.apache.org>.
Github user markhamstra commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35825888
  
    Huh?  I don't get the point of these at all.
    
    At first glance, allCollect looks like a really bad idea.  Collecting the entire contents of an RDD to the driver process only to immediately turn around and push all of that data (or in this case, multiple copies of the data!) back across the network is an anti-pattern and generally a very poor design choice that cannot scale to large data -- if you can handle all of the data within the driver process, then why are you using a distributed, big-data framework in the first place?
    
    allCollectBroadcast makes even less sense to me.  Some workflows do demand collecting a relatively small amount of data to the driver and then broadcasting a small amount back to the workers for use in further computations, but why would I then want to go through the extra step of pushing the broadcast values into a strange-looking RDD instead of just using the broadcast variable directly?
    
    It's going to take a lot of persuading to convince me that either of these are things we want to promote and support in the 1.0 API.  That doesn't mean that I'm not listening, but I am far from convinced at this point.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35841766
  
    Hey @coderxiang - this is interesting functionality but I'm -1 on including it in the standard API. The main reason is that this will perform poorly on most large datasets and make it easy for people to shoot themselves in the foot. A second reason is that the use case isn't totally clear - as per some of @markhamstra's comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: allCollect functions for RDD

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35824526
  
    @coderxiang I mean they suggest including the JIRA id in the PR title,I remember so...there was a discussion in dev list...cc: @pwendell


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35848534
  
    @coderxiang btw - it might be something where we make it a private API so it can be used inside of Spark if other packages need this to do broadcast joins. It would be good to understand a bit the intended use case though.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35967399
  
    @mengxr For your examples (training multiple models in parallel and broadcast join), why is `allCollect` better than directly using broadcast variables?  Why do you want to access your model/table through an RDD with a single array value?  It seems like you'd need to use something like `zipPartitions()` to pair up the model with your data, so why not just directly reference the broadcast variable in a `mapPartitions()` on your varying dataset?
    
    In the past, I've thought about implementing a `SparkContext.broadcast[T](rdd: RDD[T]): Broadcast[T]` for creating broadcast variables from RDDs.  This can be implemented through peer-to-peer broadcasting of the  RDD fragments rather than broadcasting the entire collected RDD from the driver; this avoids bottlenecking on the driver's send bandwidth.
    
    I tried to prototype efficient broadcasting of RDDs, but ran into some difficulties.  My implementation used its own Broadcast subclass that fetched RDD partitions from remote block managers when trying to access the broadcast variable's value.  The problem is that we need to run some job on the broadcasted RDD to generate the partitions that the Broadcast subclass will fetch, and we need to ensure that those partitions are computed and stored before we attempt to fetch them.  Thus, any transformation that references a BroadcastRDD variable must declare a dependency on the stage that produces those blocks.  Usually, RDDs declare their dependencies when they're constructed; with BroadcastRDDs, we'd need to actually look inside the UDF to find any references in order to build the proper lineage.
    
    On a related note, it would be nice to implement an `allReduce` function that's equivalent to `reduce().broadcast()` but performs similar optimizations to avoid coordinating through the driver.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-spark pull request: SPARK-1122: allCollect functions for...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35849312
  
    @markhamstra @pwendell For the use cases, this allCollect operation may be useful in the grid search for a good set of training parameters for machine learning problems. For example, if the dataset is only 500MB but training takes half an hour to finish and we have to try 100 different combinations of training parameters (e.g., rank, regularization constants, and termination tolerance), the wall-clock time can be reduced by distributing the dataset to multiple nodes and training in parallel. Another use case is the replicated join, though locality issues need to be addressed. I agree with you that the implementation is not efficient, which puts heavy load on the driver.
    
    @coderxiang , could you try to improve the implementation? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---

[GitHub] incubator-spark pull request: allCollect functions for RDD

Posted by CodingCat <gi...@git.apache.org>.
Github user CodingCat commented on the pull request:

    https://github.com/apache/incubator-spark/pull/635#issuecomment-35824430
  
    I think now they require that every PR corresponds to a certain JIRA...https://spark-project.atlassian.net/browse/SPARK


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. To do so, please top-post your response.
If your project does not have this feature enabled and wishes so, or if the
feature is enabled but not working, please contact infrastructure at
infrastructure@apache.org or file a JIRA ticket with INFRA.
---