You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tdhopper <gi...@git.apache.org> on 2014/10/22 21:57:32 UTC

[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

GitHub user tdhopper opened a pull request:

    https://github.com/apache/spark/pull/2895

    Clarify docstring for Pyspark's foreachPartition

    Due to the underlying use of `mapPartitions` which requires a function that maps partitions to partitions, `foreachPartition` requires the function passed to be a generator function or return an iterable (although these results are discarded). 
    
    This is currently not stated in the documentation except through the unexplained example. It would help users to understand that example and not waste time with this error:
    
    ```
    TypeError: 'NoneType' object is not iterable
    ```

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tdhopper/spark patch-1

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/2895.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2895
    
----
commit e22f753ebcd5c7f44f9699481ad26d2263290097
Author: Timothy Hopper <td...@gmail.com>
Date:   2014-10-22T19:56:19Z

    Clarify docstring for foreachPartition
    
    Due to the underlying use of `mapPartitions` which requires a function that maps partitions to partitions, `foreachPartition` requires the function passed to be a generator function or return an iterable (although these results are discarded). 
    
    This is currently not stated in the documentation except through the unexplained example. It would help users to understand that example and not waste time with this error:
    
    ```
    TypeError: 'NoneType' object is not iterable
    ```

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by tdhopper <gi...@git.apache.org>.
Github user tdhopper commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60234425
  
    Oh. Now that I look at master, @JoshRosen, I see that it's already been fixed by @davis [here](https://github.com/apache/spark/commit/1789cd46e38d1426deb6a4b14bddcbb8c751f585). The fix just isn't in 1.1. I guess we should close this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60145810
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by tdhopper <gi...@git.apache.org>.
Github user tdhopper commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-67162827
  
    @JoshRosen: Yup. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by davies <gi...@git.apache.org>.
Github user davies commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60292102
  
    @JoshRosen  It will be better if we could easily backport them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by tdhopper <gi...@git.apache.org>.
Github user tdhopper commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60304723
  
    I'd love to see this happen.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60306682
  
    (Imagine what `@Override`'s notification inbox must look like...)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60159250
  
    Actually, we might want to just fix this and allow `foreachPartition` and `foreach` to accept those types of UDFs, too. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60306509
  
    @davis I think someone mentioned you instead of @davies (an off-by-one-character error).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by tdhopper <gi...@git.apache.org>.
Github user tdhopper closed the pull request at:

    https://github.com/apache/spark/pull/2895


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60287875
  
    @davies Do you think we should backport #2093 to branch-1.1 in order to fix this issue?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60287652
  
    Maybe we can backport SPARK-2871 to 1.1, since it looks like it also fixes a bunch of preservesPartitioning bugs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by davis <gi...@git.apache.org>.
Github user davis commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-60306345
  
    hm why am i notified here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: Clarify docstring for Pyspark's foreachPartiti...

Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:

    https://github.com/apache/spark/pull/2895#issuecomment-67102429
  
    If you don't mind, could you close this PR since it has been subsumed by another commit?  If we want to track the progress / backport status of a different fix, then we should do that in JIRA.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org