You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mortada <gi...@git.apache.org> on 2016/01/21 17:58:48 UTC
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
GitHub user mortada opened a pull request:
https://github.com/apache/spark/pull/10867
[SPARK-12760] [DOCS] invalid lambda expression in python example for …
…local vs cluster
@srowen thanks for the PR at https://github.com/apache/spark/pull/10866! sorry it took me a while.
This is related to https://github.com/apache/spark/pull/10866, basically the assignment in the lambda expression in the python example is actually invalid
```
In [1]: data = [1, 2, 3, 4, 5]
In [2]: counter = 0
In [3]: rdd = sc.parallelize(data)
In [4]: rdd.foreach(lambda x: counter += x)
File "<ipython-input-4-fcb86c182bad>", line 1
rdd.foreach(lambda x: counter += x)
^
SyntaxError: invalid syntax
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mortada/spark doc_python_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10867.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10867
----
commit fc9f16a2ffb5846ecc03c4df584f611e6728573d
Author: Mortada Mehyar <mo...@gmail.com>
Date: 2016-01-21T16:51:28Z
[SPARK-12760] [DOCS] invalid lambda expression in python example for local vs cluster
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-174176274
Merged to master/1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10867
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by mortada <gi...@git.apache.org>.
Github user mortada commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173648674
@srowen I tested the python code in cluster mode (5 ec2 workers) and this works fine
```
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.10.56:35937 with 6.6 GB RAM, BlockManagerId(4, 172.31.10.56, 35937)
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.10.55:59871 with 6.6 GB RAM, BlockManagerId(0, 172.31.10.55, 59871)
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.10.53:39162 with 6.6 GB RAM, BlockManagerId(1, 172.31.10.53, 39162)
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.10.54:59145 with 6.6 GB RAM, BlockManagerId(2, 172.31.10.54, 59145)
16/01/21 17:33:29 INFO BlockManagerMasterEndpoint: Registering block manager 172.31.10.57:35000 with 6.6 GB RAM, BlockManagerId(3, 172.31.10.57, 35000)
In [1]: data = [1, 2, 3, 4, 5]
In [2]: counter = 0
In [3]: rdd = sc.parallelize(data)
In [4]: def increment_counter(x):
global counter
counter += x
...:
In [5]: rdd.foreach(increment_counter)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.31.10.55:59871 (size: 3.2 KB, free: 6.6 GB)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.31.10.56:35937 (size: 3.2 KB, free: 6.6 GB)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.31.10.57:35000 (size: 3.2 KB, free: 6.6 GB)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.31.10.53:39162 (size: 3.2 KB, free: 6.6 GB)
16/01/21 17:34:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.31.10.54:59145 (size: 3.2 KB, free: 6.6 GB)
(other output skipped)
In [6]: print("Counter value: ", counter)
Counter value: 0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by mortada <gi...@git.apache.org>.
Github user mortada commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173639832
@srowen it compiles for local, let me test that on a cluster
I noticed that the next line is actually also invalid python
```
In [7]: print("Counter value: " + counter)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-7-e340457a6af8> in <module>()
----> 1 print("Counter value: " + counter)
TypeError: Can't convert 'int' object to str implicitly
```
I just updated the PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173966638
**[Test build #49919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49919/consoleFull)** for PR 10867 at commit [`2e1c016`](https://github.com/apache/spark/commit/2e1c0161f8506bccae619e5aae62a69cd16ebd13).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173959712
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173966797
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49919/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173966795
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173635004
Does it still execute without error on a cluster? (even if it doesn't actually increment the counter in the way someone might expect.) Certainly if it doesn't compile we need to change this, but want to make sure the result with "global" executes too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173643525
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173963608
**[Test build #49919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49919/consoleFull)** for PR 10867 at commit [`2e1c016`](https://github.com/apache/spark/commit/2e1c0161f8506bccae619e5aae62a69cd16ebd13).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173959614
LGTM then. I can merge this with the other doc updates
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173635852
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173643528
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49880/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12760] [DOCS] invalid lambda expression...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10867#issuecomment-173636353
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org