You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sryza <gi...@git.apache.org> on 2014/11/05 08:35:24 UTC
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
GitHub user sryza opened a pull request:
https://github.com/apache/spark/pull/3107
SPARK-4230. Doc for spark.default.parallelism is incorrect
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sryza/spark sandy-spark-4230
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3107.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3107
----
commit 14ca79b7fad5cfb542a3010452546dc2cbcbb28a
Author: Sandy Ryza <sa...@cloudera.com>
Date: 2014-11-05T02:24:28Z
SPARK-4230. Doc for spark.default.parallelism is incorrect
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-61777707
[Test build #22921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22921/consoleFull) for PR 3107 at commit [`14ca79b`](https://github.com/apache/spark/commit/14ca79b7fad5cfb542a3010452546dc2cbcbb28a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-62334702
[Test build #23129 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23129/consoleFull) for PR 3107 at commit [`37a1d19`](https://github.com/apache/spark/commit/37a1d19db5e983487e4bb05ac553082c94944a13).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3107#discussion_r20060629
--- Diff: docs/configuration.md ---
@@ -556,6 +556,9 @@ Apart from these, the following properties are also available, and may be useful
<tr>
<td><code>spark.default.parallelism</code></td>
<td>
+ For distributed shuffle operations like <code>reduceByKey</code> and <code>join</code>, the
+ largest number of partitions in parent RDD. For operations like <code>parallelize</code> with
--- End diff --
I was worried "the number of partitions of the largest parent RDD" could be construed as "the number of partitions in the parent RDD containing the most data".
Do you think "the largest number of partitions in _a_ parent RDD" or "the largest number of partitions in one of the operation's input RDDs" would be more clear?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3107#discussion_r20063433
--- Diff: docs/configuration.md ---
@@ -556,6 +556,9 @@ Apart from these, the following properties are also available, and may be useful
<tr>
<td><code>spark.default.parallelism</code></td>
<td>
+ For distributed shuffle operations like <code>reduceByKey</code> and <code>join</code>, the
+ largest number of partitions in parent RDD. For operations like <code>parallelize</code> with
--- End diff --
Yeah - if you just add "in a parent RDD" then that seems good!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3107#discussion_r20063448
--- Diff: docs/configuration.md ---
@@ -563,8 +566,8 @@ Apart from these, the following properties are also available, and may be useful
</ul>
</td>
<td>
- Default number of tasks to use across the cluster for distributed shuffle operations
- (<code>groupByKey</code>, <code>reduceByKey</code>, etc) when not set by user.
+ Default number of output partitions for operations like <code>join</code>,
--- End diff --
Ah I see - what about "Default number of partitions in RDD's returned by join, reduceByKey..."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-62338395
[Test build #23129 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23129/consoleFull) for PR 3107 at commit [`37a1d19`](https://github.com/apache/spark/commit/37a1d19db5e983487e4bb05ac553082c94944a13).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3107
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-62339705
Test failure looks unrelated
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-61777715
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22921/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-62338398
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23129/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-61769974
[Test build #22921 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22921/consoleFull) for PR 3107 at commit [`14ca79b`](https://github.com/apache/spark/commit/14ca79b7fad5cfb542a3010452546dc2cbcbb28a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-62313723
Had some minor wording questions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by sryza <gi...@git.apache.org>.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3107#discussion_r20060674
--- Diff: docs/configuration.md ---
@@ -563,8 +566,8 @@ Apart from these, the following properties are also available, and may be useful
</ul>
</td>
<td>
- Default number of tasks to use across the cluster for distributed shuffle operations
- (<code>groupByKey</code>, <code>reduceByKey</code>, etc) when not set by user.
+ Default number of output partitions for operations like <code>join</code>,
--- End diff --
My thinking was that Spark's APIs have no mention of the concept of a "shuffle partition" (e.g. the term is referenced nowhere on https://spark.apache.org/docs/latest/programming-guide.html), but even novice Spark users are meant to understand that every transformation has input and output RDDs and that every RDD has a number of partitions.
Maybe "Default number of partitions for the RDDs produced by operations like ..."?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/3107#issuecomment-62451083
LG - pulling it in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3107#discussion_r20060444
--- Diff: docs/configuration.md ---
@@ -563,8 +566,8 @@ Apart from these, the following properties are also available, and may be useful
</ul>
</td>
<td>
- Default number of tasks to use across the cluster for distributed shuffle operations
- (<code>groupByKey</code>, <code>reduceByKey</code>, etc) when not set by user.
+ Default number of output partitions for operations like <code>join</code>,
--- End diff --
Should this say "number of shuffle partitions" - it's slightly weird to me to say "output" when this refers to something that is totally internal to Spark - it's output on the map side but input on he read side. In other cases I think output tends to mean things like saving as HDFS data, etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-4230. Doc for spark.default.parallelism ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/3107#discussion_r20060453
--- Diff: docs/configuration.md ---
@@ -556,6 +556,9 @@ Apart from these, the following properties are also available, and may be useful
<tr>
<td><code>spark.default.parallelism</code></td>
<td>
+ For distributed shuffle operations like <code>reduceByKey</code> and <code>join</code>, the
+ largest number of partitions in parent RDD. For operations like <code>parallelize</code> with
--- End diff --
Is this just "the number of partitions in the parent RDD" (why largest?) Doesn't the parentRDD have the a fixed number of partitions? Or is this a maximum across all parents...?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org