You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by lins05 <gi...@git.apache.org> on 2017/04/03 09:00:37 UTC
[GitHub] spark pull request #17519: [SPARK-15352][Doc] follow-up: add configuration d...
GitHub user lins05 opened a pull request:
https://github.com/apache/spark/pull/17519
[SPARK-15352][Doc] follow-up: add configuration docs for topology-awa block replication
## What changes were proposed in this pull request?
Add configuration docs for topology-awa block replication
## How was this patch tested?
Generating the docs and preview it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lins05/spark spark-15352-add-docs-for-topology-aware-block-replication
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17519.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17519
----
commit c86044e803546c31e4da793bde201120c06976ca
Author: Shuai Lin <li...@gmail.com>
Date: 2017-04-03T09:00:22Z
[SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17519
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75476/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17519
**[Test build #75478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75478/testReport)** for PR 17519 at commit [`bf56ba2`](https://github.com/apache/spark/commit/bf56ba274b00962bd5ccc9c0b098f17b934ea5fb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17519
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75478/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17519
Hi @lins05, is this PR active?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17519
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by shubhamchopra <gi...@git.apache.org>.
Github user shubhamchopra commented on the issue:
https://github.com/apache/spark/pull/17519
@lins05 Apologies for the delay in responding and thanks for adding the docs. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17519
cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17519
**[Test build #75476 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75476/testReport)** for PR 17519 at commit [`c86044e`](https://github.com/apache/spark/commit/c86044e803546c31e4da793bde201120c06976ca).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17519: [SPARK-15352][Doc] follow-up: add configuration d...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17519
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/17519
Thanks @shubhamchopra. Could you please help review & merge this one @cloud-fan ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17519: [SPARK-15352][Doc] follow-up: add configuration d...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17519#discussion_r123424505
--- Diff: docs/configuration.md ---
@@ -1004,14 +1004,48 @@ Apart from these, the following properties are also available, and may be useful
</td>
</tr>
<tr>
- <td><code>spark.storage.replication.proactive<code></td>
+ <td><code>spark.storage.replication.proactive</code></td>
<td>false</td>
<td>
Enables proactive block replication for RDD blocks. Cached RDD block replicas lost due to
executor failures are replenished if there are any existing available replicas. This tries
to get the replication level of the block to the initial number.
</td>
</tr>
+<tr>
+ <td><code>spark.storage.replication.policy</code></td>
+ <td>
+ org.apache.spark.storage.<br />RandomBlockReplicationPolicy
+ </td>
+ <td>
+ The policy to use for choosing peers when replicating blocks. The default policy would randomly
+ choose the peers to replicate to. A more resilient replication policy is provided by
+ <code>org.apache.spark.storage.BasicBlockReplicationPolicy</code>, which makes use of the
+ topology information of the hosts to choose the peers, much like the HDFS blocks replication
+ strategy: it would try to choose the first replica within the same rack, and a third replica on
+ a different rack. See <code>spark.storage.replication.topologyMapper</code> below for how to
+ provide the topology information for the hosts.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.storage.replication.topologyMapper</code></td>
+ <td>
+ org.apache.spark.storage.<br />DefaultTopologyMapper
+ </td>
+ <td>
+ The topology information of a host is determined by a topology mapping service defined by the
+ abstract class <code>org.apache.spark.storage.TopologyMapper</code>, which can be configured by
+ this property. A default implementation that assumes all hosts are in the same rack is provided
+ by <code>org.apache.spark.storage.DefaultTopologyMapper</code>. A file-based implementation is
+ provided by <code>org.apache.spark.storage.FileBasedTopologyMapper</code>, which reads the
+ topology information from the file <code>org.apache.spark.storage.topologyFile</code>. Each line
+ of this file is of the format of <code>host1 = /rack1</code> and provides a mapping from a host
+ name to its rack information. <em>Note:</em> This configuration only takes effect when
+ <code>spark.storage.replication.policy</code> is set to a a policy that takes the topology
--- End diff --
nit: double `a`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17519
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17519
**[Test build #75476 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75476/testReport)** for PR 17519 at commit [`c86044e`](https://github.com/apache/spark/commit/c86044e803546c31e4da793bde201120c06976ca).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/17519
@shubhamchopra @cloud-fan PTAL
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17519
**[Test build #75478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75478/testReport)** for PR 17519 at commit [`bf56ba2`](https://github.com/apache/spark/commit/bf56ba274b00962bd5ccc9c0b098f17b934ea5fb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17519: [SPARK-15352][Doc] follow-up: add configuration docs for...
Posted by lins05 <gi...@git.apache.org>.
Github user lins05 commented on the issue:
https://github.com/apache/spark/pull/17519
ping @shubhamchopra @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17519: [SPARK-15352][Doc] follow-up: add configuration d...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17519#discussion_r123424493
--- Diff: docs/configuration.md ---
@@ -1004,14 +1004,48 @@ Apart from these, the following properties are also available, and may be useful
</td>
</tr>
<tr>
- <td><code>spark.storage.replication.proactive<code></td>
+ <td><code>spark.storage.replication.proactive</code></td>
<td>false</td>
<td>
Enables proactive block replication for RDD blocks. Cached RDD block replicas lost due to
executor failures are replenished if there are any existing available replicas. This tries
to get the replication level of the block to the initial number.
</td>
</tr>
+<tr>
+ <td><code>spark.storage.replication.policy</code></td>
+ <td>
+ org.apache.spark.storage.<br />RandomBlockReplicationPolicy
+ </td>
+ <td>
+ The policy to use for choosing peers when replicating blocks. The default policy would randomly
+ choose the peers to replicate to. A more resilient replication policy is provided by
+ <code>org.apache.spark.storage.BasicBlockReplicationPolicy</code>, which makes use of the
+ topology information of the hosts to choose the peers, much like the HDFS blocks replication
+ strategy: it would try to choose the first replica within the same rack, and a third replica on
+ a different rack. See <code>spark.storage.replication.topologyMapper</code> below for how to
+ provide the topology information for the hosts.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.storage.replication.topologyMapper</code></td>
+ <td>
+ org.apache.spark.storage.<br />DefaultTopologyMapper
+ </td>
+ <td>
+ The topology information of a host is determined by a topology mapping service defined by the
+ abstract class <code>org.apache.spark.storage.TopologyMapper</code>, which can be configured by
+ this property. A default implementation that assumes all hosts are in the same rack is provided
+ by <code>org.apache.spark.storage.DefaultTopologyMapper</code>. A file-based implementation is
+ provided by <code>org.apache.spark.storage.FileBasedTopologyMapper</code>, which reads the
+ topology information from the file <code>org.apache.spark.storage.topologyFile</code>. Each line
--- End diff --
shall we also add an entry for `org.apache.spark.storage.topologyFile`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org