You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2017/07/05 13:40:47 UTC
spark git commit: [SPARK-20858][DOC][MINOR] Document ListenerBus
event queue size
Repository: spark
Updated Branches:
refs/heads/master e3e2b5da3 -> 960298ee6
[SPARK-20858][DOC][MINOR] Document ListenerBus event queue size
## What changes were proposed in this pull request?
This change adds a new configuration option `spark.scheduler.listenerbus.eventqueue.size` to the configuration docs to specify the capacity of the spark listener bus event queue. Default value is 10000.
This is doc PR for [SPARK-15703](https://issues.apache.org/jira/browse/SPARK-15703).
I added option to the `Scheduling` section, however it might be more related to `Spark UI` section.
## How was this patch tested?
Manually verified correct rendering of configuration option.
Author: sadikovi <iv...@lincolnuni.ac.nz>
Author: Ivan Sadikov <iv...@team.telstra.com>
Closes #18476 from sadikovi/SPARK-20858.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/960298ee
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/960298ee
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/960298ee
Branch: refs/heads/master
Commit: 960298ee66b9b8a80f84df679ce5b4b3846267f4
Parents: e3e2b5d
Author: sadikovi <iv...@lincolnuni.ac.nz>
Authored: Wed Jul 5 14:40:44 2017 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Jul 5 14:40:44 2017 +0100
----------------------------------------------------------------------
docs/configuration.md | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/960298ee/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index bd6a1f9..c785a66 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -725,7 +725,7 @@ Apart from these, the following properties are also available, and may be useful
<td><code>spark.ui.retainedJobs</code></td>
<td>1000</td>
<td>
- How many jobs the Spark UI and status APIs remember before garbage collecting.
+ How many jobs the Spark UI and status APIs remember before garbage collecting.
This is a target maximum, and fewer elements may be retained in some circumstances.
</td>
</tr>
@@ -733,7 +733,7 @@ Apart from these, the following properties are also available, and may be useful
<td><code>spark.ui.retainedStages</code></td>
<td>1000</td>
<td>
- How many stages the Spark UI and status APIs remember before garbage collecting.
+ How many stages the Spark UI and status APIs remember before garbage collecting.
This is a target maximum, and fewer elements may be retained in some circumstances.
</td>
</tr>
@@ -741,7 +741,7 @@ Apart from these, the following properties are also available, and may be useful
<td><code>spark.ui.retainedTasks</code></td>
<td>100000</td>
<td>
- How many tasks the Spark UI and status APIs remember before garbage collecting.
+ How many tasks the Spark UI and status APIs remember before garbage collecting.
This is a target maximum, and fewer elements may be retained in some circumstances.
</td>
</tr>
@@ -1390,6 +1390,15 @@ Apart from these, the following properties are also available, and may be useful
</td>
</tr>
<tr>
+ <td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
+ <td>10000</td>
+ <td>
+ Capacity for event queue in Spark listener bus, must be greater than 0. Consider increasing
+ value (e.g. 20000) if listener events are dropped. Increasing this value may result in the
+ driver using more memory.
+ </td>
+</tr>
+<tr>
<td><code>spark.blacklist.enabled</code></td>
<td>
false
@@ -1475,8 +1484,8 @@ Apart from these, the following properties are also available, and may be useful
<td><code>spark.blacklist.application.fetchFailure.enabled</code></td>
<td>false</td>
<td>
- (Experimental) If set to "true", Spark will blacklist the executor immediately when a fetch
- failure happenes. If external shuffle service is enabled, then the whole node will be
+ (Experimental) If set to "true", Spark will blacklist the executor immediately when a fetch
+ failure happenes. If external shuffle service is enabled, then the whole node will be
blacklisted.
</td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org