You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2020/04/05 04:18:37 UTC
[spark] branch branch-3.0 updated (a5444db -> dd97bcb)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git.
from a5444db [SPARK-31002][CORE][DOC][3.0] Add version information to the configuration of Core
new d7846cb [SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy
new 42b26cd [SPARK-30910][CORE][DOC] Add version information to the configuration of R
new 6e04912 [SPARK-30909][CORE][DOC] Add version information to the configuration of Python
new 2c5b13f [SPARK-30888][CORE][DOC] Add version information to the configuration of Network
new dd31736 [SPARK-30908][CORE][DOC] Add version information to the configuration of Kryo
new 231c0bb [SPARK-30891][CORE][DOC] Add version information to the configuration of History
new 056e60f [SPARK-30912][CORE][DOC] Add version information to the configuration of Streaming.scala
new dd97bcb [SPARK-30914][CORE][DOC] Add version information to the configuration of UI
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/spark/internal/config/Deploy.scala | 11 ++++
.../org/apache/spark/internal/config/History.scala | 28 +++++++++++
.../org/apache/spark/internal/config/Kryo.scala | 8 +++
.../org/apache/spark/internal/config/Network.scala | 14 ++++++
.../org/apache/spark/internal/config/Python.scala | 6 +++
.../scala/org/apache/spark/internal/config/R.scala | 5 ++
.../apache/spark/internal/config/Streaming.scala | 7 +++
.../org/apache/spark/internal/config/UI.scala | 26 ++++++++++
docs/configuration.md | 58 +++++++++++++++++++---
docs/monitoring.md | 26 +++++++++-
10 files changed, 181 insertions(+), 8 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 03/08: [SPARK-30909][CORE][DOC] Add version information to
the configuration of Python
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 6e04912c528038834501301587865c1a621112f7
Author: beliefer <be...@163.com>
AuthorDate: Thu Feb 27 10:57:34 2020 +0900
[SPARK-30909][CORE][DOC] Add version information to the configuration of Python
### What changes were proposed in this pull request?
1.Add version information to the configuration of `Python`.
2.Update the docs of `Python`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.python.worker.reuse | 1.2.0 | SPARK-3030 | 2aea0da84c58a179917311290083456dfa043db7#diff-0a67bc4d171abe4df8eb305b0f4123a2 |
spark.python.task.killTimeout | 2.2.2 | SPARK-22535 | be68f86e11d64209d9e325ce807025318f383bea#diff-0a67bc4d171abe4df8eb305b0f4123a2 |
spark.python.use.daemon | 2.3.0 | SPARK-22554 | 57c5514de9dba1c14e296f85fb13fef23ce8c73f#diff-9008ad45db34a7eee2e265a50626841b |
spark.python.daemon.module | 2.4.0 | SPARK-22959 | afae8f2bc82597593595af68d1aa2d802210ea8b#diff-9008ad45db34a7eee2e265a50626841b |
spark.python.worker.module | 2.4.0 | SPARK-22959 | afae8f2bc82597593595af68d1aa2d802210ea8b#diff-9008ad45db34a7eee2e265a50626841b |
spark.executor.pyspark.memory | 2.4.0 | SPARK-25004 | 7ad18ee9f26e75dbe038c6034700f9cd4c0e2baa#diff-6bdad48cfc34314e89599655442ff210 |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27704 from beliefer/add-version-to-python-config.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
core/src/main/scala/org/apache/spark/internal/config/Python.scala | 6 ++++++
docs/configuration.md | 6 ++++--
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/Python.scala b/core/src/main/scala/org/apache/spark/internal/config/Python.scala
index 26a0598..188d884 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/Python.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/Python.scala
@@ -22,26 +22,32 @@ import org.apache.spark.network.util.ByteUnit
private[spark] object Python {
val PYTHON_WORKER_REUSE = ConfigBuilder("spark.python.worker.reuse")
+ .version("1.2.0")
.booleanConf
.createWithDefault(true)
val PYTHON_TASK_KILL_TIMEOUT = ConfigBuilder("spark.python.task.killTimeout")
+ .version("2.2.2")
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("2s")
val PYTHON_USE_DAEMON = ConfigBuilder("spark.python.use.daemon")
+ .version("2.3.0")
.booleanConf
.createWithDefault(true)
val PYTHON_DAEMON_MODULE = ConfigBuilder("spark.python.daemon.module")
+ .version("2.4.0")
.stringConf
.createOptional
val PYTHON_WORKER_MODULE = ConfigBuilder("spark.python.worker.module")
+ .version("2.4.0")
.stringConf
.createOptional
val PYSPARK_EXECUTOR_MEMORY = ConfigBuilder("spark.executor.pyspark.memory")
+ .version("2.4.0")
.bytesConf(ByteUnit.MiB)
.createOptional
}
diff --git a/docs/configuration.md b/docs/configuration.md
index 4aff1ef..88edaf6 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -136,7 +136,7 @@ of the most common options to set are:
### Application Properties
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.app.name</code></td>
<td>(none)</td>
@@ -258,6 +258,7 @@ of the most common options to set are:
limitations are inherited. For instance, Windows does not support resource limiting and actual
resource is not limited on MacOS.
</td>
+ <td>2.4.0</td>
</tr>
<tr>
<td><code>spark.executor.memoryOverhead</code></td>
@@ -420,7 +421,7 @@ Apart from these, the following properties are also available, and may be useful
### Runtime Environment
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.driver.extraClassPath</code></td>
<td>(none)</td>
@@ -659,6 +660,7 @@ Apart from these, the following properties are also available, and may be useful
if there is a large broadcast, then the broadcast will not need to be transferred
from JVM to Python worker for every task.
</td>
+ <td>1.2.0</td>
</tr>
<tr>
<td><code>spark.files</code></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 01/08: [SPARK-30887][CORE][DOC] Add version information to
the configuration of Deploy
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit d7846cbc670bc7daa8e1520eedf80410c8943722
Author: beliefer <be...@163.com>
AuthorDate: Tue Feb 25 11:39:11 2020 +0900
[SPARK-30887][CORE][DOC] Add version information to the configuration of Deploy
### What changes were proposed in this pull request?
1.Add version information to the configuration of `Deploy`.
2.Update the docs of `Deploy`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.deploy.recoveryMode | 0.8.1 | None | d66c01f2b6defb3db6c1be99523b734a4d960532#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.recoveryMode.factory | 1.2.0 | SPARK-1830 | deefd9d7377a8091a1d184b99066febd0e9f6afd#diff-29dffdccd5a7f4c8b496c293e87c8668 | This configuration appears in branch-1.3, but the version number in the pom.xml file corresponding to the commit is 1.2.0-SNAPSHOT
spark.deploy.recoveryDirectory | 0.8.1 | None | d66c01f2b6defb3db6c1be99523b734a4d960532#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.zookeeper.url | 0.8.1 | None | d66c01f2b6defb3db6c1be99523b734a4d960532#diff-4457313ca662a1cd60197122d924585c |
spark.deploy.zookeeper.dir | 0.8.1 | None | d66c01f2b6defb3db6c1be99523b734a4d960532#diff-a84228cb45c7d5bd93305a1f5bf720b6 |
spark.deploy.retainedApplications | 0.8.0 | None | 46eecd110a4017ea0c86cbb1010d0ccd6a5eb2ef#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.retainedDrivers | 1.1.0 | None | 7446f5ff93142d2dd5c79c63fa947f47a1d4db8b#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.dead.worker.persistence | 0.8.0 | None | 46eecd110a4017ea0c86cbb1010d0ccd6a5eb2ef#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.maxExecutorRetries | 1.6.3 | SPARK-16956 | ace458f0330f22463ecf7cbee7c0465e10fba8a8#diff-29dffdccd5a7f4c8b496c293e87c8668 |
spark.deploy.spreadOut | 0.6.1 | None | bb2b9ff37cd2503cc6ea82c5dd395187b0910af0#diff-0e7ae91819fc8f7b47b0f97be7116325 |
spark.deploy.defaultCores | 0.9.0 | None | d8bcc8e9a095c1b20dd7a17b6535800d39bff80e#diff-29dffdccd5a7f4c8b496c293e87c8668 |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27668 from beliefer/add-version-to-deploy-config.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
.../main/scala/org/apache/spark/internal/config/Deploy.scala | 11 +++++++++++
docs/configuration.md | 5 ++++-
2 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/Deploy.scala b/core/src/main/scala/org/apache/spark/internal/config/Deploy.scala
index ceab957..d494c5e 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/Deploy.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/Deploy.scala
@@ -19,48 +19,59 @@ package org.apache.spark.internal.config
private[spark] object Deploy {
val RECOVERY_MODE = ConfigBuilder("spark.deploy.recoveryMode")
+ .version("0.8.1")
.stringConf
.createWithDefault("NONE")
val RECOVERY_MODE_FACTORY = ConfigBuilder("spark.deploy.recoveryMode.factory")
+ .version("1.2.0")
.stringConf
.createWithDefault("")
val RECOVERY_DIRECTORY = ConfigBuilder("spark.deploy.recoveryDirectory")
+ .version("0.8.1")
.stringConf
.createWithDefault("")
val ZOOKEEPER_URL = ConfigBuilder("spark.deploy.zookeeper.url")
.doc(s"When `${RECOVERY_MODE.key}` is set to ZOOKEEPER, this " +
"configuration is used to set the zookeeper URL to connect to.")
+ .version("0.8.1")
.stringConf
.createOptional
val ZOOKEEPER_DIRECTORY = ConfigBuilder("spark.deploy.zookeeper.dir")
+ .version("0.8.1")
.stringConf
.createOptional
val RETAINED_APPLICATIONS = ConfigBuilder("spark.deploy.retainedApplications")
+ .version("0.8.0")
.intConf
.createWithDefault(200)
val RETAINED_DRIVERS = ConfigBuilder("spark.deploy.retainedDrivers")
+ .version("1.1.0")
.intConf
.createWithDefault(200)
val REAPER_ITERATIONS = ConfigBuilder("spark.dead.worker.persistence")
+ .version("0.8.0")
.intConf
.createWithDefault(15)
val MAX_EXECUTOR_RETRIES = ConfigBuilder("spark.deploy.maxExecutorRetries")
+ .version("1.6.3")
.intConf
.createWithDefault(10)
val SPREAD_OUT_APPS = ConfigBuilder("spark.deploy.spreadOut")
+ .version("0.6.1")
.booleanConf
.createWithDefault(true)
val DEFAULT_CORES = ConfigBuilder("spark.deploy.defaultCores")
+ .version("0.9.0")
.intConf
.createWithDefault(Int.MaxValue)
diff --git a/docs/configuration.md b/docs/configuration.md
index f7b7e16..2f5b570 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -2577,22 +2577,25 @@ Spark subsystems.
### Deploy
<table class="table">
- <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.deploy.recoveryMode</code></td>
<td>NONE</td>
<td>The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches.
This is only applicable for cluster mode when running with Standalone or Mesos.</td>
+ <td>0.8.1</td>
</tr>
<tr>
<td><code>spark.deploy.zookeeper.url</code></td>
<td>None</td>
<td>When `spark.deploy.recoveryMode` is set to ZOOKEEPER, this configuration is used to set the zookeeper URL to connect to.</td>
+ <td>0.8.1</td>
</tr>
<tr>
<td><code>spark.deploy.zookeeper.dir</code></td>
<td>None</td>
<td>When `spark.deploy.recoveryMode` is set to ZOOKEEPER, this configuration is used to set the zookeeper directory to store recovery state.</td>
+ <td>0.8.1</td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 04/08: [SPARK-30888][CORE][DOC] Add version information to
the configuration of Network
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 2c5b13f33add9943248c66873ada2c2e1e5d703a
Author: beliefer <be...@163.com>
AuthorDate: Thu Feb 27 11:05:11 2020 +0900
[SPARK-30888][CORE][DOC] Add version information to the configuration of Network
### What changes were proposed in this pull request?
1.Add version information to the configuration of `Network`.
2.Update the docs of `Network`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.network.crypto.saslFallback | 2.2.0 | SPARK-19139 | 8f3f73abc1fe62496722476460c174af0250e3fe#diff-0ac65da2bc6b083fb861fe410c7688c2 |
spark.network.crypto.enabled | 2.2.0 | SPARK-19139 | 8f3f73abc1fe62496722476460c174af0250e3fe#diff-6bdad48cfc34314e89599655442ff210 |
spark.network.remoteReadNioBufferConversion | 2.4.0 | SPARK-24307 | 2c82745686f4456c4d5c84040a431dcb5b6cb60b#diff-2b643ea78c1add0381754b1f47eec132 |
spark.network.timeout | 1.3.0 | SPARK-4688 | d3f07fd23cc26a70f44c52e24445974d4885d58a#diff-1df6b5af3d8f9f16255ff8c7a06f402f |
spark.network.timeoutInterval | 1.3.2 | SPARK-5529 | ec196ab1c7569d7ab0a50c9d7338c2835f2c84d5#diff-47779b72f095f7e7f926898fa1a425ee |
spark.rpc.askTimeout | 1.4.0 | SPARK-6490 | 8136810dfad12008ac300116df7bc8448740f1ae#diff-529fc5c06b9731c1fbda6f3db60b16aa |
spark.rpc.connect.threads | 1.6.0 | SPARK-6028 | 084e4e126211d74a79e8dbd2d0e604dd3c650822#diff-0c89b4a60c30a7cd2224bb64d93da942 |
spark.rpc.io.numConnectionsPerPeer | 1.6.0 | SPARK-10745 | 34a77679877bc40b58a10ec539a8da00fed7db39#diff-0c89b4a60c30a7cd2224bb64d93da942 |
spark.rpc.io.threads | 1.6.0 | SPARK-6028 | 084e4e126211d74a79e8dbd2d0e604dd3c650822#diff-0c89b4a60c30a7cd2224bb64d93da942 |
spark.rpc.lookupTimeout | 1.4.0 | SPARK-6490 | 8136810dfad12008ac300116df7bc8448740f1ae#diff-529fc5c06b9731c1fbda6f3db60b16aa |
spark.rpc.message.maxSize | 2.0.0 | SPARK-7997 | bc1babd63da4ee56e6d371eb24805a5d714e8295#diff-529fc5c06b9731c1fbda6f3db60b16aa |
spark.rpc.netty.dispatcher.numThreads | 1.6.0 | SPARK-11079 | 1797055dbf1d2fd7714d7c65c8d2efde2f15efc1#diff-05133dfc4bfdb6a27aa092d86ce24866 |
spark.rpc.numRetries | 1.4.0 | SPARK-6490 | 8136810dfad12008ac300116df7bc8448740f1ae#diff-529fc5c06b9731c1fbda6f3db60b16aa |
spark.rpc.retry.wait | 1.4.0 | SPARK-6490 | 8136810dfad12008ac300116df7bc8448740f1ae#diff-529fc5c06b9731c1fbda6f3db60b16aa |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27674 from beliefer/add-version-to-network-config.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
.../org/apache/spark/internal/config/Network.scala | 14 ++++++++++++++
docs/configuration.md | 20 ++++++++++++++++++--
2 files changed, 32 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/Network.scala b/core/src/main/scala/org/apache/spark/internal/config/Network.scala
index 129e31a..0961d06 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/Network.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/Network.scala
@@ -23,71 +23,85 @@ private[spark] object Network {
private[spark] val NETWORK_CRYPTO_SASL_FALLBACK =
ConfigBuilder("spark.network.crypto.saslFallback")
+ .version("2.2.0")
.booleanConf
.createWithDefault(true)
private[spark] val NETWORK_CRYPTO_ENABLED =
ConfigBuilder("spark.network.crypto.enabled")
+ .version("2.2.0")
.booleanConf
.createWithDefault(false)
private[spark] val NETWORK_REMOTE_READ_NIO_BUFFER_CONVERSION =
ConfigBuilder("spark.network.remoteReadNioBufferConversion")
+ .version("2.4.0")
.booleanConf
.createWithDefault(false)
private[spark] val NETWORK_TIMEOUT =
ConfigBuilder("spark.network.timeout")
+ .version("1.3.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("120s")
private[spark] val NETWORK_TIMEOUT_INTERVAL =
ConfigBuilder("spark.network.timeoutInterval")
+ .version("1.3.2")
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString(STORAGE_BLOCKMANAGER_TIMEOUTINTERVAL.defaultValueString)
private[spark] val RPC_ASK_TIMEOUT =
ConfigBuilder("spark.rpc.askTimeout")
+ .version("1.4.0")
.stringConf
.createOptional
private[spark] val RPC_CONNECT_THREADS =
ConfigBuilder("spark.rpc.connect.threads")
+ .version("1.6.0")
.intConf
.createWithDefault(64)
private[spark] val RPC_IO_NUM_CONNECTIONS_PER_PEER =
ConfigBuilder("spark.rpc.io.numConnectionsPerPeer")
+ .version("1.6.0")
.intConf
.createWithDefault(1)
private[spark] val RPC_IO_THREADS =
ConfigBuilder("spark.rpc.io.threads")
+ .version("1.6.0")
.intConf
.createOptional
private[spark] val RPC_LOOKUP_TIMEOUT =
ConfigBuilder("spark.rpc.lookupTimeout")
+ .version("1.4.0")
.stringConf
.createOptional
private[spark] val RPC_MESSAGE_MAX_SIZE =
ConfigBuilder("spark.rpc.message.maxSize")
+ .version("2.0.0")
.intConf
.createWithDefault(128)
private[spark] val RPC_NETTY_DISPATCHER_NUM_THREADS =
ConfigBuilder("spark.rpc.netty.dispatcher.numThreads")
+ .version("1.6.0")
.intConf
.createOptional
private[spark] val RPC_NUM_RETRIES =
ConfigBuilder("spark.rpc.numRetries")
+ .version("1.4.0")
.intConf
.createWithDefault(3)
private[spark] val RPC_RETRY_WAIT =
ConfigBuilder("spark.rpc.retry.wait")
+ .version("1.4.0")
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefaultString("3s")
}
diff --git a/docs/configuration.md b/docs/configuration.md
index 88edaf6..295cb3d 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1658,7 +1658,7 @@ Apart from these, the following properties are also available, and may be useful
### Networking
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.rpc.message.maxSize</code></td>
<td>128</td>
@@ -1667,6 +1667,7 @@ Apart from these, the following properties are also available, and may be useful
output size information sent between executors and the driver. Increase this if you are running
jobs with many thousands of map and reduce tasks and see messages about the RPC message size.
</td>
+ <td>2.0.0</td>
</tr>
<tr>
<td><code>spark.blockManager.port</code></td>
@@ -1674,6 +1675,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Port for all block managers to listen on. These exist on both the driver and the executors.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.driver.blockManager.port</code></td>
@@ -1682,6 +1684,7 @@ Apart from these, the following properties are also available, and may be useful
Driver-specific port for the block manager to listen on, for cases where it cannot use the same
configuration as executors.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.driver.bindAddress</code></td>
@@ -1695,6 +1698,7 @@ Apart from these, the following properties are also available, and may be useful
the different ports used by the driver (RPC, block manager and UI) need to be forwarded from the
container's host.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.driver.host</code></td>
@@ -1703,6 +1707,7 @@ Apart from these, the following properties are also available, and may be useful
Hostname or IP address for the driver.
This is used for communicating with the executors and the standalone Master.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.driver.port</code></td>
@@ -1711,6 +1716,7 @@ Apart from these, the following properties are also available, and may be useful
Port for the driver to listen on.
This is used for communicating with the executors and the standalone Master.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.rpc.io.backLog</code></td>
@@ -1720,6 +1726,7 @@ Apart from these, the following properties are also available, and may be useful
need to be increased, so that incoming connections are not dropped when a large number of
connections arrives in a short period of time.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.network.timeout</code></td>
@@ -1731,6 +1738,7 @@ Apart from these, the following properties are also available, and may be useful
<code>spark.shuffle.io.connectionTimeout</code>, <code>spark.rpc.askTimeout</code> or
<code>spark.rpc.lookupTimeout</code> if they are not configured.
</td>
+ <td>1.3.0</td>
</tr>
<tr>
<td><code>spark.network.io.preferDirectBufs</code></td>
@@ -1740,7 +1748,8 @@ Apart from these, the following properties are also available, and may be useful
Off-heap buffers are used to reduce garbage collection during shuffle and cache
block transfer. For environments where off-heap memory is tightly limited, users may wish to
turn this off to force all allocations to be on-heap.
- </td>
+ </td>
+ <td></td>
</tr>
<tr>
<td><code>spark.port.maxRetries</code></td>
@@ -1752,6 +1761,7 @@ Apart from these, the following properties are also available, and may be useful
essentially allows it to try a range of ports from the start port specified
to port + maxRetries.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.rpc.numRetries</code></td>
@@ -1760,6 +1770,7 @@ Apart from these, the following properties are also available, and may be useful
Number of times to retry before an RPC task gives up.
An RPC task will run at most times of this number.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td><code>spark.rpc.retry.wait</code></td>
@@ -1767,6 +1778,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Duration for an RPC ask operation to wait before retrying.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td><code>spark.rpc.askTimeout</code></td>
@@ -1774,6 +1786,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Duration for an RPC ask operation to wait before timing out.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td><code>spark.rpc.lookupTimeout</code></td>
@@ -1781,6 +1794,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Duration for an RPC remote endpoint lookup operation to wait before timing out.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td><code>spark.core.connection.ack.wait.timeout</code></td>
@@ -1790,6 +1804,7 @@ Apart from these, the following properties are also available, and may be useful
out and giving up. To avoid unwilling timeout caused by long pause like GC,
you can set larger value.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.network.maxRemoteBlockSizeFetchToMem</code></td>
@@ -1801,6 +1816,7 @@ Apart from these, the following properties are also available, and may be useful
For users who enabled external shuffle service, this feature can only work when
external shuffle service is at least 2.3.0.
</td>
+ <td></td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 05/08: [SPARK-30908][CORE][DOC] Add version information to
the configuration of Kryo
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit dd31736c0ca55dfe3842a6ad920d85f4301c563f
Author: beliefer <be...@163.com>
AuthorDate: Mon Mar 2 15:14:47 2020 +0900
[SPARK-30908][CORE][DOC] Add version information to the configuration of Kryo
### What changes were proposed in this pull request?
1.Add version information to the configuration of `Kryo`.
2.Update the docs of `Kryo`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.kryo.registrationRequired | 1.1.0 | SPARK-2102 | efdaeb111917dd0314f1d00ee8524bed1e2e21ca#diff-1f81c62dad0e2dfc387a974bb08c497c |
spark.kryo.registrator | 0.5.0 | None | 91c07a33d90ab0357e8713507134ecef5c14e28a#diff-792ed56b3398163fa14e8578549d0d98 | This is not a release version, do we need to record it?
spark.kryo.classesToRegister | 1.2.0 | SPARK-1813 | 6bb56faea8d238ea22c2de33db93b1b39f492b3a#diff-529fc5c06b9731c1fbda6f3db60b16aa |
spark.kryo.unsafe | 2.1.0 | SPARK-928 | bc167a2a53f5a795d089e8a884569b1b3e2cd439#diff-1f81c62dad0e2dfc387a974bb08c497c |
spark.kryo.pool | 3.0.0 | SPARK-26466 | 38f030725c561979ca98b2a6cc7ca6c02a1f80ed#diff-a3c6b992784f9abeb9f3047d3dcf3ed9 |
spark.kryo.referenceTracking | 0.8.0 | None | 0a8cc309211c62f8824d76618705c817edcf2424#diff-1f81c62dad0e2dfc387a974bb08c497c |
spark.kryoserializer.buffer | 1.4.0 | SPARK-5932 | 2d222fb39dd978e5a33cde6ceb59307cbdf7b171#diff-1f81c62dad0e2dfc387a974bb08c497c |
spark.kryoserializer.buffer.max | 1.4.0 | SPARK-5932 | 2d222fb39dd978e5a33cde6ceb59307cbdf7b171#diff-1f81c62dad0e2dfc387a974bb08c497c |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27734 from beliefer/add-version-to-kryo-config.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
core/src/main/scala/org/apache/spark/internal/config/Kryo.scala | 8 ++++++++
docs/configuration.md | 9 ++++++++-
2 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/Kryo.scala b/core/src/main/scala/org/apache/spark/internal/config/Kryo.scala
index 717a099..646d855 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/Kryo.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/Kryo.scala
@@ -22,35 +22,43 @@ import org.apache.spark.network.util.ByteUnit
private[spark] object Kryo {
val KRYO_REGISTRATION_REQUIRED = ConfigBuilder("spark.kryo.registrationRequired")
+ .version("1.1.0")
.booleanConf
.createWithDefault(false)
val KRYO_USER_REGISTRATORS = ConfigBuilder("spark.kryo.registrator")
+ .version("0.5.0")
.stringConf
.createOptional
val KRYO_CLASSES_TO_REGISTER = ConfigBuilder("spark.kryo.classesToRegister")
+ .version("1.2.0")
.stringConf
.toSequence
.createWithDefault(Nil)
val KRYO_USE_UNSAFE = ConfigBuilder("spark.kryo.unsafe")
+ .version("2.1.0")
.booleanConf
.createWithDefault(false)
val KRYO_USE_POOL = ConfigBuilder("spark.kryo.pool")
+ .version("3.0.0")
.booleanConf
.createWithDefault(true)
val KRYO_REFERENCE_TRACKING = ConfigBuilder("spark.kryo.referenceTracking")
+ .version("0.8.0")
.booleanConf
.createWithDefault(true)
val KRYO_SERIALIZER_BUFFER_SIZE = ConfigBuilder("spark.kryoserializer.buffer")
+ .version("1.4.0")
.bytesConf(ByteUnit.KiB)
.createWithDefaultString("64k")
val KRYO_SERIALIZER_MAX_BUFFER_SIZE = ConfigBuilder("spark.kryoserializer.buffer.max")
+ .version("1.4.0")
.bytesConf(ByteUnit.MiB)
.createWithDefaultString("64m")
diff --git a/docs/configuration.md b/docs/configuration.md
index 295cb3d..c586ecf 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1204,7 +1204,7 @@ Apart from these, the following properties are also available, and may be useful
### Compression and Serialization
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.broadcast.compress</code></td>
<td>true</td>
@@ -1278,6 +1278,7 @@ Apart from these, the following properties are also available, and may be useful
with Kryo.
See the <a href="tuning.html#data-serialization">tuning guide</a> for more details.
</td>
+ <th>1.2.0</th>
</tr>
<tr>
<td><code>spark.kryo.referenceTracking</code></td>
@@ -1288,6 +1289,7 @@ Apart from these, the following properties are also available, and may be useful
copies of the same object. Can be disabled to improve performance if you know this is not the
case.
</td>
+ <th>0.8.0</th>
</tr>
<tr>
<td><code>spark.kryo.registrationRequired</code></td>
@@ -1299,6 +1301,7 @@ Apart from these, the following properties are also available, and may be useful
significant performance overhead, so enabling this option can enforce strictly that a
user has not omitted classes from registration.
</td>
+ <th>1.1.0</th>
</tr>
<tr>
<td><code>spark.kryo.registrator</code></td>
@@ -1312,6 +1315,7 @@ Apart from these, the following properties are also available, and may be useful
<code>KryoRegistrator</code></a>.
See the <a href="tuning.html#data-serialization">tuning guide</a> for more details.
</td>
+ <th>0.5.0</th>
</tr>
<tr>
<td><code>spark.kryo.unsafe</code></td>
@@ -1320,6 +1324,7 @@ Apart from these, the following properties are also available, and may be useful
Whether to use unsafe based Kryo serializer. Can be
substantially faster by using Unsafe Based IO.
</td>
+ <th>2.1.0</th>
</tr>
<tr>
<td><code>spark.kryoserializer.buffer.max</code></td>
@@ -1329,6 +1334,7 @@ Apart from these, the following properties are also available, and may be useful
This must be larger than any object you attempt to serialize and must be less than 2048m.
Increase this if you get a "buffer limit exceeded" exception inside Kryo.
</td>
+ <th>1.4.0</th>
</tr>
<tr>
<td><code>spark.kryoserializer.buffer</code></td>
@@ -1338,6 +1344,7 @@ Apart from these, the following properties are also available, and may be useful
Note that there will be one buffer <i>per core</i> on each worker. This buffer will grow up to
<code>spark.kryoserializer.buffer.max</code> if needed.
</td>
+ <th>1.4.0</th>
</tr>
<tr>
<td><code>spark.rdd.compress</code></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 02/08: [SPARK-30910][CORE][DOC] Add version information to
the configuration of R
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 42b26cdafc9f3bafe0d225724b97c8e6bdb74e8d
Author: beliefer <be...@163.com>
AuthorDate: Thu Feb 27 10:56:38 2020 +0900
[SPARK-30910][CORE][DOC] Add version information to the configuration of R
### What changes were proposed in this pull request?
1.Add version information to the configuration of `R`.
2.Update the docs of `R`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.r.backendConnectionTimeout | 2.1.0 | SPARK-17919 | 2881a2d1d1a650a91df2c6a01275eba14a43b42a#diff-025470e1b7094d7cf4a78ea353fb3981 |
spark.r.numRBackendThreads | 1.4.0 | SPARK-8282 | 28e8a6ea65fd08ab9cefc4d179d5c66ffefd3eb4#diff-697f7f2fc89808e0113efc71ed235db2 |
spark.r.heartBeatInterval | 2.1.0 | SPARK-17919 | 2881a2d1d1a650a91df2c6a01275eba14a43b42a#diff-fe903bf14db371aa320b7cc516f2463c |
spark.sparkr.r.command | 1.5.3 | SPARK-10971 | 9695f452e86a88bef3bcbd1f3c0b00ad9e9ac6e1#diff-025470e1b7094d7cf4a78ea353fb3981 |
spark.r.command | 1.5.3 | SPARK-10971 | 9695f452e86a88bef3bcbd1f3c0b00ad9e9ac6e1#diff-025470e1b7094d7cf4a78ea353fb3981 |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27708 from beliefer/add-version-to-R-config.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
core/src/main/scala/org/apache/spark/internal/config/R.scala | 5 +++++
docs/configuration.md | 8 +++++++-
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/R.scala b/core/src/main/scala/org/apache/spark/internal/config/R.scala
index 26e06a5..46fc198 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/R.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/R.scala
@@ -19,22 +19,27 @@ package org.apache.spark.internal.config
private[spark] object R {
val R_BACKEND_CONNECTION_TIMEOUT = ConfigBuilder("spark.r.backendConnectionTimeout")
+ .version("2.1.0")
.intConf
.createWithDefault(6000)
val R_NUM_BACKEND_THREADS = ConfigBuilder("spark.r.numRBackendThreads")
+ .version("1.4.0")
.intConf
.createWithDefault(2)
val R_HEARTBEAT_INTERVAL = ConfigBuilder("spark.r.heartBeatInterval")
+ .version("2.1.0")
.intConf
.createWithDefault(100)
val SPARKR_COMMAND = ConfigBuilder("spark.sparkr.r.command")
+ .version("1.5.3")
.stringConf
.createWithDefault("Rscript")
val R_COMMAND = ConfigBuilder("spark.r.command")
+ .version("1.5.3")
.stringConf
.createOptional
}
diff --git a/docs/configuration.md b/docs/configuration.md
index 2f5b570..4aff1ef 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -2513,13 +2513,14 @@ Spark subsystems.
### SparkR
<table class="table">
-<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td><code>spark.r.numRBackendThreads</code></td>
<td>2</td>
<td>
Number of threads used by RBackend to handle RPC calls from SparkR package.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td><code>spark.r.command</code></td>
@@ -2527,6 +2528,7 @@ Spark subsystems.
<td>
Executable for executing R scripts in cluster modes for both driver and workers.
</td>
+ <td>1.5.3</td>
</tr>
<tr>
<td><code>spark.r.driver.command</code></td>
@@ -2534,6 +2536,7 @@ Spark subsystems.
<td>
Executable for executing R scripts in client modes for driver. Ignored in cluster modes.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.r.shell.command</code></td>
@@ -2542,6 +2545,7 @@ Spark subsystems.
Executable for executing sparkR shell in client modes for driver. Ignored in cluster modes. It is the same as environment variable <code>SPARKR_DRIVER_R</code>, but take precedence over it.
<code>spark.r.shell.command</code> is used for sparkR shell while <code>spark.r.driver.command</code> is used for running R script.
</td>
+ <td></td>
</tr>
<tr>
<td><code>spark.r.backendConnectionTimeout</code></td>
@@ -2549,6 +2553,7 @@ Spark subsystems.
<td>
Connection timeout set by R process on its connection to RBackend in seconds.
</td>
+ <td>2.1.0</td>
</tr>
<tr>
<td><code>spark.r.heartBeatInterval</code></td>
@@ -2556,6 +2561,7 @@ Spark subsystems.
<td>
Interval for heartbeats sent from SparkR backend to R process to prevent connection timeout.
</td>
+ <td>2.1.0</td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 07/08: [SPARK-30912][CORE][DOC] Add version information to
the configuration of Streaming.scala
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 056e60fa7c7e45a59be769c189456a83760c4aef
Author: beliefer <be...@163.com>
AuthorDate: Mon Mar 2 15:16:40 2020 +0900
[SPARK-30912][CORE][DOC] Add version information to the configuration of Streaming.scala
### What changes were proposed in this pull request?
1.Add version information to the configuration of `Streaming`.
2.Update the docs of `Streaming`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.streaming.dynamicAllocation.enabled | 3.0.0 | SPARK-26941 | cad475dcc9376557f882859856286e858002389a#diff-335c3bbf4ca27cf65a6f850b1a7c89dd |
spark.streaming.dynamicAllocation.testing | 3.0.0 | SPARK-26941 | cad475dcc9376557f882859856286e858002389a#diff-335c3bbf4ca27cf65a6f850b1a7c89dd |
spark.streaming.dynamicAllocation.minExecutors | 3.0.0 | SPARK-26941 | cad475dcc9376557f882859856286e858002389a#diff-335c3bbf4ca27cf65a6f850b1a7c89dd |
spark.streaming.dynamicAllocation.maxExecutors | 3.0.0 | SPARK-26941 | cad475dcc9376557f882859856286e858002389a#diff-335c3bbf4ca27cf65a6f850b1a7c89dd |
spark.streaming.dynamicAllocation.scalingInterval | 3.0.0 | SPARK-26941 | cad475dcc9376557f882859856286e858002389a#diff-335c3bbf4ca27cf65a6f850b1a7c89dd |
spark.streaming.dynamicAllocation.scalingUpRatio | 3.0.0 | SPARK-26941 | cad475dcc9376557f882859856286e858002389a#diff-335c3bbf4ca27cf65a6f850b1a7c89dd |
spark.streaming.dynamicAllocation.scalingDownRatio | 3.0.0 | SPARK-26941 | cad475dcc9376557f882859856286e858002389a#diff-335c3bbf4ca27cf65a6f850b1a7c89dd |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27745 from beliefer/add-version-to-streaming-config.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
.../main/scala/org/apache/spark/internal/config/Streaming.scala | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/Streaming.scala b/core/src/main/scala/org/apache/spark/internal/config/Streaming.scala
index 6e58c09..44a467b 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/Streaming.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/Streaming.scala
@@ -23,16 +23,19 @@ private[spark] object Streaming {
private[spark] val STREAMING_DYN_ALLOCATION_ENABLED =
ConfigBuilder("spark.streaming.dynamicAllocation.enabled")
+ .version("3.0.0")
.booleanConf
.createWithDefault(false)
private[spark] val STREAMING_DYN_ALLOCATION_TESTING =
ConfigBuilder("spark.streaming.dynamicAllocation.testing")
+ .version("3.0.0")
.booleanConf
.createWithDefault(false)
private[spark] val STREAMING_DYN_ALLOCATION_MIN_EXECUTORS =
ConfigBuilder("spark.streaming.dynamicAllocation.minExecutors")
+ .version("3.0.0")
.intConf
.checkValue(_ > 0, "The min executor number of streaming dynamic " +
"allocation must be positive.")
@@ -40,6 +43,7 @@ private[spark] object Streaming {
private[spark] val STREAMING_DYN_ALLOCATION_MAX_EXECUTORS =
ConfigBuilder("spark.streaming.dynamicAllocation.maxExecutors")
+ .version("3.0.0")
.intConf
.checkValue(_ > 0, "The max executor number of streaming dynamic " +
"allocation must be positive.")
@@ -47,6 +51,7 @@ private[spark] object Streaming {
private[spark] val STREAMING_DYN_ALLOCATION_SCALING_INTERVAL =
ConfigBuilder("spark.streaming.dynamicAllocation.scalingInterval")
+ .version("3.0.0")
.timeConf(TimeUnit.SECONDS)
.checkValue(_ > 0, "The scaling interval of streaming dynamic " +
"allocation must be positive.")
@@ -54,6 +59,7 @@ private[spark] object Streaming {
private[spark] val STREAMING_DYN_ALLOCATION_SCALING_UP_RATIO =
ConfigBuilder("spark.streaming.dynamicAllocation.scalingUpRatio")
+ .version("3.0.0")
.doubleConf
.checkValue(_ > 0, "The scaling up ratio of streaming dynamic " +
"allocation must be positive.")
@@ -61,6 +67,7 @@ private[spark] object Streaming {
private[spark] val STREAMING_DYN_ALLOCATION_SCALING_DOWN_RATIO =
ConfigBuilder("spark.streaming.dynamicAllocation.scalingDownRatio")
+ .version("3.0.0")
.doubleConf
.checkValue(_ > 0, "The scaling down ratio of streaming dynamic " +
"allocation must be positive.")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 08/08: [SPARK-30914][CORE][DOC] Add version information to
the configuration of UI
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit dd97bcbda5be70ff6d3e7192bb7782092e80483c
Author: beliefer <be...@163.com>
AuthorDate: Fri Mar 6 11:08:57 2020 +0900
[SPARK-30914][CORE][DOC] Add version information to the configuration of UI
### What changes were proposed in this pull request?
1.Add version information to the configuration of `UI`.
2.Update the docs of `UI`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.ui.showConsoleProgress | 1.2.1 | SPARK-4017 | 04b1bdbae31c3039125100e703121daf7d9dabf5#diff-364713d7776956cb8b0a771e9b62f82d |
spark.ui.consoleProgress.update.interval | 2.1.0 | SPARK-16919 | e076fb05ac83a3ed6995e29bb03ea07ea05e39db#diff-fbf4e388a66b6a37e984b91cd71a3e2c |
spark.ui.enabled | 1.1.1 | SPARK-3490 | 937de93e80e6d299c4d08be426da2d5bc2d66f98#diff-364713d7776956cb8b0a771e9b62f82d |
spark.ui.port | 0.7.0 | None | f03d9760fd8ac67fd0865cb355ba75d2eff507fe#diff-ed8dbcebe16fda5ecd6df1a981dc6fee |
spark.ui.filters | 1.0.0 | SPARK-1189 | 7edbea41b43e0dc11a2de156be220db8b7952d01#diff-f79a5ead735b3d0b34b6b94486918e1c |
spark.ui.allowFramingFrom | 1.6.0 | SPARK-10589 | 5dbaf3d3911bbfa003bc75459aaad66b4f6e0c67#diff-f79a5ead735b3d0b34b6b94486918e1c |
spark.ui.reverseProxy | 2.1.0 | SPARK-15487 | 92ce8d4849a0341c4636e70821b7be57ad3055b1#diff-364713d7776956cb8b0a771e9b62f82d |
spark.ui.reverseProxyUrl | 2.1.0 | SPARK-15487 | 92ce8d4849a0341c4636e70821b7be57ad3055b1#diff-364713d7776956cb8b0a771e9b62f82d |
spark.ui.killEnabled | 1.0.0 | SPARK-1202 | 211f97447b5f078afcb1619a08d2e2349325f61a#diff-a40023c80383451b6e29ee7a6e0593e9 |
spark.ui.threadDumpsEnabled | 1.2.0 | SPARK-611 | 866c7bbe56f9c7fd96d3f4afe8a76405dc877a6e#diff-5d18fb70c572369a0fff0b97de94f265 |
spark.ui.prometheus.enabled | 3.0.0 | SPARK-29064 | bbfaadb280a80b511a98d18881641c6d9851dd51#diff-f70174ad0759db1fb4cb36a7ff9324a7 |
spark.ui.xXssProtection | 2.3.0 | SPARK-22188 | 5a07aca4d464e96d75ea17bf6768e24b829872ec#diff-6bdad48cfc34314e89599655442ff210 |
spark.ui.xContentTypeOptions.enabled | 2.3.0 | SPARK-22188 | 5a07aca4d464e96d75ea17bf6768e24b829872ec#diff-6bdad48cfc34314e89599655442ff210 |
spark.ui.strictTransportSecurity | 2.3.0 | SPARK-22188 | 5a07aca4d464e96d75ea17bf6768e24b829872ec#diff-6bdad48cfc34314e89599655442ff210 |
spark.ui.requestHeaderSize | 2.2.3 | SPARK-26118 | 9ceee6f188e6c3794d31ce15cc61d29f907bebf7#diff-6bdad48cfc34314e89599655442ff210 |
spark.ui.timeline.tasks.maximum | 1.4.0 | SPARK-7296 | a5f7b3b9c7f05598a1cc8e582e5facee1029cd5e#diff-fa4cfb2cce1b925f55f41f2dfa8c8501 |
spark.acls.enable | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe6571decfdc406ec6d505fd92f9f2b85a618c#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.ui.view.acls | 1.0.0 | SPARK-1189 | 7edbea41b43e0dc11a2de156be220db8b7952d01#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.ui.view.acls.groups | 2.0.0 | SPARK-4224 | ae79032dcf160796851ca29116cca146c4d86ada#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.admin.acls | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe6571decfdc406ec6d505fd92f9f2b85a618c#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.admin.acls.groups | 2.0.0 | SPARK-4224 | ae79032dcf160796851ca29116cca146c4d86ada#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.modify.acls | 1.1.0 | SPARK-1890 and SPARK-1891 | e3fe6571decfdc406ec6d505fd92f9f2b85a618c#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.modify.acls.groups | 2.0.0 | SPARK-4224 | ae79032dcf160796851ca29116cca146c4d86ada#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.user.groups.mapping | 2.0.0 | SPARK-4224 | ae79032dcf160796851ca29116cca146c4d86ada#diff-afd88f677ec5ff8b5e96a5cbbe00cd98 |
spark.ui.proxyRedirectUri | 3.0.0 | SPARK-30240 | a9fbd310300e57ed58818d7347f3c3172701c491#diff-f70174ad0759db1fb4cb36a7ff9324a7 |
spark.ui.custom.executor.log.url | 3.0.0 | SPARK-26792 | d5bda2c9e8dde6afc075cc7f65b15fa9aa82231c#diff-f70174ad0759db1fb4cb36a7ff9324a7 |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27806 from beliefer/add-version-to-UI-config.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
.../org/apache/spark/internal/config/UI.scala | 26 ++++++++++++++++++++++
docs/configuration.md | 10 +++++++++
2 files changed, 36 insertions(+)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/UI.scala b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
index 60d9857..231eecf 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/UI.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
@@ -25,31 +25,37 @@ private[spark] object UI {
val UI_SHOW_CONSOLE_PROGRESS = ConfigBuilder("spark.ui.showConsoleProgress")
.doc("When true, show the progress bar in the console.")
+ .version("1.2.1")
.booleanConf
.createWithDefault(false)
val UI_CONSOLE_PROGRESS_UPDATE_INTERVAL =
ConfigBuilder("spark.ui.consoleProgress.update.interval")
+ .version("2.1.0")
.timeConf(TimeUnit.MILLISECONDS)
.createWithDefault(200)
val UI_ENABLED = ConfigBuilder("spark.ui.enabled")
.doc("Whether to run the web UI for the Spark application.")
+ .version("1.1.1")
.booleanConf
.createWithDefault(true)
val UI_PORT = ConfigBuilder("spark.ui.port")
.doc("Port for your application's dashboard, which shows memory and workload data.")
+ .version("0.7.0")
.intConf
.createWithDefault(4040)
val UI_FILTERS = ConfigBuilder("spark.ui.filters")
.doc("Comma separated list of filter class names to apply to the Spark Web UI.")
+ .version("1.0.0")
.stringConf
.toSequence
.createWithDefault(Nil)
val UI_ALLOW_FRAMING_FROM = ConfigBuilder("spark.ui.allowFramingFrom")
+ .version("1.6.0")
.stringConf
.createOptional
@@ -61,6 +67,7 @@ private[spark] object UI {
"through spark master/proxy public URL. This setting affects all the workers and " +
"application UIs running in the cluster and must be set on all the workers, drivers " +
" and masters.")
+ .version("2.1.0")
.booleanConf
.createWithDefault(false)
@@ -69,15 +76,18 @@ private[spark] object UI {
"in front of Spark Master. This is useful when running proxy for authentication e.g. " +
"OAuth proxy. Make sure this is a complete URL including scheme (http/https) and port to " +
"reach your proxy.")
+ .version("2.1.0")
.stringConf
.createOptional
val UI_KILL_ENABLED = ConfigBuilder("spark.ui.killEnabled")
.doc("Allows jobs and stages to be killed from the web UI.")
+ .version("1.0.0")
.booleanConf
.createWithDefault(true)
val UI_THREAD_DUMPS_ENABLED = ConfigBuilder("spark.ui.threadDumpsEnabled")
+ .version("1.2.0")
.booleanConf
.createWithDefault(true)
@@ -85,73 +95,88 @@ private[spark] object UI {
.internal()
.doc("Expose executor metrics at /metrics/executors/prometheus. " +
"For master/worker/driver metrics, you need to configure `conf/metrics.properties`.")
+ .version("3.0.0")
.booleanConf
.createWithDefault(false)
val UI_X_XSS_PROTECTION = ConfigBuilder("spark.ui.xXssProtection")
.doc("Value for HTTP X-XSS-Protection response header")
+ .version("2.3.0")
.stringConf
.createWithDefaultString("1; mode=block")
val UI_X_CONTENT_TYPE_OPTIONS = ConfigBuilder("spark.ui.xContentTypeOptions.enabled")
.doc("Set to 'true' for setting X-Content-Type-Options HTTP response header to 'nosniff'")
+ .version("2.3.0")
.booleanConf
.createWithDefault(true)
val UI_STRICT_TRANSPORT_SECURITY = ConfigBuilder("spark.ui.strictTransportSecurity")
.doc("Value for HTTP Strict Transport Security Response Header")
+ .version("2.3.0")
.stringConf
.createOptional
val UI_REQUEST_HEADER_SIZE = ConfigBuilder("spark.ui.requestHeaderSize")
.doc("Value for HTTP request header size in bytes.")
+ .version("2.2.3")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("8k")
val UI_TIMELINE_TASKS_MAXIMUM = ConfigBuilder("spark.ui.timeline.tasks.maximum")
+ .version("1.4.0")
.intConf
.createWithDefault(1000)
val ACLS_ENABLE = ConfigBuilder("spark.acls.enable")
+ .version("1.1.0")
.booleanConf
.createWithDefault(false)
val UI_VIEW_ACLS = ConfigBuilder("spark.ui.view.acls")
+ .version("1.0.0")
.stringConf
.toSequence
.createWithDefault(Nil)
val UI_VIEW_ACLS_GROUPS = ConfigBuilder("spark.ui.view.acls.groups")
+ .version("2.0.0")
.stringConf
.toSequence
.createWithDefault(Nil)
val ADMIN_ACLS = ConfigBuilder("spark.admin.acls")
+ .version("1.1.0")
.stringConf
.toSequence
.createWithDefault(Nil)
val ADMIN_ACLS_GROUPS = ConfigBuilder("spark.admin.acls.groups")
+ .version("2.0.0")
.stringConf
.toSequence
.createWithDefault(Nil)
val MODIFY_ACLS = ConfigBuilder("spark.modify.acls")
+ .version("1.1.0")
.stringConf
.toSequence
.createWithDefault(Nil)
val MODIFY_ACLS_GROUPS = ConfigBuilder("spark.modify.acls.groups")
+ .version("2.0.0")
.stringConf
.toSequence
.createWithDefault(Nil)
val USER_GROUPS_MAPPING = ConfigBuilder("spark.user.groups.mapping")
+ .version("2.0.0")
.stringConf
.createWithDefault("org.apache.spark.security.ShellBasedGroupsMappingProvider")
val PROXY_REDIRECT_URI = ConfigBuilder("spark.ui.proxyRedirectUri")
.doc("Proxy address to use when responding with HTTP redirects.")
+ .version("3.0.0")
.stringConf
.createOptional
@@ -163,6 +188,7 @@ private[spark] object UI {
"This configuration replaces original log urls in event log, which will be also effective " +
"when accessing the application on history server. The new log urls must be permanent, " +
"otherwise you might have dead link for executor log urls.")
+ .version("3.0.0")
.stringConf
.createOptional
}
diff --git a/docs/configuration.md b/docs/configuration.md
index c586ecf..9cbe341 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1031,6 +1031,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Whether to run the web UI for the Spark application.
</td>
+ <td>1.1.1</td>
</tr>
<tr>
<td><code>spark.ui.killEnabled</code></td>
@@ -1038,6 +1039,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Allows jobs and stages to be killed from the web UI.
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td><code>spark.ui.liveUpdate.period</code></td>
@@ -1062,6 +1064,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Port for your application's dashboard, which shows memory and workload data.
</td>
+ <td>0.7.0</td>
</tr>
<tr>
<td><code>spark.ui.retainedJobs</code></td>
@@ -1093,6 +1096,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
Enable running Spark Master as reverse proxy for worker and application UIs. In this mode, Spark master will reverse proxy the worker and application UIs to enable access without requiring direct access to their hosts. Use it with caution, as worker and application UI will not be accessible directly, you will only be able to access them through spark master/proxy public URL. This setting affects all the workers and application UIs running in the cluster and must be set on all the wor [...]
</td>
+ <td>2.1.0</td>
</tr>
<tr>
<td><code>spark.ui.reverseProxyUrl</code></td>
@@ -1100,6 +1104,7 @@ Apart from these, the following properties are also available, and may be useful
<td>
This is the URL where your proxy is running. This URL is for proxy which is running in front of Spark Master. This is useful when running proxy for authentication e.g. OAuth proxy. Make sure this is a complete URL including scheme (http/https) and port to reach your proxy.
</td>
+ <td>2.1.0</td>
</tr>
<tr>
<td><code>spark.ui.proxyRedirectUri</code></td>
@@ -1112,6 +1117,7 @@ Apart from these, the following properties are also available, and may be useful
<code>X-Forwarded-Context</code> request header), or by setting the proxy base in the Spark
app's configuration.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.ui.showConsoleProgress</code></td>
@@ -1123,6 +1129,7 @@ Apart from these, the following properties are also available, and may be useful
<br/>
<em>Note:</em> In shell environment, the default value of spark.ui.showConsoleProgress is true.
</td>
+ <td>1.2.1</td>
</tr>
<tr>
<td><code>spark.ui.custom.executor.log.url</code></td>
@@ -1138,6 +1145,7 @@ Apart from these, the following properties are also available, and may be useful
<p/>
For now, only YARN mode supports this configuration
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td><code>spark.worker.ui.retainedExecutors</code></td>
@@ -1190,6 +1198,7 @@ Apart from these, the following properties are also available, and may be useful
<br /><code>spark.com.test.filter1.param.name1=foo</code>
<br /><code>spark.com.test.filter1.param.name2=bar</code>
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td><code>spark.ui.requestHeaderSize</code></td>
@@ -1198,6 +1207,7 @@ Apart from these, the following properties are also available, and may be useful
The maximum allowed size for a HTTP request header, in bytes unless otherwise specified.
This setting applies for the Spark History Server too.
</td>
+ <td>2.2.3</td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[spark] 06/08: [SPARK-30891][CORE][DOC] Add version information to
the configuration of History
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 231c0bbe61c07eb237fa4e13457acd88e0038794
Author: beliefer <be...@163.com>
AuthorDate: Mon Mar 2 15:15:49 2020 +0900
[SPARK-30891][CORE][DOC] Add version information to the configuration of History
### What changes were proposed in this pull request?
1.Add version information to the configuration of `History`.
2.Update the docs of `History`.
I sorted out some information show below.
Item name | Since version | JIRA ID | Commit ID | Note
-- | -- | -- | -- | --
spark.history.fs.logDirectory | 1.1.0 | SPARK-1768 | 21ddd7d1e9f8e2a726427f32422c31706a20ba3f#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.safemodeCheck.interval | 1.6.0 | SPARK-11020 | cf04fdfe71abc395163a625cc1f99ec5e54cc07e#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.update.interval | 1.4.0 | SPARK-6046 | 4527761bcd6501c362baf2780905a0018b9a74ba#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.enabled | 1.3.0 | SPARK-3562 | 8942b522d8a3269a2a357e3a274ed4b3e66ebdde#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e | Branch branch-1.3 does not exist, exists in branch-1.4, but it is 1.3.0-SNAPSHOT in pom.xml
spark.history.fs.cleaner.interval | 1.4.0 | SPARK-5933 | 1991337336596f94698e79c2366f065c374128ab#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.maxAge | 1.4.0 | SPARK-5933 | 1991337336596f94698e79c2366f065c374128ab#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.cleaner.maxNum | 3.0.0 | SPARK-28294 | bbc2be4f425c4c26450e1bf21db407e81046ce21#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.store.path | 2.3.0 | SPARK-20642 | 74daf622de4e534d5a5929b424a6e836850eefad#diff-19f35f981fdc5b0a46f070b879a9a9fc |
spark.history.store.maxDiskUsage | 2.3.0 | SPARK-20654 | 8b497046c647a21bbed1bdfbdcb176745a1d5cd5#diff-19f35f981fdc5b0a46f070b879a9a9fc |
spark.history.ui.port | 1.0.0 | SPARK-1276 | 9ae80bf9bd3e4da7443af97b41fe26aa5d35d70b#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.fs.inProgressOptimization.enabled | 2.4.0 | SPARK-6951 | 653fe02415a537299e15f92b56045569864b6183#diff-19f35f981fdc5b0a46f070b879a9a9fc |
spark.history.fs.endEventReparseChunkSize | 2.4.0 | SPARK-6951 | 653fe02415a537299e15f92b56045569864b6183#diff-19f35f981fdc5b0a46f070b879a9a9fc |
spark.history.fs.eventLog.rolling.maxFilesToRetain | 3.0.0 | SPARK-30481 | a2fe73b83c0e7c61d1c83b236565a71e3d005a71#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.fs.eventLog.rolling.compaction.score.threshold | 3.0.0 | SPARK-30481 | a2fe73b83c0e7c61d1c83b236565a71e3d005a71#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.fs.driverlog.cleaner.enabled | 3.0.0 | SPARK-25118 | 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.fs.driverlog.cleaner.interval | 3.0.0 | SPARK-25118 | 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.fs.driverlog.cleaner.maxAge | 3.0.0 | SPARK-25118 | 5f11e8c4cb9a5db037ac239b8fcc97f3a746e772#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.ui.acls.enable | 1.0.1 | Spark 1489 | c8dd13221215275948b1a6913192d40e0c8cbadd#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.ui.admin.acls | 2.1.1 | SPARK-19033 | 4ca1788805e4a0131ba8f0ccb7499ee0e0242837#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.ui.admin.acls.groups | 2.1.1 | SPARK-19033 | 4ca1788805e4a0131ba8f0ccb7499ee0e0242837#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.fs.numReplayThreads | 2.0.0 | SPARK-13988 | 6fdd0e32a6c3fdce1f3f7e1f8d252af05c419f7b#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.retainedApplications | 1.0.0 | SPARK-1276 | 9ae80bf9bd3e4da7443af97b41fe26aa5d35d70b#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.provider | 1.1.0 | SPARK-1768 | 21ddd7d1e9f8e2a726427f32422c31706a20ba3f#diff-a7befb99e7bd7e3ab5c46c2568aa5b3e |
spark.history.kerberos.enabled | 1.0.1 | Spark-1490 | 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.kerberos.principal | 1.0.1 | Spark-1490 | 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.kerberos.keytab | 1.0.1 | Spark-1490 | 866b03ef4d27b2160563b58d577de29ba6eb4442#diff-b49b5b9c31ddb36a9061004b5b723058 |
spark.history.custom.executor.log.url | 3.0.0 | SPARK-26311 | ae5b2a6a92be4986ef5b8062d7fb59318cff6430#diff-6bddeb5e25239974fc13db66266b167b |
spark.history.custom.executor.log.url.applyIncompleteApplication | 3.0.0 | SPARK-26311 | ae5b2a6a92be4986ef5b8062d7fb59318cff6430#diff-6bddeb5e25239974fc13db66266b167b |
### Why are the changes needed?
Supplemental configuration version information.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
Exists UT
Closes #27751 from beliefer/add-version-to-history-config.
Authored-by: beliefer <be...@163.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
.../org/apache/spark/internal/config/History.scala | 28 ++++++++++++++++++++++
docs/monitoring.md | 26 +++++++++++++++++++-
2 files changed, 53 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/History.scala b/core/src/main/scala/org/apache/spark/internal/config/History.scala
index 8f99908..581777d 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/History.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/History.scala
@@ -26,46 +26,56 @@ private[spark] object History {
val DEFAULT_LOG_DIR = "file:/tmp/spark-events"
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
+ .version("1.1.0")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
+ .version("1.6.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
+ .version("1.4.0")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")
val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
+ .version("1.4.0")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("7d")
val MAX_LOG_NUM = ConfigBuilder("spark.history.fs.cleaner.maxNum")
.doc("The maximum number of log files in the event log directory.")
+ .version("3.0.0")
.intConf
.createWithDefault(Int.MaxValue)
val LOCAL_STORE_DIR = ConfigBuilder("spark.history.store.path")
.doc("Local directory where to cache application history information. By default this is " +
"not set, meaning all history information will be kept in memory.")
+ .version("2.3.0")
.stringConf
.createOptional
val MAX_LOCAL_DISK_USAGE = ConfigBuilder("spark.history.store.maxDiskUsage")
+ .version("2.3.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10g")
val HISTORY_SERVER_UI_PORT = ConfigBuilder("spark.history.ui.port")
.doc("Web UI port to bind Spark History Server")
+ .version("1.0.0")
.intConf
.createWithDefault(18080)
@@ -73,6 +83,7 @@ private[spark] object History {
ConfigBuilder("spark.history.fs.inProgressOptimization.enabled")
.doc("Enable optimized handling of in-progress logs. This option may leave finished " +
"applications that fail to rename their event logs listed as in-progress.")
+ .version("2.4.0")
.booleanConf
.createWithDefault(true)
@@ -81,6 +92,7 @@ private[spark] object History {
.doc("How many bytes to parse at the end of log files looking for the end event. " +
"This is used to speed up generation of application listings by skipping unnecessary " +
"parts of event log files. It can be disabled by setting this config to 0.")
+ .version("2.4.0")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("1m")
@@ -90,6 +102,7 @@ private[spark] object History {
"By default, all event log files will be retained. Please set the configuration " +
s"and ${EVENT_LOG_ROLLING_MAX_FILE_SIZE.key} accordingly if you want to control " +
"the overall size of event log files.")
+ .version("3.0.0")
.intConf
.checkValue(_ > 0, "Max event log files to retain should be higher than 0.")
.createWithDefault(Integer.MAX_VALUE)
@@ -99,54 +112,67 @@ private[spark] object History {
.doc("The threshold score to determine whether it's good to do the compaction or not. " +
"The compaction score is calculated in analyzing, and being compared to this value. " +
"Compaction will proceed only when the score is higher than the threshold value.")
+ .version("3.0.0")
.internal()
.doubleConf
.createWithDefault(0.7d)
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
+ .version("3.0.0")
.fallbackConf(CLEANER_ENABLED)
val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
.fallbackConf(CLEANER_INTERVAL_S)
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
+ .version("3.0.0")
.fallbackConf(MAX_LOG_AGE_S)
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
+ .version("2.1.1")
.stringConf
.toSequence
.createWithDefault(Nil)
val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
+ .version("2.0.0")
.intConf
.createWithDefaultFunction(() => Math.ceil(Runtime.getRuntime.availableProcessors() / 4f).toInt)
val RETAINED_APPLICATIONS = ConfigBuilder("spark.history.retainedApplications")
+ .version("1.0.0")
.intConf
.createWithDefault(50)
val PROVIDER = ConfigBuilder("spark.history.provider")
+ .version("1.1.0")
.stringConf
.createOptional
val KERBEROS_ENABLED = ConfigBuilder("spark.history.kerberos.enabled")
+ .version("1.0.1")
.booleanConf
.createWithDefault(false)
val KERBEROS_PRINCIPAL = ConfigBuilder("spark.history.kerberos.principal")
+ .version("1.0.1")
.stringConf
.createOptional
val KERBEROS_KEYTAB = ConfigBuilder("spark.history.kerberos.keytab")
+ .version("1.0.1")
.stringConf
.createOptional
@@ -156,6 +182,7 @@ private[spark] object History {
"some path variables via patterns which can vary on cluster manager. Please check the " +
"documentation for your cluster manager to see which patterns are supported, if any. " +
"This configuration has no effect on a live application, it only affects the history server.")
+ .version("3.0.0")
.stringConf
.createOptional
@@ -165,6 +192,7 @@ private[spark] object History {
s"${CUSTOM_EXECUTOR_LOG_URL.key}, to incomplete application as well. " +
"Even if this is true, this still only affects the behavior of the history server, " +
"not running spark applications.")
+ .version("3.0.0")
.booleanConf
.createWithDefault(true)
}
diff --git a/docs/monitoring.md b/docs/monitoring.md
index 131cd2a..b72fd02 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -143,13 +143,14 @@ Security options for the Spark History Server are covered more detail in the
[Security](security.html#web-ui) page.
<table class="table">
- <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
+ <tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since Version</th></tr>
<tr>
<td>spark.history.provider</td>
<td><code>org.apache.spark.deploy.history.FsHistoryProvider</code></td>
<td>Name of the class implementing the application history backend. Currently there is only
one implementation, provided by Spark, which looks for application logs stored in the
file system.</td>
+ <td>1.1.0</td>
</tr>
<tr>
<td>spark.history.fs.logDirectory</td>
@@ -160,6 +161,7 @@ Security options for the Spark History Server are covered more detail in the
an HDFS path <code>hdfs://namenode/shared/spark-logs</code>
or that of an alternative filesystem supported by the Hadoop APIs.
</td>
+ <td>1.1.0</td>
</tr>
<tr>
<td>spark.history.fs.update.interval</td>
@@ -171,6 +173,7 @@ Security options for the Spark History Server are covered more detail in the
As soon as an update has completed, listings of the completed and incomplete applications
will reflect the changes.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td>spark.history.retainedApplications</td>
@@ -180,6 +183,7 @@ Security options for the Spark History Server are covered more detail in the
the oldest applications will be removed from the cache. If an application is not in the cache,
it will have to be loaded from disk if it is accessed from the UI.
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td>spark.history.ui.maxApplications</td>
@@ -188,6 +192,7 @@ Security options for the Spark History Server are covered more detail in the
The number of applications to display on the history summary page. Application UIs are still
available by accessing their URLs directly even if they are not displayed on the history summary page.
</td>
+ <td>2.0.1</td>
</tr>
<tr>
<td>spark.history.ui.port</td>
@@ -195,6 +200,7 @@ Security options for the Spark History Server are covered more detail in the
<td>
The port to which the web interface of the history server binds.
</td>
+ <td>1.0.0</td>
</tr>
<tr>
<td>spark.history.kerberos.enabled</td>
@@ -203,6 +209,7 @@ Security options for the Spark History Server are covered more detail in the
Indicates whether the history server should use kerberos to login. This is required
if the history server is accessing HDFS files on a secure Hadoop cluster.
</td>
+ <td>1.0.1</td>
</tr>
<tr>
<td>spark.history.kerberos.principal</td>
@@ -210,6 +217,7 @@ Security options for the Spark History Server are covered more detail in the
<td>
When <code>spark.history.kerberos.enabled=true</code>, specifies kerberos principal name for the History Server.
</td>
+ <td>1.0.1</td>
</tr>
<tr>
<td>spark.history.kerberos.keytab</td>
@@ -217,6 +225,7 @@ Security options for the Spark History Server are covered more detail in the
<td>
When <code>spark.history.kerberos.enabled=true</code>, specifies location of the kerberos keytab file for the History Server.
</td>
+ <td>1.0.1</td>
</tr>
<tr>
<td>spark.history.fs.cleaner.enabled</td>
@@ -224,6 +233,7 @@ Security options for the Spark History Server are covered more detail in the
<td>
Specifies whether the History Server should periodically clean up event logs from storage.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td>spark.history.fs.cleaner.interval</td>
@@ -236,6 +246,7 @@ Security options for the Spark History Server are covered more detail in the
<code>spark.history.fs.cleaner.maxNum</code>, Spark tries to clean up the completed attempts
from the applications based on the order of their oldest attempt time.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td>spark.history.fs.cleaner.maxAge</td>
@@ -243,6 +254,7 @@ Security options for the Spark History Server are covered more detail in the
<td>
When <code>spark.history.fs.cleaner.enabled=true</code>, job history files older than this will be deleted when the filesystem history cleaner runs.
</td>
+ <td>1.4.0</td>
</tr>
<tr>
<td>spark.history.fs.cleaner.maxNum</td>
@@ -253,6 +265,7 @@ Security options for the Spark History Server are covered more detail in the
This should be smaller than the underlying file system limit like
`dfs.namenode.fs-limits.max-directory-items` in HDFS.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td>spark.history.fs.endEventReparseChunkSize</td>
@@ -262,6 +275,7 @@ Security options for the Spark History Server are covered more detail in the
This is used to speed up generation of application listings by skipping unnecessary
parts of event log files. It can be disabled by setting this config to 0.
</td>
+ <td>2.4.0</td>
</tr>
<tr>
<td>spark.history.fs.inProgressOptimization.enabled</td>
@@ -270,6 +284,7 @@ Security options for the Spark History Server are covered more detail in the
Enable optimized handling of in-progress logs. This option may leave finished
applications that fail to rename their event logs listed as in-progress.
</td>
+ <td>2.4.0</td>
</tr>
<tr>
<td>spark.history.fs.driverlog.cleaner.enabled</td>
@@ -277,6 +292,7 @@ Security options for the Spark History Server are covered more detail in the
<td>
Specifies whether the History Server should periodically clean up driver logs from storage.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td>spark.history.fs.driverlog.cleaner.interval</td>
@@ -285,6 +301,7 @@ Security options for the Spark History Server are covered more detail in the
When <code>spark.history.fs.driverlog.cleaner.enabled=true</code>, specifies how often the filesystem driver log cleaner checks for files to delete.
Files are only deleted if they are older than <code>spark.history.fs.driverlog.cleaner.maxAge</code>
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td>spark.history.fs.driverlog.cleaner.maxAge</td>
@@ -292,6 +309,7 @@ Security options for the Spark History Server are covered more detail in the
<td>
When <code>spark.history.fs.driverlog.cleaner.enabled=true</code>, driver log files older than this will be deleted when the driver log cleaner runs.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td>spark.history.fs.numReplayThreads</td>
@@ -299,6 +317,7 @@ Security options for the Spark History Server are covered more detail in the
<td>
Number of threads that will be used by history server to process event logs.
</td>
+ <td>2.0.0</td>
</tr>
<tr>
<td>spark.history.store.maxDiskUsage</td>
@@ -307,6 +326,7 @@ Security options for the Spark History Server are covered more detail in the
Maximum disk usage for the local directory where the cache application history information
are stored.
</td>
+ <td>2.3.0</td>
</tr>
<tr>
<td>spark.history.store.path</td>
@@ -316,6 +336,7 @@ Security options for the Spark History Server are covered more detail in the
server will store application data on disk instead of keeping it in memory. The data
written to disk will be re-used in the event of a history server restart.
</td>
+ <td>2.3.0</td>
</tr>
<tr>
<td>spark.history.custom.executor.log.url</td>
@@ -329,6 +350,7 @@ Security options for the Spark History Server are covered more detail in the
<p/>
For now, only YARN mode supports this configuration
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td>spark.history.custom.executor.log.url.applyIncompleteApplication</td>
@@ -339,6 +361,7 @@ Security options for the Spark History Server are covered more detail in the
Please note that incomplete applications may include applications which didn't shutdown gracefully.
Even this is set to `true`, this configuration has no effect on a live application, it only affects the history server.
</td>
+ <td>3.0.0</td>
</tr>
<tr>
<td>spark.history.fs.eventLog.rolling.maxFilesToRetain</td>
@@ -348,6 +371,7 @@ Security options for the Spark History Server are covered more detail in the
all event log files will be retained. The lowest value is 1 for technical reason.<br/>
Please read the section of "Applying compaction of old event log files" for more details.
</td>
+ <td>3.0.0</td>
</tr>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org