You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2020/02/17 11:46:40 UTC
[spark] branch branch-3.0 updated: [SPARK-30827][DOCS] Document
direct relationship among configurations in "spark.history.*" namespace
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new cea5cbc [SPARK-30827][DOCS] Document direct relationship among configurations in "spark.history.*" namespace
cea5cbc is described below
commit cea5cbcc88d8e0109aa0967eee9fda157526d583
Author: Jungtaek Lim (HeartSaVioR) <ka...@gmail.com>
AuthorDate: Mon Feb 17 20:45:24 2020 +0900
[SPARK-30827][DOCS] Document direct relationship among configurations in "spark.history.*" namespace
### What changes were proposed in this pull request?
This patch adds direct relationship among configurations under "spark.history" namespace.
### Why are the changes needed?
Refer the discussion thread: https://lists.apache.org/thread.html/r43c4e57cace116aca1f0f099e8a577cf202859e3671a04077867b84a%40%3Cdev.spark.apache.org%3E
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Locally ran jekyll and confirmed. Screenshots for the modified spots:
<img width="1159" alt="Screen Shot 2020-02-15 at 8 20 14 PM" src="https://user-images.githubusercontent.com/1317309/74587003-d5922b00-5030-11ea-954b-ee37fc08470a.png">
<img width="1158" alt="Screen Shot 2020-02-15 at 8 20 44 PM" src="https://user-images.githubusercontent.com/1317309/74587005-d62ac180-5030-11ea-98fc-98b1c9d83ff4.png">
<img width="1149" alt="Screen Shot 2020-02-15 at 8 19 56 PM" src="https://user-images.githubusercontent.com/1317309/74587002-d1660d80-5030-11ea-84b5-dec3d7f5c97c.png">
Closes #27575 from HeartSaVioR/SPARK-30827.
Authored-by: Jungtaek Lim (HeartSaVioR) <ka...@gmail.com>
Signed-off-by: HyukjinKwon <gu...@apache.org>
(cherry picked from commit 5445fe92887d04513d77af0890db572266136220)
Signed-off-by: HyukjinKwon <gu...@apache.org>
---
.../org/apache/spark/internal/config/History.scala | 2 +-
docs/monitoring.md | 18 ++++++++----------
2 files changed, 9 insertions(+), 11 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/History.scala b/core/src/main/scala/org/apache/spark/internal/config/History.scala
index 14fb5ff..8f99908 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/History.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/History.scala
@@ -162,7 +162,7 @@ private[spark] object History {
val APPLY_CUSTOM_EXECUTOR_LOG_URL_TO_INCOMPLETE_APP =
ConfigBuilder("spark.history.custom.executor.log.url.applyIncompleteApplication")
.doc("Whether to apply custom executor log url, as specified by " +
- "`spark.history.custom.executor.log.url`, to incomplete application as well. " +
+ s"${CUSTOM_EXECUTOR_LOG_URL.key}, to incomplete application as well. " +
"Even if this is true, this still only affects the behavior of the history server, " +
"not running spark applications.")
.booleanConf
diff --git a/docs/monitoring.md b/docs/monitoring.md
index fc95f6a..3061c67 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -159,23 +159,21 @@ Security options for the Spark History Server are covered more detail in the
<td>false</td>
<td>
Indicates whether the history server should use kerberos to login. This is required
- if the history server is accessing HDFS files on a secure Hadoop cluster. If this is
- true, it uses the configs <code>spark.history.kerberos.principal</code> and
- <code>spark.history.kerberos.keytab</code>.
+ if the history server is accessing HDFS files on a secure Hadoop cluster.
</td>
</tr>
<tr>
<td>spark.history.kerberos.principal</td>
<td>(none)</td>
<td>
- Kerberos principal name for the History Server.
+ When <code>spark.history.kerberos.enabled=true</code>, specifies kerberos principal name for the History Server.
</td>
</tr>
<tr>
<td>spark.history.kerberos.keytab</td>
<td>(none)</td>
<td>
- Location of the kerberos keytab file for the History Server.
+ When <code>spark.history.kerberos.enabled=true</code>, specifies location of the kerberos keytab file for the History Server.
</td>
</tr>
<tr>
@@ -189,7 +187,7 @@ Security options for the Spark History Server are covered more detail in the
<td>spark.history.fs.cleaner.interval</td>
<td>1d</td>
<td>
- How often the filesystem job history cleaner checks for files to delete.
+ When <code>spark.history.fs.cleaner.enabled=true</code>, specifies how often the filesystem job history cleaner checks for files to delete.
Files are deleted if at least one of two conditions holds.
First, they're deleted if they're older than <code>spark.history.fs.cleaner.maxAge</code>.
They are also deleted if the number of files is more than
@@ -201,14 +199,14 @@ Security options for the Spark History Server are covered more detail in the
<td>spark.history.fs.cleaner.maxAge</td>
<td>7d</td>
<td>
- Job history files older than this will be deleted when the filesystem history cleaner runs.
+ When <code>spark.history.fs.cleaner.enabled=true</code>, job history files older than this will be deleted when the filesystem history cleaner runs.
</td>
</tr>
<tr>
<td>spark.history.fs.cleaner.maxNum</td>
<td>Int.MaxValue</td>
<td>
- The maximum number of files in the event log directory.
+ When <code>spark.history.fs.cleaner.enabled=true</code>, specifies the maximum number of files in the event log directory.
Spark tries to clean up the completed attempt logs to maintain the log directory under this limit.
This should be smaller than the underlying file system limit like
`dfs.namenode.fs-limits.max-directory-items` in HDFS.
@@ -242,7 +240,7 @@ Security options for the Spark History Server are covered more detail in the
<td>spark.history.fs.driverlog.cleaner.interval</td>
<td><code>spark.history.fs.cleaner.interval</code></td>
<td>
- How often the filesystem driver log cleaner checks for files to delete.
+ When <code>spark.history.fs.driverlog.cleaner.enabled=true</code>, specifies how often the filesystem driver log cleaner checks for files to delete.
Files are only deleted if they are older than <code>spark.history.fs.driverlog.cleaner.maxAge</code>
</td>
</tr>
@@ -250,7 +248,7 @@ Security options for the Spark History Server are covered more detail in the
<td>spark.history.fs.driverlog.cleaner.maxAge</td>
<td><code>spark.history.fs.cleaner.maxAge</code></td>
<td>
- Driver log files older than this will be deleted when the driver log cleaner runs.
+ When <code>spark.history.fs.driverlog.cleaner.enabled=true</code>, driver log files older than this will be deleted when the driver log cleaner runs.
</td>
</tr>
<tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org