You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ya...@apache.org on 2024/05/30 09:16:46 UTC
(spark) branch master updated: [SPARK-48471][CORE] Improve documentation and usage guide for history server
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new df15c8d7744b [SPARK-48471][CORE] Improve documentation and usage guide for history server
df15c8d7744b is described below
commit df15c8d7744becfd44cd4a447c362e8e007bd574
Author: Kent Yao <ya...@apache.org>
AuthorDate: Thu May 30 17:16:31 2024 +0800
[SPARK-48471][CORE] Improve documentation and usage guide for history server
### What changes were proposed in this pull request?
In this PR, we improve documentation and usage guide for the history server by:
- Identify and print **unrecognized options** specified by users
- Obtain and print all history server-related configurations dynamically instead of using an incomplete, outdated hardcoded list.
- Ensure all configurations are documented for the usage guide
### Why are the changes needed?
- Revise the help guide for the history server to make it more user-friendly. Missing configuration in the help guide is not always reachable in our official documentation. E.g. spark.history.fs.safemodeCheck.interval is still missing from the doc since added in 1.6.
- Missusage shall be reported to users
### Does this PR introduce _any_ user-facing change?
No, the print style is still AS-IS with items increased
### How was this patch tested?
#### without this pr
```
Usage: ./sbin/start-history-server.sh [options]
24/05/30 15:37:23 INFO SignalUtils: Registering signal handler for TERM
24/05/30 15:37:23 INFO SignalUtils: Registering signal handler for HUP
24/05/30 15:37:23 INFO SignalUtils: Registering signal handler for INT
Options:
--properties-file FILE Path to a custom Spark properties file.
Default is conf/spark-defaults.conf.
Configuration options can be set by setting the corresponding JVM system property.
History Server options are always available; additional options depend on the provider.
History Server options:
spark.history.ui.port Port where server will listen for connections
(default 18080)
spark.history.acls.enable Whether to enable view acls for all applications
(default false)
spark.history.provider Name of history provider class (defaults to
file system-based provider)
spark.history.retainedApplications Max number of application UIs to keep loaded in memory
(default 50)
FsHistoryProvider options:
spark.history.fs.logDirectory Directory where app logs are stored
(default: file:/tmp/spark-events)
spark.history.fs.update.interval How often to reload log data from storage
(in seconds, default: 10)
```
#### For error
```java
Unrecognized options: --conf spark.history.ui.port=10000
Usage: HistoryServer [options]
Options:
--properties-file FILE Path to a custom Spark properties file.
Default is conf/spark-defaults.conf.
```
#### For help
```java
sbin/start-history-server.sh --help
Usage: ./sbin/start-history-server.sh [options]
{"ts":"2024-05-30T07:15:29.740Z","level":"INFO","msg":"Registering signal handler for TERM","context":{"signal":"TERM"},"logger":"SignalUtils"}
{"ts":"2024-05-30T07:15:29.741Z","level":"INFO","msg":"Registering signal handler for HUP","context":{"signal":"HUP"},"logger":"SignalUtils"}
{"ts":"2024-05-30T07:15:29.741Z","level":"INFO","msg":"Registering signal handler for INT","context":{"signal":"INT"},"logger":"SignalUtils"}
Options:
--properties-file FILE Path to a custom Spark properties file.
Default is conf/spark-defaults.conf.
Configuration options can be set by setting the corresponding JVM system property.
History Server options are always available; additional options depend on the provider.
History Server options:
spark.history.custom.executor.log.url Specifies custom spark executor log url for supporting
external log service instead of using cluster managers'
application log urls in the history server. Spark will
support some path variables via patterns which can vary on
cluster manager. Please check the documentation for your
cluster manager to see which patterns are supported, if any.
This configuration has no effect on a live application, it
only affects the history server.
(Default: <undefined>)
spark.history.custom.executor.log.url.applyIncompleteApplication Whether to apply custom executor log url, as specified by
spark.history.custom.executor.log.url, to incomplete
application as well. Even if this is true, this still only
affects the behavior of the history server, not running
spark applications.
(Default: true)
spark.history.kerberos.enabled Indicates whether the history server should use kerberos to
login. This is required if the history server is accessing
HDFS files on a secure Hadoop cluster.
(Default: false)
spark.history.kerberos.keytab When spark.history.kerberos.enabled=true, specifies location
of the kerberos keytab file for the History Server.
(Default: <undefined>)
spark.history.kerberos.principal When spark.history.kerberos.enabled=true, specifies kerberos
principal name for the History Server.
(Default: <undefined>)
spark.history.provider Name of the class implementing the application history
backend.
(Default: org.apache.spark.deploy.history.FsHistoryProvider)
spark.history.retainedApplications The number of applications to retain UI data for in the
cache. If this cap is exceeded, then the oldest applications
will be removed from the cache. If an application is not in
the cache, it will have to be loaded from disk if it is
accessed from the UI.
(Default: 50)
spark.history.store.hybridStore.diskBackend Specifies a disk-based store used in hybrid store; ROCKSDB
or LEVELDB (deprecated).
(Default: ROCKSDB)
spark.history.store.hybridStore.enabled Whether to use HybridStore as the store when parsing event
logs. HybridStore will first write data to an in-memory
store and having a background thread that dumps data to a
disk store after the writing to in-memory store is
completed.
(Default: false)
spark.history.store.hybridStore.maxMemoryUsage Maximum memory space that can be used to create HybridStore.
The HybridStore co-uses the heap memory, so the heap memory
should be increased through the memory option for SHS if the
HybridStore is enabled.
(Default: 2g)
spark.history.store.maxDiskUsage Maximum disk usage for the local directory where the cache
application history information are stored.
(Default: 10g)
spark.history.store.path Local directory where to cache application history
information. By default this is not set, meaning all history
information will be kept in memory.
(Default: <undefined>)
spark.history.store.serializer Serializer for writing/reading in-memory UI objects to/from
disk-based KV Store; JSON or PROTOBUF. JSON serializer is
the only choice before Spark 3.4.0, thus it is the default
value. PROTOBUF serializer is fast and compact, and it is
the default serializer for disk-based KV store of live UI.
(Default: JSON)
spark.history.ui.acls.enable Specifies whether ACLs should be checked to authorize users
viewing the applications in the history server. If enabled,
access control checks are performed regardless of what the
individual applications had set for spark.ui.acls.enable.
The application owner will always have authorization to view
their own application and any users specified via
spark.ui.view.acls and groups specified via
spark.ui.view.acls.groups when the application was run will
also have authorization to view that application. If
disabled, no access control checks are made for any
application UIs available through the history server.
(Default: false)
spark.history.ui.admin.acls Comma separated list of users that have view access to all
the Spark applications in history server.
(Default: )
spark.history.ui.admin.acls.groups Comma separated list of groups that have view access to all
the Spark applications in history server.
(Default: )
spark.history.ui.port Web UI port to bind Spark History Server
(Default: 18080)
FsHistoryProvider options:
spark.history.fs.cleaner.enabled Whether the History Server should periodically clean up
event logs from storage
(Default: false)
spark.history.fs.cleaner.interval When spark.history.fs.cleaner.enabled=true, specifies how
often the filesystem job history cleaner checks for files to
delete.
(Default: 1d)
spark.history.fs.cleaner.maxAge When spark.history.fs.cleaner.enabled=true, history files
older than this will be deleted when the filesystem history
cleaner runs.
(Default: 7d)
```
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #46802 from yaooqinn/SPARK-48471.
Authored-by: Kent Yao <ya...@apache.org>
Signed-off-by: Kent Yao <ya...@apache.org>
---
.../spark/deploy/history/HistoryServer.scala | 1 -
.../deploy/history/HistoryServerArguments.scala | 88 +++++++++++++---------
.../org/apache/spark/internal/config/History.scala | 50 ++++++++++--
.../history/HistoryServerArgumentsSuite.scala | 13 ++++
4 files changed, 110 insertions(+), 42 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala b/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
index cad1797590a7..6e559dc4492e 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala
@@ -302,7 +302,6 @@ object HistoryServer extends Logging {
val securityManager = createSecurityManager(conf)
val providerName = conf.get(History.PROVIDER)
- .getOrElse(classOf[FsHistoryProvider].getName())
val provider = Utils.classForName[ApplicationHistoryProvider](providerName)
.getConstructor(classOf[SparkConf])
.newInstance(conf)
diff --git a/core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala b/core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala
index 01cc59e1d2e6..2fdf7a473a29 100644
--- a/core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/history/HistoryServerArguments.scala
@@ -21,6 +21,7 @@ import scala.annotation.tailrec
import org.apache.spark.SparkConf
import org.apache.spark.internal.Logging
+import org.apache.spark.internal.config.{ConfigEntry, History}
import org.apache.spark.util.Utils
/**
@@ -44,47 +45,62 @@ private[history] class HistoryServerArguments(conf: SparkConf, args: Array[Strin
case Nil =>
- case _ =>
- printUsageAndExit(1)
+ case other =>
+ val errorMsg = s"Unrecognized options: ${other.mkString(" ")}\n"
+ printUsageAndExit(1, errorMsg)
}
}
- // This mutates the SparkConf, so all accesses to it must be made after this line
- Utils.loadDefaultSparkProperties(conf, propertiesFile)
+ // This mutates the SparkConf, so all accesses to it must be made after this line
+ Utils.loadDefaultSparkProperties(conf, propertiesFile)
- private def printUsageAndExit(exitCode: Int): Unit = {
- // scalastyle:off println
- System.err.println(
- """
- |Usage: HistoryServer [options]
- |
- |Options:
- | --properties-file FILE Path to a custom Spark properties file.
- | Default is conf/spark-defaults.conf.
- |
- |Configuration options can be set by setting the corresponding JVM system property.
- |History Server options are always available; additional options depend on the provider.
- |
- |History Server options:
- |
- | spark.history.ui.port Port where server will listen for connections
- | (default 18080)
- | spark.history.acls.enable Whether to enable view acls for all applications
- | (default false)
- | spark.history.provider Name of history provider class (defaults to
- | file system-based provider)
- | spark.history.retainedApplications Max number of application UIs to keep loaded in memory
- | (default 50)
- |FsHistoryProvider options:
- |
- | spark.history.fs.logDirectory Directory where app logs are stored
- | (default: file:/tmp/spark-events)
- | spark.history.fs.update.interval How often to reload log data from storage
- | (in seconds, default: 10)
- |""".stripMargin)
- // scalastyle:on println
+ // scalastyle:off line.size.limit println
+ private def printUsageAndExit(exitCode: Int, error: String = ""): Unit = {
+ val configs = History.getClass.getDeclaredFields
+ .filter(f => classOf[ConfigEntry[_]].isAssignableFrom(f.getType))
+ .map { f =>
+ f.setAccessible(true)
+ f.get(History).asInstanceOf[ConfigEntry[_]]
+ }
+ val maxConfigLength = configs.map(_.key.length).max
+ val sb = new StringBuilder(
+ s"""
+ |${error}Usage: HistoryServer [options]
+ |
+ |Options:
+ | ${"--properties-file FILE".padTo(maxConfigLength, ' ')} Path to a custom Spark properties file.
+ | ${"".padTo(maxConfigLength, ' ')} Default is conf/spark-defaults.conf.
+ |
+ |Configuration options can be set by setting the corresponding JVM system property.
+ |History Server options are always available; additional options depend on the provider.
+ |
+ |""".stripMargin)
+
+ def printConfigs(configs: Array[ConfigEntry[_]]): Unit = {
+ configs.sortBy(_.key).foreach { conf =>
+ sb.append(" ").append(conf.key.padTo(maxConfigLength, ' '))
+ var currentDocLen = 0
+ val intention = "\n" + " " * (maxConfigLength + 2)
+ conf.doc.split("\\s+").foreach { word =>
+ if (currentDocLen + word.length > 60) {
+ sb.append(intention).append(" ").append(word)
+ currentDocLen = word.length + 1
+ } else {
+ sb.append(" ").append(word)
+ currentDocLen += word.length + 1
+ }
+ }
+ sb.append(intention).append(" (Default: ").append(conf.defaultValueString).append(")\n")
+ }
+ }
+ val (common, fs) = configs.partition(!_.key.startsWith("spark.history.fs."))
+ sb.append("History Server options:\n")
+ printConfigs(common)
+ sb.append("FsHistoryProvider options:\n")
+ printConfigs(fs)
+ System.err.println(sb.toString())
+ // scalastyle:on line.size.limit println
System.exit(exitCode)
}
-
}
diff --git a/core/src/main/scala/org/apache/spark/internal/config/History.scala b/core/src/main/scala/org/apache/spark/internal/config/History.scala
index 2306856f9331..64a8681ca295 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/History.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/History.scala
@@ -28,16 +28,19 @@ private[spark] object History {
val HISTORY_LOG_DIR = ConfigBuilder("spark.history.fs.logDirectory")
.version("1.1.0")
+ .doc("Directory where app logs are stored")
.stringConf
.createWithDefault(DEFAULT_LOG_DIR)
val SAFEMODE_CHECK_INTERVAL_S = ConfigBuilder("spark.history.fs.safemodeCheck.interval")
.version("1.6.0")
+ .doc("Interval between HDFS safemode checks for the event log directory")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("5s")
val UPDATE_INTERVAL_S = ConfigBuilder("spark.history.fs.update.interval")
.version("1.4.0")
+ .doc("How often(in seconds) to reload log data from storage")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("10s")
@@ -53,16 +56,21 @@ private[spark] object History {
val CLEANER_ENABLED = ConfigBuilder("spark.history.fs.cleaner.enabled")
.version("1.4.0")
+ .doc("Whether the History Server should periodically clean up event logs from storage")
.booleanConf
.createWithDefault(false)
val CLEANER_INTERVAL_S = ConfigBuilder("spark.history.fs.cleaner.interval")
.version("1.4.0")
+ .doc("When spark.history.fs.cleaner.enabled=true, specifies how often the filesystem " +
+ "job history cleaner checks for files to delete.")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("1d")
val MAX_LOG_AGE_S = ConfigBuilder("spark.history.fs.cleaner.maxAge")
.version("1.4.0")
+ .doc("When spark.history.fs.cleaner.enabled=true, history files older than this will be " +
+ "deleted when the filesystem history cleaner runs.")
.timeConf(TimeUnit.SECONDS)
.createWithDefaultString("7d")
@@ -96,6 +104,8 @@ private[spark] object History {
val MAX_LOCAL_DISK_USAGE = ConfigBuilder("spark.history.store.maxDiskUsage")
.version("2.3.0")
+ .doc("Maximum disk usage for the local directory where the cache application history " +
+ "information are stored.")
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10g")
@@ -145,60 +155,90 @@ private[spark] object History {
val DRIVER_LOG_CLEANER_ENABLED = ConfigBuilder("spark.history.fs.driverlog.cleaner.enabled")
.version("3.0.0")
+ .doc("Specifies whether the History Server should periodically clean up driver logs from " +
+ "storage.")
.fallbackConf(CLEANER_ENABLED)
- val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
- .version("3.0.0")
- .fallbackConf(CLEANER_INTERVAL_S)
-
val MAX_DRIVER_LOG_AGE_S = ConfigBuilder("spark.history.fs.driverlog.cleaner.maxAge")
.version("3.0.0")
+ .doc(s"When ${DRIVER_LOG_CLEANER_ENABLED.key}=true, driver log files older than this will be " +
+ s"deleted when the driver log cleaner runs.")
.fallbackConf(MAX_LOG_AGE_S)
+ val DRIVER_LOG_CLEANER_INTERVAL = ConfigBuilder("spark.history.fs.driverlog.cleaner.interval")
+ .version("3.0.0")
+ .doc(s" When ${DRIVER_LOG_CLEANER_ENABLED.key}=true, specifies how often the filesystem " +
+ s"driver log cleaner checks for files to delete. Files are only deleted if they are older " +
+ s"than ${MAX_DRIVER_LOG_AGE_S.key}.")
+ .fallbackConf(CLEANER_INTERVAL_S)
+
val HISTORY_SERVER_UI_ACLS_ENABLE = ConfigBuilder("spark.history.ui.acls.enable")
.version("1.0.1")
+ .doc("Specifies whether ACLs should be checked to authorize users viewing the applications " +
+ "in the history server. If enabled, access control checks are performed regardless of " +
+ "what the individual applications had set for spark.ui.acls.enable. The application owner " +
+ "will always have authorization to view their own application and any users specified via " +
+ "spark.ui.view.acls and groups specified via spark.ui.view.acls.groups when the " +
+ "application was run will also have authorization to view that application. If disabled, " +
+ "no access control checks are made for any application UIs available through the history " +
+ "server.")
.booleanConf
.createWithDefault(false)
val HISTORY_SERVER_UI_ADMIN_ACLS = ConfigBuilder("spark.history.ui.admin.acls")
.version("2.1.1")
+ .doc("Comma separated list of users that have view access to all the Spark applications in " +
+ "history server.")
.stringConf
.toSequence
.createWithDefault(Nil)
val HISTORY_SERVER_UI_ADMIN_ACLS_GROUPS = ConfigBuilder("spark.history.ui.admin.acls.groups")
.version("2.1.1")
+ .doc("Comma separated list of groups that have view access to all the Spark applications " +
+ "in history server.")
.stringConf
.toSequence
.createWithDefault(Nil)
val NUM_REPLAY_THREADS = ConfigBuilder("spark.history.fs.numReplayThreads")
.version("2.0.0")
+ .doc("Number of threads that will be used by history server to process event logs.")
.intConf
.createWithDefaultFunction(() => Math.ceil(Runtime.getRuntime.availableProcessors() / 4f).toInt)
val RETAINED_APPLICATIONS = ConfigBuilder("spark.history.retainedApplications")
.version("1.0.0")
+ .doc("The number of applications to retain UI data for in the cache. If this cap is " +
+ "exceeded, then the oldest applications will be removed from the cache. If an application " +
+ "is not in the cache, it will have to be loaded from disk if it is accessed from the UI.")
.intConf
.createWithDefault(50)
val PROVIDER = ConfigBuilder("spark.history.provider")
.version("1.1.0")
+ .doc("Name of the class implementing the application history backend.")
.stringConf
- .createOptional
+ .createWithDefault("org.apache.spark.deploy.history.FsHistoryProvider")
val KERBEROS_ENABLED = ConfigBuilder("spark.history.kerberos.enabled")
.version("1.0.1")
+ .doc("Indicates whether the history server should use kerberos to login. This is required " +
+ "if the history server is accessing HDFS files on a secure Hadoop cluster.")
.booleanConf
.createWithDefault(false)
val KERBEROS_PRINCIPAL = ConfigBuilder("spark.history.kerberos.principal")
.version("1.0.1")
+ .doc(s"When ${KERBEROS_ENABLED.key}=true, specifies kerberos principal name for " +
+ s" the History Server.")
.stringConf
.createOptional
val KERBEROS_KEYTAB = ConfigBuilder("spark.history.kerberos.keytab")
.version("1.0.1")
+ .doc(s"When ${KERBEROS_ENABLED.key}=true, specifies location of the kerberos keytab file " +
+ s"for the History Server.")
.stringConf
.createOptional
diff --git a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerArgumentsSuite.scala b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerArgumentsSuite.scala
index 5903ae71ec66..2b9b110a4142 100644
--- a/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerArgumentsSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerArgumentsSuite.scala
@@ -22,6 +22,7 @@ import java.nio.charset.StandardCharsets._
import com.google.common.io.Files
import org.apache.spark._
+import org.apache.spark.internal.config.{ConfigEntry, History}
import org.apache.spark.internal.config.History._
import org.apache.spark.internal.config.Tests._
@@ -52,4 +53,16 @@ class HistoryServerArgumentsSuite extends SparkFunSuite {
assert(conf.get("spark.test.CustomPropertyB") === "notblah")
}
}
+
+ test("SPARK-48471: all history configurations should have documentations") {
+ val configs = History.getClass.getDeclaredFields
+ .filter(f => classOf[ConfigEntry[_]].isAssignableFrom(f.getType))
+ .map { f =>
+ f.setAccessible(true)
+ f.get(History).asInstanceOf[ConfigEntry[_]]
+ }
+ configs.foreach { config =>
+ assert(config.doc.nonEmpty, s"Config ${config.key} doesn't have documentation")
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org