You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "John Zhuge (Jira)" <ji...@apache.org> on 2026/08/27 18:42:00 UTC
[jira] [Updated] (SPARK-59059) Support per-logger log level configuration with static Spark conf
[ https://issues.apache.org/jira/browse/SPARK-59059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John Zhuge updated SPARK-59059:
-------------------------------
Description:
SPARK-43782 added `spark.log.level`, which sets the level of the *root* logger at startup, as a static-conf equivalent of `SparkContext.setLogLevel()`. There is still no static-conf way to set the level of an individual logger.
Today, turning on DEBUG for one class means shipping a `log4j2.properties` and pointing both the driver and the executors at it:
{code:java}
spark-submit \
--files /tmp/log4j2.properties#log4j2.properties \
--conf spark.driver.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
--conf spark.executor.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
...
{code}
That is awkward in cluster mode and impossible for a Spark Connect client, which has no `SparkContext` and often cannot change the log4j configuration of the platform it connects to, the same motivation that was accepted for SPARK-43782.
Proposal: support `spark.log.level.<logger>`, applied on the driver and on the executors after the root level from `spark.log.level`, so it takes precedence for the named logger only.
{code:java}
--conf spark.log.level=WARN
--conf spark.log.level.org.apache.spark.sql.execution=DEBUG
{code}
Note: SPARK-16784 asked for a version of this and was bulk-closed as Incomplete in 2021 without a fix.
was:
SPARK-43782 added `spark.log.level`, which sets the level of the *root* logger at startup, as a static-conf equivalent of `SparkContext.setLogLevel()`. There is still no static-conf way to set the level of an individual logger.
Today, turning on DEBUG for one class means shipping a `log4j2.properties` and pointing both the driver and the executors at it:
{code:java}
spark-submit \
--files /tmp/log4j2.properties#log4j2.properties \
--conf spark.driver.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
--conf spark.executor.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
...
{code}
That is awkward in cluster mode and impossible for a Spark Connect client, which has no `SparkContext` and often cannot change the log4j configuration of the platform it connects to – the same motivation that was accepted for SPARK-43782.
Proposal: support `spark.log.level.<logger>`, applied on the driver and on the executors after the root level from `spark.log.level`, so it takes precedence for the named logger only.
{code:java}
--conf spark.log.level=WARN
--conf spark.log.level.org.apache.spark.sql.execution=DEBUG
{code}
Note: SPARK-16784 asked for a version of this and was bulk-closed as Incomplete in 2021 without a fix.
> Support per-logger log level configuration with static Spark conf
> -----------------------------------------------------------------
>
> Key: SPARK-59059
> URL: https://issues.apache.org/jira/browse/SPARK-59059
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.2.0
> Reporter: John Zhuge
> Priority: Trivial
>
> SPARK-43782 added `spark.log.level`, which sets the level of the *root* logger at startup, as a static-conf equivalent of `SparkContext.setLogLevel()`. There is still no static-conf way to set the level of an individual logger.
> Today, turning on DEBUG for one class means shipping a `log4j2.properties` and pointing both the driver and the executors at it:
> {code:java}
> spark-submit \
> --files /tmp/log4j2.properties#log4j2.properties \
> --conf spark.driver.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
> --conf spark.executor.extraJavaOptions=-Dlog4j.configurationFile=log4j2.properties \
> ...
> {code}
> That is awkward in cluster mode and impossible for a Spark Connect client, which has no `SparkContext` and often cannot change the log4j configuration of the platform it connects to, the same motivation that was accepted for SPARK-43782.
> Proposal: support `spark.log.level.<logger>`, applied on the driver and on the executors after the root level from `spark.log.level`, so it takes precedence for the named logger only.
> {code:java}
> --conf spark.log.level=WARN
> --conf spark.log.level.org.apache.spark.sql.execution=DEBUG
> {code}
>
> Note: SPARK-16784 asked for a version of this and was bulk-closed as Incomplete in 2021 without a fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org