[spark] branch master updated: [SPARK-44683][SS] Logging level isn't passed to RocksDB state store provider correctly

[ka...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 630b1777904 [SPARK-44683][SS] Logging level isn't passed to RocksDB state store provider correctly
630b1777904 is described below

commit 630b1777904f15c7ac05c3cd61c0006cd692bc93
Author: Siying Dong <si...@databricks.com>
AuthorDate: Tue Aug 8 11:11:56 2023 +0900

    [SPARK-44683][SS] Logging level isn't passed to RocksDB state store provider correctly
    
    ### What changes were proposed in this pull request?
    The logging level is passed into RocksDB in a correct way.
    
    ### Why are the changes needed?
    We pass log4j's log level to RocksDB so that RocksDB debug log can go to log4j. However, we pass in log level after we create the logger. However, the way it is set isn't effective. This has two impacts: (1) setting DEBUG level don't make RocksDB generate DEBUG level logs; (2) setting WARN or ERROR level does prevent INFO level logging, but RocksDB still makes JNI calls to Scala, which is an unnecessary overhead.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Manually change the log level and observe the log lines in unit tests.
    
    Closes #42354 from siying/rocks_log_level.
    
    Authored-by: Siying Dong <si...@databricks.com>
    Signed-off-by: Jungtaek Lim <ka...@gmail.com>
---
 .../org/apache/spark/sql/execution/streaming/state/RocksDB.scala     | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
index d4366fe732b..a2868df9411 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
@@ -611,8 +611,11 @@ class RocksDB(
     if (log.isWarnEnabled) dbLogLevel = InfoLogLevel.WARN_LEVEL
     if (log.isInfoEnabled) dbLogLevel = InfoLogLevel.INFO_LEVEL
     if (log.isDebugEnabled) dbLogLevel = InfoLogLevel.DEBUG_LEVEL
-    dbOptions.setLogger(dbLogger)
+    dbLogger.setInfoLogLevel(dbLogLevel)
+    // The log level set in dbLogger is effective and the one to dbOptions isn't applied to
+    // customized logger. We still set it as it might show up in RocksDB config file or logging.
     dbOptions.setInfoLogLevel(dbLogLevel)
+    dbOptions.setLogger(dbLogger)
     logInfo(s"Set RocksDB native logging level to $dbLogLevel")
     dbLogger
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org