You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2025/03/04 05:11:12 UTC
(spark) branch master updated: [SPARK-51375][SQL][CONNECT] Suppress `SparkConnect(Execution|Session)Manager.periodicMaintenance` log messages
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d5a17546f9e0 [SPARK-51375][SQL][CONNECT] Suppress `SparkConnect(Execution|Session)Manager.periodicMaintenance` log messages
d5a17546f9e0 is described below
commit d5a17546f9e06a8cebdc8c5e65904efd81318bb6
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon Mar 3 21:11:02 2025 -0800
[SPARK-51375][SQL][CONNECT] Suppress `SparkConnect(Execution|Session)Manager.periodicMaintenance` log messages
### What changes were proposed in this pull request?
This PR aims to suppress `SparkConnect(Execution|Session)Manager.periodicMaintenance` methods' start and end log messages by lowering the log level from `INFO` to `DEBUG` from Apache Spark 4.0.0.
### Why are the changes needed?
From Apache Spark 4.0.0, we have a `SparkConnect`-on binary distribution.
- https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc2-bin/spark-4.0.0-bin-hadoop3-spark-connect.tgz
While testing Apache Spark 4.0.0 RC2, I found that these are too verbose in the production environments because it shows four info messages at every `30s` without giving important information.
```
25/03/03 23:27:45 INFO SparkConnectSessionManager: Started periodic run of SparkConnectSessionManager maintenance.
25/03/03 23:27:45 INFO SparkConnectSessionManager: Finished periodic run of SparkConnectSessionManager maintenance.
25/03/03 23:27:46 INFO SparkConnectExecutionManager: Started periodic run of SparkConnectExecutionManager maintenance.
25/03/03 23:27:46 INFO SparkConnectExecutionManager: Finished periodic run of SparkConnectExecutionManager maintenance.
25/03/03 23:28:15 INFO SparkConnectSessionManager: Started periodic run of SparkConnectSessionManager maintenance.
25/03/03 23:28:15 INFO SparkConnectSessionManager: Finished periodic run of SparkConnectSessionManager maintenance.
25/03/03 23:28:16 INFO SparkConnectExecutionManager: Started periodic run of SparkConnectExecutionManager maintenance.
25/03/03 23:28:16 INFO SparkConnectExecutionManager: Finished periodic run of SparkConnectExecutionManager maintenance.
```
Since Apache Spark will leave log messages for the actual clean-up operations like the following, we had better lower these start and end messages to `DEBUG` levels.
https://github.com/apache/spark/blob/4b2ae4864104b5e56a6422a119d60b2e1e35f357/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala#L260-L262
https://github.com/apache/spark/blob/4b2ae4864104b5e56a6422a119d60b2e1e35f357/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala#L253-L255
### Does this PR introduce _any_ user-facing change?
No behavior change because this is only a log-message level change.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50139 from dongjoon-hyun/SPARK-51375.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../spark/sql/connect/service/SparkConnectExecutionManager.scala | 4 ++--
.../apache/spark/sql/connect/service/SparkConnectSessionManager.scala | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala
index a156be189c65..c158de48574f 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectExecutionManager.scala
@@ -249,7 +249,7 @@ private[connect] class SparkConnectExecutionManager() extends Logging {
// Visible for testing.
private[connect] def periodicMaintenance(timeoutNs: Long): Unit = {
// Find any detached executions that expired and should be removed.
- logInfo("Started periodic run of SparkConnectExecutionManager maintenance.")
+ logDebug("Started periodic run of SparkConnectExecutionManager maintenance.")
val nowNs = System.nanoTime()
executions.forEach((_, executeHolder) => {
@@ -266,7 +266,7 @@ private[connect] class SparkConnectExecutionManager() extends Logging {
}
})
- logInfo("Finished periodic run of SparkConnectExecutionManager maintenance.")
+ logDebug("Finished periodic run of SparkConnectExecutionManager maintenance.")
}
// For testing.
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala
index 8581bb7b98f0..d8b3e21e7ad6 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectSessionManager.scala
@@ -234,7 +234,7 @@ class SparkConnectSessionManager extends Logging {
defaultInactiveTimeoutMs: Long,
ignoreCustomTimeout: Boolean): Unit = {
// Find any sessions that expired and should be removed.
- logInfo("Started periodic run of SparkConnectSessionManager maintenance.")
+ logDebug("Started periodic run of SparkConnectSessionManager maintenance.")
def shouldExpire(info: SessionHolderInfo, nowMs: Long): Boolean = {
val timeoutMs = if (info.customInactiveTimeoutMs.isDefined && !ignoreCustomTimeout) {
@@ -262,7 +262,7 @@ class SparkConnectSessionManager extends Logging {
}
})
- logInfo("Finished periodic run of SparkConnectSessionManager maintenance.")
+ logDebug("Finished periodic run of SparkConnectSessionManager maintenance.")
}
private def newIsolatedSession(): SparkSession = {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org