You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/05/30 10:55:01 UTC

[spark] branch master updated: [SPARK-39322][CORE][FOLLOWUP] Revise log messages for dynamic allocation and shuffle decommission

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new be2c8c01158 [SPARK-39322][CORE][FOLLOWUP] Revise log messages for dynamic allocation and shuffle decommission
be2c8c01158 is described below

commit be2c8c0115861e6975b658a7b0455bae828b7553
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon May 30 03:54:37 2022 -0700

    [SPARK-39322][CORE][FOLLOWUP] Revise log messages for dynamic allocation and shuffle decommission
    
    ### What changes were proposed in this pull request?
    
    This PR is a follow-up for #36705 to revise the missed log message change.
    
    ### Why are the changes needed?
    
    Like the documentation, this PR updates the log message correspondingly.
    - Lower log level to `INFO` from `WARN`
    - Provide a specific message according to the configurations.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a log-message-only change.
    
    ### How was this patch tested?
    
    Pass the CIs.
    
    Closes #36725 from dongjoon-hyun/SPARK-39322-2.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 .../scala/org/apache/spark/ExecutorAllocationManager.scala   | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
index c4b619300b5..b9bc3c63ff4 100644
--- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
+++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
@@ -204,13 +204,11 @@ private[spark] class ExecutorAllocationManager(
         s"s${DYN_ALLOCATION_SUSTAINED_SCHEDULER_BACKLOG_TIMEOUT.key} must be > 0!")
     }
     if (!conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
-      // If dynamic allocation shuffle tracking or worker decommissioning along with
-      // storage shuffle decommissioning is enabled we have *experimental* support for
-      // decommissioning without a shuffle service.
-      if (conf.get(config.DYN_ALLOCATION_SHUFFLE_TRACKING_ENABLED) ||
-          (decommissionEnabled &&
-            conf.get(config.STORAGE_DECOMMISSION_SHUFFLE_BLOCKS_ENABLED))) {
-        logWarning("Dynamic allocation without a shuffle service is an experimental feature.")
+      if (conf.get(config.DYN_ALLOCATION_SHUFFLE_TRACKING_ENABLED)) {
+        logInfo("Dynamic allocation is enabled without a shuffle service.")
+      } else if (decommissionEnabled &&
+          conf.get(config.STORAGE_DECOMMISSION_SHUFFLE_BLOCKS_ENABLED)) {
+        logInfo("Shuffle data decommission is enabled without a shuffle service.")
       } else if (!testing) {
         throw new SparkException("Dynamic allocation of executors requires the external " +
           "shuffle service. You may enable this through spark.shuffle.service.enabled.")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org