You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/05/30 10:55:01 UTC
[spark] branch master updated: [SPARK-39322][CORE][FOLLOWUP] Revise log messages for dynamic allocation and shuffle decommission
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new be2c8c01158 [SPARK-39322][CORE][FOLLOWUP] Revise log messages for dynamic allocation and shuffle decommission
be2c8c01158 is described below
commit be2c8c0115861e6975b658a7b0455bae828b7553
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon May 30 03:54:37 2022 -0700
[SPARK-39322][CORE][FOLLOWUP] Revise log messages for dynamic allocation and shuffle decommission
### What changes were proposed in this pull request?
This PR is a follow-up for #36705 to revise the missed log message change.
### Why are the changes needed?
Like the documentation, this PR updates the log message correspondingly.
- Lower log level to `INFO` from `WARN`
- Provide a specific message according to the configurations.
### Does this PR introduce _any_ user-facing change?
No. This is a log-message-only change.
### How was this patch tested?
Pass the CIs.
Closes #36725 from dongjoon-hyun/SPARK-39322-2.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../scala/org/apache/spark/ExecutorAllocationManager.scala | 12 +++++-------
1 file changed, 5 insertions(+), 7 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
index c4b619300b5..b9bc3c63ff4 100644
--- a/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
+++ b/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
@@ -204,13 +204,11 @@ private[spark] class ExecutorAllocationManager(
s"s${DYN_ALLOCATION_SUSTAINED_SCHEDULER_BACKLOG_TIMEOUT.key} must be > 0!")
}
if (!conf.get(config.SHUFFLE_SERVICE_ENABLED)) {
- // If dynamic allocation shuffle tracking or worker decommissioning along with
- // storage shuffle decommissioning is enabled we have *experimental* support for
- // decommissioning without a shuffle service.
- if (conf.get(config.DYN_ALLOCATION_SHUFFLE_TRACKING_ENABLED) ||
- (decommissionEnabled &&
- conf.get(config.STORAGE_DECOMMISSION_SHUFFLE_BLOCKS_ENABLED))) {
- logWarning("Dynamic allocation without a shuffle service is an experimental feature.")
+ if (conf.get(config.DYN_ALLOCATION_SHUFFLE_TRACKING_ENABLED)) {
+ logInfo("Dynamic allocation is enabled without a shuffle service.")
+ } else if (decommissionEnabled &&
+ conf.get(config.STORAGE_DECOMMISSION_SHUFFLE_BLOCKS_ENABLED)) {
+ logInfo("Shuffle data decommission is enabled without a shuffle service.")
} else if (!testing) {
throw new SparkException("Dynamic allocation of executors requires the external " +
"shuffle service. You may enable this through spark.shuffle.service.enabled.")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org