[jira] [Commented] (SPARK-33557) spark.storage.blockManagerSlaveTimeoutMs default value does not follow spark.network.timeout value when the latter was changed

["Yang Jie (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-33557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17239631#comment-17239631 ] 

Yang Jie commented on SPARK-33557:
----------------------------------

It seems that changing value of "spark.network.timeout" doesn't really change the value of STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT and their relationship is maintained by code.

For example, the treatment of "spark.shuffle.io.connectionTimeout" is as follows:

 
{code:java}
/** Connect timeout in milliseconds. Default 120 secs. */
public int connectionTimeoutMs() {
  long defaultNetworkTimeoutS = JavaUtils.timeStringAsSec(
    conf.get("spark.network.timeout", "120s"));
  long defaultTimeoutMs = JavaUtils.timeStringAsSec(
    conf.get(SPARK_NETWORK_IO_CONNECTIONTIMEOUT_KEY, defaultNetworkTimeoutS + "s")) * 1000;
  return (int) defaultTimeoutMs;
}
{code}
 

 

But it seems that there is no similar treatment forSTORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT in HeartbeatReceiver and MesosCoarseGrainedSchedulerBackend

 
{code:java}
private val executorTimeoutMs = sc.conf.get(config.STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT)
{code}
 

 
{code:java}
mesosExternalShuffleClient.get
  .registerDriverWithShuffleService(
    agent.hostname,
    externalShufflePort,
    sc.conf.get(config.STORAGE_BLOCKMANAGER_HEARTBEAT_TIMEOUT),
    sc.conf.get(config.EXECUTOR_HEARTBEAT_INTERVAL))
{code}
 

Maybe need to be fixed by code changes.

 

 

> spark.storage.blockManagerSlaveTimeoutMs default value does not follow spark.network.timeout value when the latter was changed
> ------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-33557
>                 URL: https://issues.apache.org/jira/browse/SPARK-33557
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 3.0.0, 3.0.1
>            Reporter: Ohad
>            Priority: Minor
>
> According to the documentation "spark.network.timeout" is the default timeout for "spark.storage.blockManagerSlaveTimeoutMs" which implies that when the user sets "spark.network.timeout"  the effective value of "spark.storage.blockManagerSlaveTimeoutMs" should also be changed if it was not specifically changed.
> However this is not the case since the default value of "spark.storage.blockManagerSlaveTimeoutMs" is always the default value of "spark.network.timeout" (120s)
>  
> "spark.storage.blockManagerSlaveTimeoutMs" is defined in the package object of "org.apache.spark.internal.config" as follows:
> {code:java}
> private[spark] val STORAGE_BLOCKMANAGER_SLAVE_TIMEOUT =
>   ConfigBuilder("spark.storage.blockManagerSlaveTimeoutMs")
>     .version("0.7.0")
>     .timeConf(TimeUnit.MILLISECONDS)
>     .createWithDefaultString(Network.NETWORK_TIMEOUT.defaultValueString)
> {code}
> So it seems like the its default value is indeed "fixed" to "spark.network.timeout" default value.
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org