[jira] [Assigned] (SPARK-21099) INFO Log Message Using Incorrect Executor Idle Timeout Value

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-21099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-21099:
------------------------------------

    Assignee: Apache Spark

> INFO Log Message Using Incorrect Executor Idle Timeout Value
> ------------------------------------------------------------
>
>                 Key: SPARK-21099
>                 URL: https://issues.apache.org/jira/browse/SPARK-21099
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.6.0, 2.1.0
>            Reporter: Hazem Mahmoud
>            Assignee: Apache Spark
>            Priority: Trivial
>
> INFO log message is using the wrong idle timeout (spark.dynamicAllocation.executorIdleTimeout) when printing the message that the executor holding the RDD cache is being removed.
> INFO spark.ExecutorAllocationManager: Removing executor 1 because it has been idle for 30 seconds (new desired total will be 0)
> It should be using spark.dynamicAllocation.cachedExecutorIdleTimeout when the RDD cache timeout is reached. I was able to confirm this by doing the following:
> 1. Update spark-defaults.conf to set the following:
> executorIdleTimeout=30
> cachedExecutorIdleTimeout=20
> 2. Update log4j.properties to set the following:
> shell.log.level=INFO
> 3. Run the following in spark-shell:
> scala> val textFile = sc.textFile("/user/spark/applicationHistory/app_1234")
> scala> textFile.cache().count()
> 4. After 30 secs you will see 2 timeout messages, but of which are 30 secs (whereas one *should* be for 20 secs)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org