You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2018/09/30 21:31:00 UTC
[jira] [Updated] (SPARK-25543) Confusing log messages at DEBUG
level, in K8s mode.
[ https://issues.apache.org/jira/browse/SPARK-25543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-25543:
----------------------------------
Affects Version/s: (was: 2.5.0)
2.4.0
> Confusing log messages at DEBUG level, in K8s mode.
> ---------------------------------------------------
>
> Key: SPARK-25543
> URL: https://issues.apache.org/jira/browse/SPARK-25543
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Priority: Minor
> Fix For: 2.4.1, 2.5.0
>
>
> Steps to reproduce.
> Start spark shell by providing a K8s master. Then turn the debug log on,
> {code}
> scala> sc.setLogLevel("DEBUG")
> {code}
> {code}
> sc.setLogLevel("DEBUG")
> scala> 2018-09-26 09:33:54 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:33:55 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:33:56 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:33:56 DEBUG ExecutorPodsPollingSnapshotSource:58 - Resynchronizing full executor pod state from Kubernetes.
> 2018-09-26 09:33:57 DEBUG ExecutorPodsAllocator:58 - Currently have 1 running executors and 0 pending executors. Map() executors have been requested but are pending appearance in the cluster.
> 2018-09-26 09:33:57 DEBUG ExecutorPodsAllocator:58 - Current number of running executors is equal to the number of requested executors. Not scaling up further.
> 2018-09-26 09:33:57 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:33:58 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:33:59 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:34:00 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:34:01 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:34:02 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:34:03 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:34:04 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:34:05 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
> 2018-09-26 09:34:06 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from ...
> {code}
> The fix is easy, first check if there are any removed executors, before producing the log message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org