You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "KaiXinXIaoLei (JIRA)" <ji...@apache.org> on 2015/07/21 05:21:04 UTC
[jira] [Commented] (SPARK-9209) Using executor allocation, a
executor is removed but it exists in ExecutorPage of the web ui
[ https://issues.apache.org/jira/browse/SPARK-9209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634468#comment-14634468 ]
KaiXinXIaoLei commented on SPARK-9209:
--------------------------------------
Here i add the event log to make this problem clear:
{"Event":"SparkListenerBlockManagerRemoved","Block Manager ID":{"Executor ID":"264","Host":"172.1.1.8","Port":23811},"Timestamp":1437104894543}
{"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"264","Host":"172.1.1.8","Port":23811},"Maximum Memory":11113416622,"Timestamp":1437104901226}
{"Event":"SparkListenerExecutorRemoved","Timestamp":1437104915802,"Executor ID":"264","Removed Reason":"remote Rpc client disassociated"}
> Using executor allocation, a executor is removed but it exists in ExecutorPage of the web ui
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-9209
> URL: https://issues.apache.org/jira/browse/SPARK-9209
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.1
> Reporter: KaiXinXIaoLei
> Fix For: 1.5.0
>
> Attachments: A Executor exists in web.png, executor is removed.png
>
>
> I set "spark.dynamicAllocation.enabled = true”, and run a big job. In driver, a executors are asked to remove, and it's remove successfully, but it exists in ExecutorPage of the web ui.
> The log in driver :
> 2015-07-17 11:48:14,543 | INFO | [sparkDriver-akka.actor.default-dispatcher-3] | Removing block manager BlockManagerId(264, 172.1.1.8, 23811)
> 2015-07-17 11:48:14,543 | INFO | [dag-scheduler-event-loop] | Removed 264 successfully in removeExecutor
> 2015-07-17 11:48:21,226 | INFO | [sparkDriver-akka.actor.default-dispatcher-3] | Registering block manager 172.1.1.8:23811 with 10.4 GB RAM, BlockManagerId(264, 172.1.1.8, 23811)
> 2015-07-17 11:48:21,228 | INFO | [sparkDriver-akka.actor.default-dispatcher-3] | Added broadcast_781_piece0 in memory on 172.1.1.8:23811 (size: 38.6 KB, free: 10.4 GB)
> 2015-07-17 11:48:35,277 | ERROR | [sparkDriver-akka.actor.default-dispatcher-16] | Lost executor 264 on datasight-195: remote Rpc client disassociated
> 2015-07-17 11:48:35,277 | WARN | [sparkDriver-akka.actor.default-dispatcher-4] | Association with remote system [akka.tcp://sparkExecutor@datasight-195:23929] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 2015-07-17 11:48:35,277 | INFO | [sparkDriver-akka.actor.default-dispatcher-16] | Re-queueing tasks for 264 from TaskSet 415.0
> 2015-07-17 11:48:35,804 | INFO | [SparkListenerBus] | Existing executor 264 has been removed (new total is 10)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org