You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by zjffdu <gi...@git.apache.org> on 2018/04/28 00:29:19 UTC
[GitHub] zeppelin pull request #2951: ZEPPELIN-3435. Interpreter timeout lifecycle le...
GitHub user zjffdu opened a pull request:
https://github.com/apache/zeppelin/pull/2951
ZEPPELIN-3435. Interpreter timeout lifecycle leads to interpreter process orphans
### What is this PR for?
This issue happens when LIfecycleManager try to close InterpreterGroup when it is in the middle of starting.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3435
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-3435
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2951.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2951
----
----
---
[GitHub] zeppelin pull request #2951: ZEPPELIN-3435. Interpreter timeout lifecycle le...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/zeppelin/pull/2951
---
[GitHub] zeppelin issue #2951: ZEPPELIN-3435. Interpreter timeout lifecycle leads to ...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2951
Thanks @felixcheung Unit test is added
---
[GitHub] zeppelin issue #2951: ZEPPELIN-3435. Interpreter timeout lifecycle leads to ...
Posted by weand <gi...@git.apache.org>.
Github user weand commented on the issue:
https://github.com/apache/zeppelin/pull/2951
:+1: no more orphans were spotted since applying changes of this PR.
---
[GitHub] zeppelin issue #2951: ZEPPELIN-3435. Interpreter timeout lifecycle leads to ...
Posted by weand <gi...@git.apache.org>.
Github user weand commented on the issue:
https://github.com/apache/zeppelin/pull/2951
I can't confirm that. LifecycleManager tried to close the interpreter group at 13:06 (see log pasted in the Jira) - 60 minutes after the last job was finished. The same interpreter group was started again at 13:17 - from Interpreter Log:
```
INFO [2018-04-27 12:06:40,066] ({dag-scheduler-event-loop} Logging.scala[logInfo]:54) - ResultStage 113 (toPandas at <ipython-input-17-a6b974ad21d2>:23) finished in 0.038 s
INFO [2018-04-27 12:06:40,066] ({Thread-19} Logging.scala[logInfo]:54) - Job 28 finished: toPandas at <ipython-input-17-a6b974ad21d2>:23, took 0.082673 s
INFO [2018-04-27 12:06:40,101] ({pool-2-thread-10} SchedulerFactory.java[jobFinished]:115) - Job 20180215-131521_501880463 finished by scheduler interpreter_101669487
INFO [2018-04-27 12:09:40,123] ({spark-dynamic-executor-allocation} Logging.scala[logInfo]:54) - Request to remove executorIds: 1, 0
INFO [2018-04-27 12:09:40,125] ({spark-dynamic-executor-allocation} Logging.scala[logInfo]:54) - Requesting to kill executor(s) 1, 0
INFO [2018-04-27 12:09:40,128] ({spark-dynamic-executor-allocation} Logging.scala[logInfo]:54) - Actual list of executor(s) to be killed is 1, 0
INFO [2018-04-27 12:09:40,135] ({spark-dynamic-executor-allocation} Logging.scala[logInfo]:54) - Removing executor 1 because it has been idle for 180 seconds (new desired total will be 1)
INFO [2018-04-27 12:09:40,135] ({spark-dynamic-executor-allocation} Logging.scala[logInfo]:54) - Removing executor 0 because it has been idle for 180 seconds (new desired total will be 0)
INFO [2018-04-27 12:09:40,547] ({dispatcher-event-loop-4} Logging.scala[logInfo]:54) - Executor 0 on 172.16.16.63 killed by driver.
INFO [2018-04-27 12:09:40,551] ({dispatcher-event-loop-4} Logging.scala[logInfo]:54) - Executor 1 on 172.16.16.63 killed by driver.
INFO [2018-04-27 12:09:40,552] ({dag-scheduler-event-loop} Logging.scala[logInfo]:54) - Executor lost: 0 (epoch 56)
INFO [2018-04-27 12:09:40,553] ({dispatcher-event-loop-2} Logging.scala[logInfo]:54) - Trying to remove executor 0 from BlockManagerMaster.
INFO [2018-04-27 12:09:40,555] ({dispatcher-event-loop-2} Logging.scala[logInfo]:54) - Removing block manager BlockManagerId(0, 172.16.16.63, 33086, None)
INFO [2018-04-27 12:09:40,556] ({SparkListenerBus} Logging.scala[logInfo]:54) - Existing executor 0 has been removed (new total is 1)
INFO [2018-04-27 12:09:40,556] ({dag-scheduler-event-loop} Logging.scala[logInfo]:54) - Removed 0 successfully in removeExecutor
INFO [2018-04-27 12:09:40,557] ({dag-scheduler-event-loop} Logging.scala[logInfo]:54) - Executor lost: 1 (epoch 56)
INFO [2018-04-27 12:09:40,557] ({dispatcher-event-loop-6} Logging.scala[logInfo]:54) - Trying to remove executor 1 from BlockManagerMaster.
INFO [2018-04-27 12:09:40,557] ({dispatcher-event-loop-6} Logging.scala[logInfo]:54) - Removing block manager BlockManagerId(1, 172.16.16.63, 34862, None)
INFO [2018-04-27 12:09:40,557] ({SparkListenerBus} Logging.scala[logInfo]:54) - Existing executor 1 has been removed (new total is 0)
INFO [2018-04-27 12:09:40,557] ({dag-scheduler-event-loop} Logging.scala[logInfo]:54) - Removed 1 successfully in removeExecutor
INFO [2018-04-27 12:15:25,523] ({Spark Context Cleaner} Logging.scala[logInfo]:54) - Cleaned accumulator 1428
INFO [2018-04-27 12:15:25,531] ({Spark Context Cleaner} Logging.scala[logInfo]:54) - Cleaned accumulator 1422
INFO [2018-04-27 12:15:25,531] ({Spark Context Cleaner} Logging.scala[logInfo]:54) - Cleaned shuffle 30
INFO [2018-04-27 12:15:25,531] ({Spark Context Cleaner} Logging.scala[logInfo]:54) - Cleaned accumulator 1433
INFO [2018-04-27 12:15:25,531] ({Spark Context Cleaner} Logging.scala[logInfo]:54) - Cleaned accumulator 1431
WARN [2018-04-27 13:17:26,528] ({main} NativeCodeLoader.java[<clinit>]:62) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
INFO [2018-04-27 13:17:26,619] ({main} RemoteInterpreterServer.java[main]:260) - URL:jar:file:/opt/zeppelin/zeppelin-0.8.1.20180426202048/lib/interpreter/zeppelin-interpreter-0.8.1-SNAPSHOT.jar!/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.class
INFO [2018-04-27 13:17:26,654] ({main} RemoteInterpreterServer.java[<init>]:161) - Launching ThriftServer at 172.16.16.65:46586
INFO [2018-04-27 13:17:26,657] ({main} RemoteInterpreterServer.java[<init>]:165) - Starting remote interpreter server on port 46586
INFO [2018-04-27 13:17:26,659] ({Thread-2} RemoteInterpreterServer.java[run]:202) - Starting remote interpreter server on port 46586
INFO [2018-04-27 13:17:27,665] ({Thread-3} RemoteInterpreterUtils.java[registerInterpreter]:165) - callbackHost: 172.16.16.65, callbackPort: 39299, callbackInfo: CallbackInfo(host:172.16.16.65, port:46586)
INFO [2018-04-27 13:17:27,772] ({pool-1-thread-1} RemoteInterpreterServer.java[createInterpreter]:310) - Instantiate interpreter org.apache.zeppelin.spark.SparkInterpreter
INFO [2018-04-27 13:17:27,775] ({pool-1-thread-1} RemoteInterpreterServer.java[createInterpreter]:310) - Instantiate interpreter org.apache.zeppelin.spark.SparkSqlInterpreter
INFO [2018-04-27 13:17:27,777] ({pool-1-thread-1} RemoteInterpreterServer.java[createInterpreter]:310) - Instantiate interpreter org.apache.zeppelin.spark.DepInterpreter
INFO [2018-04-27 13:17:27,788] ({pool-1-thread-1} RemoteInterpreterServer.java[createInterpreter]:310) - Instantiate interpreter org.apache.zeppelin.spark.PySparkInterpreter
INFO [2018-04-27 13:17:27,792] ({pool-1-thread-1} RemoteInterpreterServer.java[createInterpreter]:310) - Instantiate interpreter org.apache.zeppelin.spark.IPySparkInterpreter
INFO [2018-04-27 13:17:27,794] ({pool-1-thread-1} RemoteInterpreterServer.java[createInterpreter]:310) - Instantiate interpreter org.apache.zeppelin.spark.SparkRInterpreter
INFO [2018-04-27 13:17:27,849] ({pool-2-thread-5} SchedulerFactory.java[jobStarted]:109) - Job 20180213-114056_1717909171 started by scheduler interpreter_2006847219
```
---
[GitHub] zeppelin issue #2951: ZEPPELIN-3435. Interpreter timeout lifecycle leads to ...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2951
Thanks @weand will merge if no more comments
---
[GitHub] zeppelin issue #2951: ZEPPELIN-3435. Interpreter timeout lifecycle leads to ...
Posted by weand <gi...@git.apache.org>.
Github user weand commented on the issue:
https://github.com/apache/zeppelin/pull/2951
... but we are now trying this in production.
---
[GitHub] zeppelin issue #2951: ZEPPELIN-3435. Interpreter timeout lifecycle leads to ...
Posted by zjffdu <gi...@git.apache.org>.
Github user zjffdu commented on the issue:
https://github.com/apache/zeppelin/pull/2951
@weand I suspect this issue happens when LifecycleManager try to close InterpreterGroup when it is in the middle of starting (you can check the log to confirm it). If it is, could you help verify this PR. ? Thanks
---