You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Abhishek Tiwari (JIRA)" <ji...@apache.org> on 2017/07/19 07:41:03 UTC
[jira] [Created] (GOBBLIN-140) Error when restarting GobblinYarn
Abhishek Tiwari created GOBBLIN-140:
---------------------------------------
Summary: Error when restarting GobblinYarn
Key: GOBBLIN-140
URL: https://issues.apache.org/jira/browse/GOBBLIN-140
Project: Apache Gobblin
Issue Type: Bug
Reporter: Joel Baranick
The following error occurs after trying to restart the GobblinYarn driver:
```
2016-02-17 06:27:15 UTC WARN [main] org.apache.helix.manager.zk.ZKHelixAdmin 495 - Root directory exists.Cleaning the root directory:/GobblinYarn
Exception in thread main org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Dir
ectory not empty for /GobblinYarn/CONTROLLER
at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:68)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
at org.apache.helix.manager.zk.ZkClient.delete(ZkClient.java:348)
at org.I0Itec.zkclient.ZkClient.deleteRecursive(ZkClient.java:516)
at org.I0Itec.zkclient.ZkClient.deleteRecursive(ZkClient.java:511)
at org.apache.helix.manager.zk.ZKHelixAdmin.addCluster(ZKHelixAdmin.java:496)
at org.apache.helix.tools.ClusterSetup.addCluster(ClusterSetup.java:154)
at gobblin.yarn.YarnHelixUtils.createGobblinYarnHelixCluster(YarnHelixUtils.java:67)
at gobblin.yarn.GobblinYarnAppLauncher.launch(GobblinYarnAppLauncher.java:243)
at gobblin.yarn.GobblinYarnAppLauncher.main(GobblinYarnAppLauncher.java:784)
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /GobblinYarn/CONTROLLER
at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.I0Itec.zkclient.ZkConnection.delete(ZkConnection.java:91)
at org.apache.helix.manager.zk.ZkClient$8.call(ZkClient.java:352)
at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
... 8 more
```
These are the logs of the driver shutting down prior to restart:
```
2016-02-17 06:25:01 UTC INFO [Thread-4] gobblin.yarn.GobblinYarnAppLauncher 301 - Stopping the GobblinYarnAppLauncher
2016-02-17 06:25:01 UTC INFO [Thread-4] org.apache.helix.messaging.DefaultMessagingService 84 - Send 1 messages with criteria instanceName=%resourceName=%partitionName=%partitionState=%
2016-02-17 06:25:02 UTC INFO [LogCopier STOPPING] gobblin.util.ExecutorsUtils 125 - Attempting to shutdown ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@73240b61[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1862]
2016-02-17 06:25:02 UTC INFO [LogCopier STOPPING] gobblin.util.ExecutorsUtils 144 - Successfully shutdown ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@73240b61[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1862]
2016-02-17 06:25:02 UTC INFO [JobExecutionInfoServer STOPPING] gobblin.rest.JobExecutionInfoServer 94 - Stopping the job execution information server
Shutting down
2016-02-17 06:25:02 UTC INFO [AdminWebServer STOPPING] org.eclipse.jetty.server.AbstractConnector 306 - Stopped ServerConnector@35e0c350{HTTP/1.1}{localhost:8280}
2016-02-17 06:25:02 UTC INFO [Thread-4] gobblin.util.ExecutorsUtils 125 - Attempting to shutdown ExecutorService: java.util.concurrent.Executors$DelegatedScheduledExecutorService@185aaf1f
2016-02-17 06:25:02 UTC INFO [Thread-4] gobblin.util.ExecutorsUtils 144 - Successfully shutdown ExecutorService: java.util.concurrent.Executors$DelegatedScheduledExecutorService@185aaf1f
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.manager.zk.ZKHelixManager 546 - disconnect ip-169-0-0-1(SPECTATOR) from GobblinYarn
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 679 - Shutting down HelixTaskExecutor
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 443 - Reset HelixTaskExecutor
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 453 - Reset exectuor for msgType: TASK_REPLY, pool: java.util.concurrent.ThreadPoolExecutor@3f197a46[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 397 - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@3f197a46[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 684 - Shutdown HelixTaskExecutor finished
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.manager.zk.ZkClient 130 - Closing zkclient: State:CONNECTED Timeout:30000 sessionid:0xd452eb397b640065 local:/169.0.0.1:51319 remoteserver:ip-138-0-0-1.ec2.internal/138.0.0.2:2181 lastZxid:60129782948 xid:17 sent:140 recv:140 queuedpkts:0 pendingresp:0 queuedevents:0
2016-02-17 06:25:02 UTC INFO [ZkClient-EventThread-17-zk.server:2181] org.I0Itec.zkclient.ZkEventThread 82 - Terminate ZkClient event thread.
2016-02-17 06:25:02 UTC INFO [main-EventThread] org.apache.zookeeper.ClientCnxn$EventThread 512 - EventThread shut down
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.zookeeper.ZooKeeper 684 - Session: 0xd452eb397b640065 closed
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.manager.zk.ZkClient 157 - Closed zkclient
2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.manager.zk.ZKHelixManager 570 - Cluster manager: ip-169-0-0-1 disconnected
2016-02-17 06:25:02 UTC INFO [Thread-4] gobblin.yarn.GobblinYarnAppLauncher 722 - Deleting application working directory hdfs://ec2-145-0-0-1.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1455654714320_0004
```
*Github Url* : https://github.com/linkedin/gobblin/issues/715
*Github Reporter* : [~jbaranick]
*Github Created At* : 2016-02-17T06:37:52Z
*Github Updated At* : 2016-03-08T06:17:34Z
h3. Comments
----
[~liyinan926] wrote on 2016-02-17T06:49:18Z : This happens sometimes. For some reason, the `ZKHelixAdmin` silently fails to fully cleanup the ZK structure corresponding to the Helix cluster instance even if it is configured to do the cleanup before everything else starts up. This causes the Yarn application to fail to start. A workaround is to manually delete the root ZK node `/GobblinYarn`.
*Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185056071
----
[~jbaranick] wrote on 2016-02-17T06:50:20Z : That has been what I do, but it would be good to track this and try to get a solution. BTW, it also bloats the logs when this happens.
*Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185056205
----
[~liyinan926] wrote on 2016-02-17T06:54:42Z : I would suggest reporting this through a bug ticket against Helix. Basically `ZKHelixAdmin` should fail or throw some meaningful exception if it fails to fully cleanup. I believe this is not too hard to fix in Helix.
*Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185057815
----
[~jbaranick] wrote on 2016-02-17T07:07:42Z : Created [HELIX-628](https://issues.apache.org/jira/browse/HELIX-628)
*Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185064210
----
[~jbaranick] wrote on 2016-02-17T21:47:02Z : Also, deleting the root ZK node /GobblinYarn is tricky and might require a number of attempts before it actually works and unblocks the yarn application.
*Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185422659
----
[~jbaranick] wrote on 2016-02-22T09:00:14Z : @liyinan926 According to `kishore g` from the helix team:
> This can happen when there all instances (participants/controllers) were not shutdown in the previous run. From your logs, it looks like there was a CONTROLLER that was still running when the new yarn application was started.
If that is the case, then this is caused by #654
*Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-187080118
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)