You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Abhishek Tiwari (JIRA)" <ji...@apache.org> on 2017/07/19 10:22:00 UTC
[jira] [Updated] (GOBBLIN-140) Error when restarting GobblinYarn
[ https://issues.apache.org/jira/browse/GOBBLIN-140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhishek Tiwari updated GOBBLIN-140:
------------------------------------
Component/s: gobblin-yarn
> Error when restarting GobblinYarn
> ---------------------------------
>
> Key: GOBBLIN-140
> URL: https://issues.apache.org/jira/browse/GOBBLIN-140
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-yarn
> Reporter: Joel Baranick
> Assignee: Abhishek Tiwari
> Labels: Bug:Generic, LaunchType:Yarn
>
> The following error occurs after trying to restart the GobblinYarn driver:
> ```
> 2016-02-17 06:27:15 UTC WARN [main] org.apache.helix.manager.zk.ZKHelixAdmin 495 - Root directory exists.Cleaning the root directory:/GobblinYarn
> Exception in thread main org.I0Itec.zkclient.exception.ZkException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Dir
> ectory not empty for /GobblinYarn/CONTROLLER
> at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:68)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:685)
> at org.apache.helix.manager.zk.ZkClient.delete(ZkClient.java:348)
> at org.I0Itec.zkclient.ZkClient.deleteRecursive(ZkClient.java:516)
> at org.I0Itec.zkclient.ZkClient.deleteRecursive(ZkClient.java:511)
> at org.apache.helix.manager.zk.ZKHelixAdmin.addCluster(ZKHelixAdmin.java:496)
> at org.apache.helix.tools.ClusterSetup.addCluster(ClusterSetup.java:154)
> at gobblin.yarn.YarnHelixUtils.createGobblinYarnHelixCluster(YarnHelixUtils.java:67)
> at gobblin.yarn.GobblinYarnAppLauncher.launch(GobblinYarnAppLauncher.java:243)
> at gobblin.yarn.GobblinYarnAppLauncher.main(GobblinYarnAppLauncher.java:784)
> Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /GobblinYarn/CONTROLLER
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at org.I0Itec.zkclient.ZkConnection.delete(ZkConnection.java:91)
> at org.apache.helix.manager.zk.ZkClient$8.call(ZkClient.java:352)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> ... 8 more
> ```
> These are the logs of the driver shutting down prior to restart:
> ```
> 2016-02-17 06:25:01 UTC INFO [Thread-4] gobblin.yarn.GobblinYarnAppLauncher 301 - Stopping the GobblinYarnAppLauncher
> 2016-02-17 06:25:01 UTC INFO [Thread-4] org.apache.helix.messaging.DefaultMessagingService 84 - Send 1 messages with criteria instanceName=%resourceName=%partitionName=%partitionState=%
> 2016-02-17 06:25:02 UTC INFO [LogCopier STOPPING] gobblin.util.ExecutorsUtils 125 - Attempting to shutdown ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@73240b61[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1862]
> 2016-02-17 06:25:02 UTC INFO [LogCopier STOPPING] gobblin.util.ExecutorsUtils 144 - Successfully shutdown ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@73240b61[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1862]
> 2016-02-17 06:25:02 UTC INFO [JobExecutionInfoServer STOPPING] gobblin.rest.JobExecutionInfoServer 94 - Stopping the job execution information server
> Shutting down
> 2016-02-17 06:25:02 UTC INFO [AdminWebServer STOPPING] org.eclipse.jetty.server.AbstractConnector 306 - Stopped ServerConnector@35e0c350{HTTP/1.1}{localhost:8280}
> 2016-02-17 06:25:02 UTC INFO [Thread-4] gobblin.util.ExecutorsUtils 125 - Attempting to shutdown ExecutorService: java.util.concurrent.Executors$DelegatedScheduledExecutorService@185aaf1f
> 2016-02-17 06:25:02 UTC INFO [Thread-4] gobblin.util.ExecutorsUtils 144 - Successfully shutdown ExecutorService: java.util.concurrent.Executors$DelegatedScheduledExecutorService@185aaf1f
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.manager.zk.ZKHelixManager 546 - disconnect ip-169-0-0-1(SPECTATOR) from GobblinYarn
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 679 - Shutting down HelixTaskExecutor
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 443 - Reset HelixTaskExecutor
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 453 - Reset exectuor for msgType: TASK_REPLY, pool: java.util.concurrent.ThreadPoolExecutor@3f197a46[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 397 - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@3f197a46[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.messaging.handling.HelixTaskExecutor 684 - Shutdown HelixTaskExecutor finished
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.manager.zk.ZkClient 130 - Closing zkclient: State:CONNECTED Timeout:30000 sessionid:0xd452eb397b640065 local:/169.0.0.1:51319 remoteserver:ip-138-0-0-1.ec2.internal/138.0.0.2:2181 lastZxid:60129782948 xid:17 sent:140 recv:140 queuedpkts:0 pendingresp:0 queuedevents:0
> 2016-02-17 06:25:02 UTC INFO [ZkClient-EventThread-17-zk.server:2181] org.I0Itec.zkclient.ZkEventThread 82 - Terminate ZkClient event thread.
> 2016-02-17 06:25:02 UTC INFO [main-EventThread] org.apache.zookeeper.ClientCnxn$EventThread 512 - EventThread shut down
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.zookeeper.ZooKeeper 684 - Session: 0xd452eb397b640065 closed
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.manager.zk.ZkClient 157 - Closed zkclient
> 2016-02-17 06:25:02 UTC INFO [Thread-4] org.apache.helix.manager.zk.ZKHelixManager 570 - Cluster manager: ip-169-0-0-1 disconnected
> 2016-02-17 06:25:02 UTC INFO [Thread-4] gobblin.yarn.GobblinYarnAppLauncher 722 - Deleting application working directory hdfs://ec2-145-0-0-1.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1455654714320_0004
> ```
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/715
> *Github Reporter* : [~jbaranick]
> *Github Created At* : 2016-02-17T06:37:52Z
> *Github Updated At* : 2016-03-08T06:17:34Z
> h3. Comments
> ----
> [~liyinan926] wrote on 2016-02-17T06:49:18Z : This happens sometimes. For some reason, the `ZKHelixAdmin` silently fails to fully cleanup the ZK structure corresponding to the Helix cluster instance even if it is configured to do the cleanup before everything else starts up. This causes the Yarn application to fail to start. A workaround is to manually delete the root ZK node `/GobblinYarn`.
>
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185056071
> ----
> [~jbaranick] wrote on 2016-02-17T06:50:20Z : That has been what I do, but it would be good to track this and try to get a solution. BTW, it also bloats the logs when this happens.
>
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185056205
> ----
> [~liyinan926] wrote on 2016-02-17T06:54:42Z : I would suggest reporting this through a bug ticket against Helix. Basically `ZKHelixAdmin` should fail or throw some meaningful exception if it fails to fully cleanup. I believe this is not too hard to fix in Helix.
>
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185057815
> ----
> [~jbaranick] wrote on 2016-02-17T07:07:42Z : Created [HELIX-628](https://issues.apache.org/jira/browse/HELIX-628)
>
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185064210
> ----
> [~jbaranick] wrote on 2016-02-17T21:47:02Z : Also, deleting the root ZK node /GobblinYarn is tricky and might require a number of attempts before it actually works and unblocks the yarn application.
>
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-185422659
> ----
> [~jbaranick] wrote on 2016-02-22T09:00:14Z : @liyinan926 According to `kishore g` from the helix team:
> > This can happen when there all instances (participants/controllers) were not shutdown in the previous run. From your logs, it looks like there was a CONTROLLER that was still running when the new yarn application was started.
> If that is the case, then this is caused by #654
>
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/715#issuecomment-187080118
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)