You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by "Abhishek Tiwari (JIRA)" <ji...@apache.org> on 2017/07/19 10:21:00 UTC
[jira] [Assigned] (GOBBLIN-145) Gobblin Yarn fails to shutdown all
containers
[ https://issues.apache.org/jira/browse/GOBBLIN-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhishek Tiwari reassigned GOBBLIN-145:
---------------------------------------
Assignee: Abhishek Tiwari
Component/s: gobblin-yarn
> Gobblin Yarn fails to shutdown all containers
> ---------------------------------------------
>
> Key: GOBBLIN-145
> URL: https://issues.apache.org/jira/browse/GOBBLIN-145
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-yarn
> Reporter: Joel Baranick
> Assignee: Abhishek Tiwari
> Labels: Bug:Generic, LaunchType:Yarn
>
> While shutting down the application, not all containers are shutdown. The following are the logs for the failure:
> ```
> 2016-02-02 21:10:05 UTC ERROR [pool-142-thread-1] gobblin.yarn.GobblinApplicationMaster - Timeout in stopping the service manager
> java.util.concurrent.TimeoutException: Timeout waiting for the services to stop.
> at com.google.common.util.concurrent.ServiceManager.awaitStopped(ServiceManager.java:338)
> at gobblin.yarn.GobblinApplicationMaster.stop(GobblinApplicationMaster.java:208)
> at gobblin.yarn.GobblinApplicationMaster.handleApplicationMasterShutdownRequest(GobblinApplicationMaster.java:287)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.google.common.eventbus.EventHandler.handleEvent(EventHandler.java:74)
> at com.google.common.eventbus.SynchronizedEventHandler.handleEvent(SynchronizedEventHandler.java:47)
> at com.google.common.eventbus.EventBus.dispatch(EventBus.java:314)
> at com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:296)
> at com.google.common.eventbus.EventBus.post(EventBus.java:267)
> at gobblin.yarn.GobblinApplicationMaster$ControllerShutdownMessageHandlerFactory$ControllerShutdownMessageHandler$1.run(GobblinApplicationMaster.java:445)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Shutting down HelixTaskExecutor
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Reset HelixTaskExecutor
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.monitoring.mbeans.MessageQueueMonitor - Unregistering ClusterStatus: cluster=GobblinYarn,messageQueue=GobblinApplicationMaster
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Reset exectuor for msgType: CONTROLLER_MSG, pool: java.util.concurrent.ThreadPoolExecutor@7394d9eb[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@7394d9eb[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Reset exectuor for msgType: SCHEDULER_MSG, pool: java.util.concurrent.ThreadPoolExecutor@316f0e95[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@316f0e95[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Reset exectuor for msgType: PARTICIPANT_ERROR_REPORT, pool: java.util.concurrent.ThreadPoolExecutor@7226718d[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@7226718d[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Reset exectuor for msgType: USER_DEFINE_MSG, pool: java.util.concurrent.ThreadPoolExecutor@39e91a87[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@39e91a87[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Reset exectuor for msgType: SHUTDOWN, pool: java.util.concurrent.ThreadPoolExecutor@76ccb2c4[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@76ccb2c4[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Reset exectuor for msgType: TASK_REPLY, pool: java.util.concurrent.ThreadPoolExecutor@78ec9eca[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Shutting down pool: java.util.concurrent.ThreadPoolExecutor@78ec9eca[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Shutdown HelixTaskExecutor finished
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.helix.messaging.handling.HelixTaskExecutor - Reset HelixTaskExecutor
> 2016-02-02 21:10:05 UTC INFO [ZkClient-EventThread-35-stage.zk.int.data.ensighten.com:2181] org.I0Itec.zkclient.ZkEventThread - Terminate ZkClient event thread.
> 2016-02-02 21:10:05 UTC INFO [pool-142-thread-1] org.apache.zookeeper.ZooKeeper - Session: 0xd451accea96b1186 closed
> 2016-02-02 21:10:05 UTC INFO [main-EventThread] org.apache.zookeeper.ClientCnxn$EventThread - EventThread shut down
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454445160766 does not exist
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454444218366 does not exist
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454443560377 does not exist
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454443440378 does not exist
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454446116585 does not exist
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-5] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454445160766: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-2] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454443440378: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454443800384 does not exist
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-7] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454446116585: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-6] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454443560377: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-3] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454444218366: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-8] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454443800384: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454443980357 does not exist
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454443680334 does not exist
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-1] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454443980357: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-10] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454443680334: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454443320333 does not exist
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-9] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454443320333: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC INFO [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Stopping the TaskStateCollectorService
> 2016-02-02 21:10:06 UTC WARN [TaskStateCollectorService STOPPING] gobblin.runtime.TaskStateCollectorService - Output task state path hdfs://SERVER.compute-1.amazonaws.com:9000/user/yarn/GobblinYarn/application_1454431761767_0015/_taskstates/job_JOB_NAME_1454443920554 does not exist
> 2016-02-02 21:10:06 UTC ERROR [DefaultQuartzScheduler_Worker-4] gobblin.runtime.AbstractJobLauncher - Failed to launch and run job job_JOB_NAME_1454443920554: org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> org.apache.helix.HelixException: HelixManager is not connected. Call HelixManager#connect()
> at org.apache.helix.manager.zk.ZKHelixManager.checkConnected(ZKHelixManager.java:292)
> at org.apache.helix.manager.zk.ZKHelixManager.getHelixPropertyStore(ZKHelixManager.java:644)
> at org.apache.helix.task.TaskUtil.getWorkflowContext(TaskUtil.java:256)
> at gobblin.yarn.GobblinHelixJobLauncher.waitForJobCompletion(GobblinHelixJobLauncher.java:251)
> at gobblin.yarn.GobblinHelixJobLauncher.runWorkUnits(GobblinHelixJobLauncher.java:151)
> at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:251)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-7] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@61de017e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-7] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@61de017e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-2] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@2452b0a3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-2] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@2452b0a3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-2] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@29138466[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-2] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@29138466[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-2] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-7] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@34959ee0[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-7] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@34959ee0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-7] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-8] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@b4292e5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-6] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@2aa79345[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-8] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@b4292e5[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-6] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@2aa79345[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-8] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@23c27ed1[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-8] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@23c27ed1[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-8] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-6] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@52dcb5f7[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-6] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@52dcb5f7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-6] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-3] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@328a7eb9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-3] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@328a7eb9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-5] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@5a29ab8d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-5] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@5a29ab8d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-3] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@1d48efd4[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-3] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@1d48efd4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-3] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-5] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@534b3f08[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-5] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@534b3f08[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-5] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-10] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@653be4b0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-10] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@653be4b0[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-9] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@3f2312bc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-9] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@3f2312bc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-9] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@b63fa9f[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-9] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@b63fa9f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-10] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@50dae290[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-10] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@50dae290[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-9] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:06 UTC INFO [DefaultQuartzScheduler_Worker-10] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-4] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@1932b09[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-4] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@1932b09[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-4] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@670474d4[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-4] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@670474d4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-4] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-1] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@63753308[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-1] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@63753308[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-1] gobblin.util.ExecutorsUtils - Attempting to shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@61e4879d[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-1] gobblin.util.ExecutorsUtils - Successfully shutdown ExecutorService: java.util.concurrent.ThreadPoolExecutor@61e4879d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
> 2016-02-02 21:10:07 UTC INFO [DefaultQuartzScheduler_Worker-1] org.quartz.core.JobRunShell - Job nexus.JOB_NAME threw a JobExecutionException:
> org.quartz.JobExecutionException: java.lang.NullPointerException [See nested exception: java.lang.NullPointerException]
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:58)
> at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
> at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
> Caused by: java.lang.NullPointerException
> at org.apache.helix.manager.zk.ZkClient$12.call(ZkClient.java:412)
> at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
> at org.apache.helix.manager.zk.ZkClient.asyncExists(ZkClient.java:408)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.getStats(ZkBaseDataAccessor.java:938)
> at org.apache.helix.manager.zk.ZkBaseDataAccessor.exists(ZkBaseDataAccessor.java:909)
> at org.apache.helix.manager.zk.ZKUtil.isClusterSetup(ZKUtil.java:70)
> at org.apache.helix.ConfigAccessor.getKeys(ConfigAccessor.java:482)
> at org.apache.helix.task.TaskUtil.getResourceConfigMap(TaskUtil.java:464)
> at org.apache.helix.task.TaskUtil.getWorkflowCfg(TaskUtil.java:108)
> at org.apache.helix.task.TaskDriver.deleteJob(TaskDriver.java:284)
> at gobblin.yarn.GobblinHelixJobLauncher.executeCancellation(GobblinHelixJobLauncher.java:171)
> at gobblin.yarn.GobblinHelixJobLauncher.close(GobblinHelixJobLauncher.java:131)
> at com.google.common.io.Closer.close(Closer.java:214)
> at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:341)
> at gobblin.yarn.GobblinHelixJob.execute(GobblinHelixJob.java:56)
> ... 2 more
> 2016-02-02 21:10:07 UTC INFO [GobblinHelixJobScheduler STOPPING] org.quartz.core.QuartzScheduler - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
> ```
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/654
> *Github Reporter* : [~jbaranick]
> *Github Created At* : 2016-02-02T21:32:19Z
> *Github Updated At* : 2017-01-12T04:37:31Z
> h3. Comments
> ----
> [~stakiar] wrote on 2016-02-04T23:52:45Z : @kadaan could you send over more of the logs from the AppMaster; specifically, I'm interested in the logs that are printed just before the first exception.
> If you can, a copy of the logs from a container would also be helpful.
> My current theory is the containers are not properly setting their exit status when they get cancelled, which causes a timeout in the AppMaster. Some notes for future reference:
> - The GobblinAppMaster handles all its shutdown logic in a Shutdown Hook
> - If a user executes `yarn job -kill [applicationId]` my guess is that YARN will attempt to shutdown the AppMaster as well as all the containers, in no specific order.
> - The problem on cancellation of a container is that `GobblinHelixTask` does not implement the `cancel` method and thus has no chance of setting its exit status
> - This causes the `GobblinHelixJobLauncher.waitForJobCompletion` method to loop indefinitely because as far as Helix knowns the job has not been cancelled
> - Eventually there is a timeout in `GobblinApplicationMaster.stop` which then causes a disconnection from Helix, which then causes all the `HelixManager is not connected exceptions`
>
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/654#issuecomment-180109103
> ----
> [~jbaranick] wrote on 2016-02-18T06:57:36Z : @sahilTakiar Attached additional logs:
> - [application_master_obfuscated.txt](https://github.com/linkedin/gobblin/files/135770/application_master_obfuscated.txt)
> - [container_obfuscated.txt](https://github.com/linkedin/gobblin/files/135771/container_obfuscated.txt)
>
>
> *Github Url* : https://github.com/linkedin/gobblin/issues/654#issuecomment-185569863
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)