[jira] [Comment Edited] (TEZ-1661) LocalTaskScheduler hangs when shutdown

["Jeff Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278574#comment-14278574 ] 

Jeff Zhang edited comment on TEZ-1661 at 1/15/15 12:14 PM:
-----------------------------------------------------------

asyncDelegateRequestThread in LocalTaskSchedulerService is not stopped when DAGAppMaster is shutdown in local mode (actually it also happens in non-local mode, but we will call system.exit when shutting tez am in non-local mode, so it would not hang in non-local mode). The tez-examples don't hang in local mode because we always call System.exit when the job is done as following. But it doesn't make sense to require user to always do that.
{code}
    int res = ToolRunner.run(new Configuration(), new WordCount(), args);
    System.exit(res);
{code}

 Attach a patch for addressing this issue. [~sseth], [~jeagles] please help review. 


was (Author: zjffdu):
asyncDelegateRequestThread in LocalTaskSchedulerService is not stopped when DAGAppMaster is shutdown in local mode (actually it also happens in non-local mode, but we will call system.exit when shutting tez am in non-local mode, so it would not hang in non-local mode). The tez-examples don't hang in local mode because we always call System.exit when the job is done as following. But it doesn't make sense to require user to always do that. Attach a patch for addressing this issue. [~sseth], [~jeagles] please help review. 
{code}
    int res = ToolRunner.run(new Configuration(), new WordCount(), args);
    System.exit(res);
{code}

> LocalTaskScheduler hangs when shutdown
> --------------------------------------
>
>                 Key: TEZ-1661
>                 URL: https://issues.apache.org/jira/browse/TEZ-1661
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.5.0
>         Environment: Local Mode
>            Reporter: Oleg Zhurakousky
>            Assignee: Jeff Zhang
>         Attachments: TEZ-1661-1.patch
>
>
> LocalTaskScheduler hangs on 'take' from the 'taskRequestQueue ' when TezClient shuts down (e.g., TezClient.stop).
> Below is jstack output observed when running in Tez local mode:
> {code}
> "Thread-53" prio=5 tid=0x00007fc876d8f800 nid=0xac07 runnable [0x000000011df90000]
>    java.lang.Thread.State: RUNNABLE
>         at java.lang.Throwable.fillInStackTrace(Native Method)
>         at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
>         - locked <0x00000007b6ce60a0> (a java.lang.InterruptedException)
>         at java.lang.Throwable.<init>(Throwable.java:250)
>         at java.lang.Exception.<init>(Exception.java:54)
>         at java.lang.InterruptedException.<init>(InterruptedException.java:57)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
>         at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
>         at java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:535)
>         at org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.processRequest(LocalTaskSchedulerService.java:310)
>         at org.apache.tez.dag.app.rm.LocalTaskSchedulerService$AsyncDelegateRequestHandler.run(LocalTaskSchedulerService.java:304)
>         at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)