[jira] [Commented] (HIVE-10781) HadoopJobExecHelper Leaks RunningJobs

["Chinna Rao Lalam (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553969#comment-14553969 ] 

Chinna Rao Lalam commented on HIVE-10781:
-----------------------------------------

Hi [~nemon],

org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.runningJobs holds running jobs, if any abrupt shutdown happens shutdown hook uses this running jobs to kill. But in a normal flow after executing the running job it should kill and remove from the list.

In normal flow i suspect before removing from the list, if some exception was thrown (while doing rj.killJob())  it won't remove from the list. I have uploaded a patch to fix this..
Please check this if it helps

> HadoopJobExecHelper Leaks RunningJobs
> -------------------------------------
>
>                 Key: HIVE-10781
>                 URL: https://issues.apache.org/jira/browse/HIVE-10781
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive, HiveServer2
>    Affects Versions: 0.13.1, 1.2.0
>            Reporter: Nemon Lou
>         Attachments: HIVE-10781.patch
>
>
> On one of our busy hadoop cluster, hiveServer2 holds more than 4000 org.apache.hadoop.mapred.JobClient$NetworkedJob instances,while only has less than 3 backgroud handler thread at the same time.
> All these instances are hold in one LinkedList from org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper's  runningJobs property,which is static.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)