You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/02/02 18:59:06 UTC
[jira] Created: (HADOOP-969) deadlock in job tracker RetireJobs
deadlock in job tracker RetireJobs
----------------------------------
Key: HADOOP-969
URL: https://issues.apache.org/jira/browse/HADOOP-969
Project: Hadoop
Issue Type: Bug
Components: mapred
Reporter: Owen O'Malley
Assigned To: Owen O'Malley
Priority: Critical
Fix For: 0.11.0
The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like:
"IPC Server handler 5 on 50020":
at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)
- waiting to lock <0x74487a80> (a java.util.Vector)
- locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
- locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
"retireJobs":
at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
- waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
- locked <0x74487bb0> (a java.util.ArrayList)
- locked <0x74487a80> (a java.util.Vector)
- locked <0x74487a58> (a java.util.TreeMap)
at java.lang.Thread.run(Thread.java:595)
Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-969) deadlock in job tracker RetireJobs
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-969:
---------------------------------
Status: Patch Available (was: Open)
> deadlock in job tracker RetireJobs
> ----------------------------------
>
> Key: HADOOP-969
> URL: https://issues.apache.org/jira/browse/HADOOP-969
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Priority: Critical
> Fix For: 0.11.0
>
> Attachments: jt-dead.patch
>
>
> The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like:
> "IPC Server handler 5 on 50020":
> at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)
> - waiting to lock <0x74487a80> (a java.util.Vector)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
> "retireJobs":
> at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
> - waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
> at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
> - locked <0x74487bb0> (a java.util.ArrayList)
> - locked <0x74487a80> (a java.util.Vector)
> - locked <0x74487a58> (a java.util.TreeMap)
> at java.lang.Thread.run(Thread.java:595)
> Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
unsubscribe
Posted by Andrew Murray <an...@mpire.com>.
unsubscribe
Re: [jira] Commented: (HADOOP-969) deadlock in job tracker RetireJobs
Posted by Nigel Daley <nd...@yahoo-inc.com>.
Diregard Hadoop QA last comment. The patch couldn't be applied
because the patch has already been committed.
On Feb 2, 2007, at 12:37 PM, Hadoop QA (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/HADOOP-969?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel#action_12469861 ]
>
> Hadoop QA commented on HADOOP-969:
> ----------------------------------
>
> -1, because the patch command could not apply the latest attachment
> (http://issues.apache.org/jira/secure/attachment/12350238/jt-
> dead.patch) as a patch to trunk revision r502726. Please note that
> this message is automatically generated and may represent a problem
> with the automation system and not the patch.
>
>> deadlock in job tracker RetireJobs
>> ----------------------------------
>>
>> Key: HADOOP-969
>> URL: https://issues.apache.org/jira/browse/HADOOP-969
>> Project: Hadoop
>> Issue Type: Bug
>> Components: mapred
>> Reporter: Owen O'Malley
>> Assigned To: Owen O'Malley
>> Priority: Critical
>> Fix For: 0.11.0
>>
>> Attachments: jt-dead.patch
>>
>>
>> The JobTracker deadlocks because RetireJobs grabs locks in the
>> wrong order. The call stacks look like:
>> "IPC Server handler 5 on 50020":
>> at
>> org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker
>> (JobTracker.java:1108)
>> - waiting to lock <0x74487a80> (a java.util.Vector)
>> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
>> at org.apache.hadoop.mapred.JobTracker.heartbeat
>> (JobTracker.java:992)
>> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>> (DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
>> "retireJobs":
>> at org.apache.hadoop.mapred.JobTracker.removeJobTasks
>> (JobTracker.java:782)
>> - waiting to lock <0x744874b0> (a
>> org.apache.hadoop.mapred.JobTracker)
>> at org.apache.hadoop.mapred.JobTracker.access$300
>> (JobTracker.java:42)
>> at org.apache.hadoop.mapred.JobTracker$RetireJobs.run
>> (JobTracker.java:312)
>> - locked <0x74487bb0> (a java.util.ArrayList)
>> - locked <0x74487a80> (a java.util.Vector)
>> - locked <0x74487a58> (a java.util.TreeMap)
>> at java.lang.Thread.run(Thread.java:595)
>> Found 1 deadlock.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Commented: (HADOOP-969) deadlock in job tracker RetireJobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469861 ]
Hadoop QA commented on HADOOP-969:
----------------------------------
-1, because the patch command could not apply the latest attachment (http://issues.apache.org/jira/secure/attachment/12350238/jt-dead.patch) as a patch to trunk revision r502726. Please note that this message is automatically generated and may represent a problem with the automation system and not the patch.
> deadlock in job tracker RetireJobs
> ----------------------------------
>
> Key: HADOOP-969
> URL: https://issues.apache.org/jira/browse/HADOOP-969
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Priority: Critical
> Fix For: 0.11.0
>
> Attachments: jt-dead.patch
>
>
> The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like:
> "IPC Server handler 5 on 50020":
> at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)
> - waiting to lock <0x74487a80> (a java.util.Vector)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
> "retireJobs":
> at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
> - waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
> at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
> - locked <0x74487bb0> (a java.util.ArrayList)
> - locked <0x74487a80> (a java.util.Vector)
> - locked <0x74487a58> (a java.util.TreeMap)
> at java.lang.Thread.run(Thread.java:595)
> Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-969) deadlock in job tracker RetireJobs
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley updated HADOOP-969:
---------------------------------
Attachment: jt-dead.patch
This code has gotten pretty hairy, but this patch goes through the list of jobs to find ones that need to be retired. It then locks everything including the JobTracker first and does the mutations to all of the maps. We really need to refactor the datastructures at some point soon...
> deadlock in job tracker RetireJobs
> ----------------------------------
>
> Key: HADOOP-969
> URL: https://issues.apache.org/jira/browse/HADOOP-969
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Priority: Critical
> Fix For: 0.11.0
>
> Attachments: jt-dead.patch
>
>
> The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like:
> "IPC Server handler 5 on 50020":
> at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)
> - waiting to lock <0x74487a80> (a java.util.Vector)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
> "retireJobs":
> at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
> - waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
> at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
> - locked <0x74487bb0> (a java.util.ArrayList)
> - locked <0x74487a80> (a java.util.Vector)
> - locked <0x74487a58> (a java.util.TreeMap)
> at java.lang.Thread.run(Thread.java:595)
> Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-969) deadlock in job tracker RetireJobs
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12469833 ]
Mahadev konar commented on HADOOP-969:
--------------------------------------
+ 1 the patch fixes the problem.
> deadlock in job tracker RetireJobs
> ----------------------------------
>
> Key: HADOOP-969
> URL: https://issues.apache.org/jira/browse/HADOOP-969
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Priority: Critical
> Fix For: 0.11.0
>
> Attachments: jt-dead.patch
>
>
> The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like:
> "IPC Server handler 5 on 50020":
> at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)
> - waiting to lock <0x74487a80> (a java.util.Vector)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
> "retireJobs":
> at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
> - waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
> at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
> - locked <0x74487bb0> (a java.util.ArrayList)
> - locked <0x74487a80> (a java.util.Vector)
> - locked <0x74487a58> (a java.util.TreeMap)
> at java.lang.Thread.run(Thread.java:595)
> Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-969) deadlock in job tracker RetireJobs
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-969:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Owen!
> deadlock in job tracker RetireJobs
> ----------------------------------
>
> Key: HADOOP-969
> URL: https://issues.apache.org/jira/browse/HADOOP-969
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assigned To: Owen O'Malley
> Priority: Critical
> Fix For: 0.11.0
>
> Attachments: jt-dead.patch
>
>
> The JobTracker deadlocks because RetireJobs grabs locks in the wrong order. The call stacks look like:
> "IPC Server handler 5 on 50020":
> at org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker.java:1108)
> - waiting to lock <0x74487a80> (a java.util.Vector)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:992)
> - locked <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:337)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:538)
> "retireJobs":
> at org.apache.hadoop.mapred.JobTracker.removeJobTasks(JobTracker.java:782)
> - waiting to lock <0x744874b0> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.access$300(JobTracker.java:42)
> at org.apache.hadoop.mapred.JobTracker$RetireJobs.run(JobTracker.java:312)
> - locked <0x74487bb0> (a java.util.ArrayList)
> - locked <0x74487a80> (a java.util.Vector)
> - locked <0x74487a58> (a java.util.TreeMap)
> at java.lang.Thread.run(Thread.java:595)
> Found 1 deadlock.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.