You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2011/08/09 00:36:27 UTC
[jira] [Created] (HADOOP-7529) Possible deadlock in metrics2
Possible deadlock in metrics2
-----------------------------
Key: HADOOP-7529
URL: https://issues.apache.org/jira/browse/HADOOP-7529
Project: Hadoop Common
Issue Type: Bug
Components: metrics
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Priority: Critical
Fix For: 0.23.0
Attachments: metrics-deadlock.png
Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083921#comment-13083921 ]
Todd Lipcon commented on HADOOP-7529:
-------------------------------------
+1, patch looks good to me. Thanks Luke!
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, hadoop-7529-v2.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083716#comment-13083716 ]
Luke Lu commented on HADOOP-7529:
---------------------------------
bq. Yep, the wiki is here: http://wiki.apache.org/hadoop/HowToUseJCarder
Thanks. I was hoping for a simpler process/script to reproduce these graphs though :) I agree with you on the wiki that it's important to avoid false positives to reduce noise. These two cycles are already borderline and practically harmless, as unlike start/stop, init/shutdown doesn't overlap registration.
bq. I'm working on getting access to the new build infrastructure so I can set up hudson jobs to run jcarder on a regular basis
This will be a time saver. +1 and Thanks!
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, hadoop-7529-v2.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Lu updated HADOOP-7529:
----------------------------
Attachment: hadoop-7529-v1.patch
Though it's unlikely to happen in practice (metrics system initialization usually doesn't overlap with metrics source registration), it's probably worth fixing. The v1 patch make metrics impl in the DefaultMetricsSystem an AtomicReference.
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, metrics-deadlock.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081342#comment-13081342 ]
Luke Lu commented on HADOOP-7529:
---------------------------------
Though it's unlikely to happen in practice (metrics system initialization usually doesn't overlap with metrics source registration), it's probably worth fixing for posterity. The v1 patch make metrics impl in the DefaultMetricsSystem an AtomicReference.
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, metrics-deadlock.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-7529:
--------------------------------
Attachment: metrics-deadlock.png
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: metrics-deadlock.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083285#comment-13083285 ]
Todd Lipcon commented on HADOOP-7529:
-------------------------------------
+1, I verified that with this patch, the jcarder cycle disappears. And the patch looks reasonable, though I don't know this area of the code well.
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, metrics-deadlock.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Lu updated HADOOP-7529:
----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed v2 to trunk. Thanks Todd for reviewing!
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, hadoop-7529-v2.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Lu updated HADOOP-7529:
----------------------------
Attachment: hadoop-7529-v2.patch
v2 patch fixes the cycle in shutdown as well.
Todd, do you have a wiki page on how to run the jcarder on hadoop?
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, hadoop-7529-v2.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Lu updated HADOOP-7529:
----------------------------
Status: Patch Available (was: Open)
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, hadoop-7529-v2.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-7529:
--------------------------------
Attachment: metrics-shutdown.png
Actually, I take it back. It fixes the first cycle, but there's a second one, attached here.
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Lu updated HADOOP-7529:
----------------------------
Status: Open (was: Patch Available)
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084279#comment-13084279 ]
Hudson commented on HADOOP-7529:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #741 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/741/])
HADOOP-7529. Fix lock cycles in metrics system. (llu)
llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1157187
Files :
* /hadoop/common/trunk/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/DefaultMetricsSystem.java
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, hadoop-7529-v2.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Lu reassigned HADOOP-7529:
-------------------------------
Assignee: Luke Lu
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, metrics-deadlock.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083665#comment-13083665 ]
Todd Lipcon commented on HADOOP-7529:
-------------------------------------
Yep, the wiki is here: http://wiki.apache.org/hadoop/HowToUseJCarder
Recently I've been using the more experimental "lockclasses" branch from my github, which supports analyzing readwrite locks, but the rest of the instructions should be the same. I'm working on getting access to the new build infrastructure so I can set up hudson jobs to run jcarder on a regular basis
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Luke Lu
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, hadoop-7529-v2.patch, metrics-deadlock.png, metrics-shutdown.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7529) Possible deadlock in metrics2
Posted by "Luke Lu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luke Lu updated HADOOP-7529:
----------------------------
Status: Patch Available (was: Open)
> Possible deadlock in metrics2
> -----------------------------
>
> Key: HADOOP-7529
> URL: https://issues.apache.org/jira/browse/HADOOP-7529
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Priority: Critical
> Fix For: 0.23.0
>
> Attachments: hadoop-7529-v1.patch, metrics-deadlock.png
>
>
> Lock cycle detected by jcarder between MetricsSystemImpl and DefaultMetricsSystem
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira