You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Vinod K V (JIRA)" <ji...@apache.org> on 2009/03/19 09:27:52 UTC
[jira] Created: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes HADOOP-5214 changes.
Deadlock triggered by FairScheduler scheduler's servlet due to changes HADOOP-5214 changes.
-------------------------------------------------------------------------------------------
Key: HADOOP-5534
URL: https://issues.apache.org/jira/browse/HADOOP-5534
Project: Hadoop Core
Issue Type: Bug
Components: contrib/fair-share
Affects Versions: 0.20.0
Reporter: Vinod K V
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala resolved HADOOP-5534.
--------------------------------------
Resolution: Fixed
Assignee: rahul k singh
Hadoop Flags: [Reviewed]
I just committed this to trunk and branch 0.20. Thanks, Rahul !
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HADOOP-5534.patch, HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683445#action_12683445 ]
Hemanth Yamijala commented on HADOOP-5534:
------------------------------------------
Looks fine to me.
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Attachments: HADOOP-5534.patch, HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683457#action_12683457 ]
Hemanth Yamijala commented on HADOOP-5534:
------------------------------------------
TestFairScheduler passed. Since this is contrib module, I didn't run the other tests.
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Attachments: HADOOP-5534.patch, HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5534:
----------------------------------
Attachment: HADOOP-5534.patch
changed indendation for smoe of the line
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Attachments: HADOOP-5534.patch, HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12684009#action_12684009 ]
Hudson commented on HADOOP-5534:
--------------------------------
Integrated in Hadoop-trunk #785 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/785/])
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Assignee: rahul k singh
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HADOOP-5534.patch, HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683455#action_12683455 ]
Vinod K V commented on HADOOP-5534:
-----------------------------------
+1.
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Attachments: HADOOP-5534.patch, HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-5534:
-------------------------------------
Priority: Blocker (was: Major)
Fix Version/s: 0.20.0
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HADOOP-5534.patch, HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated HADOOP-5534:
------------------------------
Summary: Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214. (was: Deadlock triggered by FairScheduler scheduler's servlet due to changes HADOOP-5214 changes.)
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683456#action_12683456 ]
Hemanth Yamijala commented on HADOOP-5534:
------------------------------------------
Results of test-patch:
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
The -1 is because this is a synchronization fix which can't be easily tested in a test case.
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Attachments: HADOOP-5534.patch, HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "rahul k singh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
rahul k singh updated HADOOP-5534:
----------------------------------
Attachment: HADOOP-5534.patch
have made the synchronization block more fine grained and removed scheduler.update from the synchronized block.
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
> Attachments: HADOOP-5534.patch
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683373#action_12683373 ]
Hemanth Yamijala commented on HADOOP-5534:
------------------------------------------
Actually, it seems the issue was caused because two patches were changing the source at roughly the same time. In fact, I committed HADOOP-5154 *after* I committed HADOOP-5214. So, more accurately, that commit broke things. Sigh.
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5534) Deadlock triggered by FairScheduler
scheduler's servlet due to changes from HADOOP-5214.
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683359#action_12683359 ]
Vinod K V commented on HADOOP-5534:
-----------------------------------
{code}
Found one Java-level deadlock:
=============================
"1774437339@qtp0-45":
waiting to lock monitor 0x0000000044597ee8 (object 0x00002aab6eae1c20, a org.apache.hadoop.mapred.FairScheduler),
which is held by "758568398@qtp0-35"
"758568398@qtp0-35":
waiting to lock monitor 0x00000000448d7428 (object 0x00002aab6eac86b0, a org.apache.hadoop.mapred.JobTracker),
which is held by "IPC Server handler 30 on 50300"
"IPC Server handler 30 on 50300":
waiting to lock monitor 0x0000000044597ee8 (object 0x00002aab6eae1c20, a org.apache.hadoop.mapred.FairScheduler),
which is held by "758568398@qtp0-35"
Java stack information for the threads listed above:
===================================================
"1774437339@qtp0-45":
at org.apache.hadoop.mapred.FairSchedulerServlet.showPools(FairSchedulerServlet.java:146)
- waiting to lock <0x00002aab6eae1c20> (a org.apache.hadoop.mapred.FairScheduler)
at org.apache.hadoop.mapred.FairSchedulerServlet.doGet(FairSchedulerServlet.java:134)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
"758568398@qtp0-35":
at org.apache.hadoop.mapred.JobTracker.getClusterStatus(JobTracker.java:2849)
- waiting to lock <0x00002aab6eac86b0> (a org.apache.hadoop.mapred.JobTracker)
at org.apache.hadoop.mapred.FairScheduler.update(FairScheduler.java:337)
at org.apache.hadoop.mapred.FairSchedulerServlet.doGet(FairSchedulerServlet.java:114)
- locked <0x00002aab6eae1c20> (a org.apache.hadoop.mapred.FairScheduler)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
"IPC Server handler 30 on 50300":
at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:222)
- waiting to lock <0x00002aab6eae1c20> (a org.apache.hadoop.mapred.FairScheduler)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2368)
- locked <0x00002aab6eac86b0> (a org.apache.hadoop.mapred.JobTracker)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
Found 1 deadlock.
{code}
> Deadlock triggered by FairScheduler scheduler's servlet due to changes from HADOOP-5214.
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-5534
> URL: https://issues.apache.org/jira/browse/HADOOP-5534
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.0
> Reporter: Vinod K V
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.