You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Junping Du (JIRA)" <ji...@apache.org> on 2012/07/18 05:27:34 UTC
[jira] [Created] (MAPREDUCE-4452) Several Findbugs issues with new
FairScheduler in YARN
Junping Du created MAPREDUCE-4452:
-------------------------------------
Summary: Several Findbugs issues with new FairScheduler in YARN
Key: MAPREDUCE-4452
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: scheduler
Affects Versions: 2.1.0-alpha
Reporter: Junping Du
Assignee: Junping Du
The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419063#comment-13419063 ]
Hadoop QA commented on MAPREDUCE-4452:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537332/MAPREDUCE-4452-v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2635//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2635//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2635//console
This message is automatically generated.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420133#comment-13420133 ]
Junping Du commented on MAPREDUCE-4452:
---------------------------------------
In new v3 patch, separating monitor objects from attributes to get rid of ML_SYNC_ON_FIELD_TO_GUARD_CHANGING_THAT_FIELD which is due to updating on lock object. Also, add synchronization to getter/setter of userMaxAppsDefault and queueMaxAppsDefault to get ride of FindBugs warning.(in fact, it is not necessary because this is int type and should be finished in one CPU instruction. However, it is not much overhead as lock level is fine-grained).
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417959#comment-13417959 ]
Patrick Wendell commented on MAPREDUCE-4452:
--------------------------------------------
The warnings have been suppressed manually, so let's hold on with this proposed fix and see if we can justify avoiding synchronization here.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new
FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated MAPREDUCE-4452:
----------------------------------
Attachment: MAPREDUCE-4452-v3.patch
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417243#comment-13417243 ]
Karthik Kambatla commented on MAPREDUCE-4452:
---------------------------------------------
Thanks for the patch, Junping.
The patch definitely fixes the findbugs warnings. However, the patch converts all methods reported by findbugs to synchronized. In cases where findbugs reports a false positive for bugs, that might affect performance. It might be best to commit this patch, but create a separate JIRA to de-synchronize methods. For every method we de-synchronize, we probably need a justification (in comments) as to why it doesn't create a buggy scenario.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420119#comment-13420119 ]
Hadoop QA commented on MAPREDUCE-4452:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537479/MAPREDUCE-4452-v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 10 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2643//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2643//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2643//console
This message is automatically generated.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419701#comment-13419701 ]
Patrick Wendell commented on MAPREDUCE-4452:
--------------------------------------------
What really confuses me here is that you've added _new locks_ for (queueMaxApps, userMaxApps, queueWeights, etc) in this patch rather than synchronizing them under the existing lock that's held when they are set.
Really this doesn't ensure that they won't be concurrently read/modified any more than before, but it looks like findbugs just check's that variables are consistently read or modified under _some lock_ rather than under the same lock. That's odd to me.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420120#comment-13420120 ]
Junping Du commented on MAPREDUCE-4452:
---------------------------------------
Patrick, you are right that the threads of modifying and reading are pending on different lock object/attribute in my previous patch, so I update to v3 patch so both of them are synchronized on related attributes. The intention to get rid of synchronized getter methods (as v1 patch) is that will make QueueManager instance can only be accessed by one thread per time although different threads are reading/modifying different attributes. I guess that is the right way to do it. What do you think?
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new
FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated MAPREDUCE-4452:
----------------------------------
Attachment: MAPREDUCE-4452-v1.patch
Update for one FindBug issue. As this patch is to fix Findbugs issue, so no additional test is attached.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419060#comment-13419060 ]
Junping Du commented on MAPREDUCE-4452:
---------------------------------------
Hi Patrick, Looks like the warnings are showed up in current precommit test again so that we don't suppressed that any more? I think it is still risky to have no any protection for thread-safe. The reason is: it is possible for update thread and refreshQueue operation to call reloadAllocs() for updating these properties while other thread is reading. To lower the performance overhead, may be we can reduce lock granularity as new attached patch (v2). What do you think?
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420136#comment-13420136 ]
Hadoop QA commented on MAPREDUCE-4452:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537485/MAPREDUCE-4452-v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2644//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2644//console
This message is automatically generated.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419345#comment-13419345 ]
Junping Du commented on MAPREDUCE-4452:
---------------------------------------
Hi Patrick, For warning suppression, that's my previous understanding too. I also see some other precommit test works fine. But the interesting thing is it doesn't work on my PreCommit test in MAPREDUCE-4309 today and the same 4 Findbugs error comes out. :)
I will mark this issue as reviewed and good to see someone can commit it.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Patrick Wendell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419334#comment-13419334 ]
Patrick Wendell commented on MAPREDUCE-4452:
--------------------------------------------
Hey Junping,
Apologies - my understanding was that the warning suppression was in place and working.
This looks great, feel free to commit it. I can look at lock-free approaches to this later, but let's get the warnings under control.
Thanks,
- Patrick
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new
FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated MAPREDUCE-4452:
----------------------------------
Attachment: MAPREDUCE-4452.patch
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419200#comment-13419200 ]
Junping Du commented on MAPREDUCE-4452:
---------------------------------------
The left two warning in Findbugs is not a issue as get/set primitive is atomic if the size of the variable is less that or equal to 32 bits. Here, both userMaxAppsDefault and queueMaxAppsDefault are int type so accessing them can still keep atomic.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new
FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated MAPREDUCE-4452:
----------------------------------
Attachment: (was: MAPREDUCE-4452-v3.patch)
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416955#comment-13416955 ]
Hadoop QA commented on MAPREDUCE-4452:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536965/MAPREDUCE-4452-v1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2620//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2620//console
This message is automatically generated.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new
FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated MAPREDUCE-4452:
----------------------------------
Status: Patch Available (was: Open)
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new
FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated MAPREDUCE-4452:
----------------------------------
Attachment: MAPREDUCE-4452-v3.patch
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452-v3.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4452) Several Findbugs issues with
new FairScheduler in YARN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416890#comment-13416890 ]
Hadoop QA commented on MAPREDUCE-4452:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536951/MAPREDUCE-4452.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2619//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2619//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2619//console
This message is automatically generated.
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4452) Several Findbugs issues with new
FairScheduler in YARN
Posted by "Junping Du (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du updated MAPREDUCE-4452:
----------------------------------
Attachment: MAPREDUCE-4452-v2.patch
> Several Findbugs issues with new FairScheduler in YARN
> ------------------------------------------------------
>
> Key: MAPREDUCE-4452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 2.1.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: MAPREDUCE-4452-v1.patch, MAPREDUCE-4452-v2.patch, MAPREDUCE-4452.patch
>
>
> The good feature of FairScheduler is added recently to YARN. As recently PreCommit test from MAPREDUCE-4309, there are several bugs found by Findbugs related to FairScheduler:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.shutdown() might ignore java.lang.Exception
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairSchedulerEventLog.logDisabled; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.queueMaxAppsDefault; locked 50% of time
> Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueueManager.userMaxAppsDefault; locked 50% of time
> The details are in:https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html#DE_MIGHT_IGNORE
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira