You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2011/06/29 08:39:28 UTC
[jira] [Created] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
-------------------------------------------------------------
Key: MAPREDUCE-2630
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Environment: All
Reporter: Josh Wills
Priority: Minor
The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-2630) MR-279: refreshQueues leads
to NPEs when used w/FifoScheduler
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAPREDUCE-2630 started by Josh Wills.
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630-simple.patch, MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2630:
----------------------------------
Attachment: MAPREDUCE-2630.patch
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2630:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just pushed this. Thanks Josh.
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2630-simple.patch, MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2630:
----------------------------------
Attachment: MAPREDUCE-2630-simple.patch
Simpler patch that just updates the impl of FifoScheduler to make it differentiate between initialization and re-initialization, ala the CapacityScheduler.
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630-simple.patch, MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2630:
----------------------------------
Attachment: (was: MAPREDUCE-2630.patch)
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059291#comment-13059291 ]
Josh Wills commented on MAPREDUCE-2630:
---------------------------------------
No worries Arun- it was a hectic week. Really enjoyed your talk at the Summit, lots of good questions from the audience.
My preference for the two separate methods was based on the idea that the containerTokenSecretManager and clusterTracker really should never change once the scheduler is init'd, and that having the two separate methods would prevent any other scheduler impls that came along from introducing the same kind of error. If that isn't true, i.e., there is some situation where we would want to change the value of the containerTokenSecretManager/clusterTracker, I'm fine to have the single reinit method.
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned MAPREDUCE-2630:
----------------------------------------
Assignee: Josh Wills
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2630:
----------------------------------
Attachment: MAPREDUCE-2630.patch
Modified patch to include new tests.
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063973#comment-13063973 ]
Hadoop QA commented on MAPREDUCE-2630:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12485614/MAPREDUCE-2630-simple.patch
against trunk revision 1145424.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/456//console
This message is automatically generated.
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630-simple.patch, MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2630:
----------------------------------
Status: Patch Available (was: In Progress)
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Assignee: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630-simple.patch, MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2630) MR-279: refreshQueues leads to
NPEs when used w/FifoScheduler
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059261#comment-13059261 ]
Arun C Murthy commented on MAPREDUCE-2630:
------------------------------------------
Josh, sorry, took me a while to get to this.
Thanks for the patch.
Could we just add a check to FifoScheduler.reinit ala CS.reinit? That way we can keep a single method in Scheduler. I'm happy to rename it 'configure' or some such. Thoughts?
> MR-279: refreshQueues leads to NPEs when used w/FifoScheduler
> -------------------------------------------------------------
>
> Key: MAPREDUCE-2630
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2630
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Environment: All
> Reporter: Josh Wills
> Priority: Minor
> Attachments: MAPREDUCE-2630.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The RM's admin service exposes a method refreshQueues that is used to update the queue configuration when used with the CapacityScheduler, but if it is used with the FifoScheduler, it will set the containerTokenSecretManager/clusterTracker fields on the FifoScheduler to null, which eventually leads to NPE. Since the FifoScheduler only has one queue that cannot be refreshed, the correct behavior is for the refreshQueues call to be a no-op.
> I will attach a patch that fixes this by splitting the ResourceScheduler's reinitialize method into separate initialize/updateQueues methods.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira