You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/10/11 00:56:29 UTC
[jira] [Created] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Separate application-init and container-init event types in NM's ApplicationImpl FSM
------------------------------------------------------------------------------------
Key: MAPREDUCE-3162
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.23.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3162:
-----------------------------------------------
Attachment: MAPREDUCE-3162-20111019.txt
Trivial update to wrap around lines longer than 80 chars.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130389#comment-13130389 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3162:
----------------------------------------------------
bq. -1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
MAPREDUCE-3209
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130622#comment-13130622 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #835 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/835/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185988
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130404#comment-13130404 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #23 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/23/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
svn merge -c r1185988 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130379#comment-13130379 ]
Hadoop QA commented on MAPREDUCE-3162:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499638/MAPREDUCE-3162-20111019.1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1063//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1063//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1063//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1063//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1063//console
This message is automatically generated.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124699#comment-13124699 ]
Todd Lipcon commented on MAPREDUCE-3162:
----------------------------------------
Thanks for the comments. I fixed this locally by handling INIT_CONTAINER events during NEW state the same as during INITING (ie by queueing them up).
Do you think that's worse? The whole "DuplicateAppInitTransition" thing was very confusing to me as I first looked at the code.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127809#comment-13127809 ]
Todd Lipcon commented on MAPREDUCE-3162:
----------------------------------------
Sorry, my bad - the race from earlier as I commented above.
Compared to the existing code, the improvement is that it cleans up the lifecycle to make more sense to people reading the code, and makes further patches that treat application initialization differently from container initialization easier, as discussed above.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3162:
-----------------------------------------------
Attachment: MAPREDUCE-3162-20111019.1.txt
Suppressing the javac warning.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130400#comment-13130400 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1108 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1108/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185988
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3162:
-----------------------------------------------
Status: Open (was: Patch Available)
Alright the patch looks good, the race condition from the previous one won't be there anymore.
+1.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124701#comment-13124701 ]
Todd Lipcon commented on MAPREDUCE-3162:
----------------------------------------
bq. More of the init used to be per-application, before changing the localization to be per-container
The issue that got me looking at this code was MAPREDUCE-3159 -- a place where the per-application action of rming the appcachedir was actually being run on every container. But it makes sense that we can avoid the shell-out to the localizer on every container initialization - it's not super expensive but the fork isn't free either. In the case that a container has no new resources associated with it, we shouldn't need to spawn one, right?
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Chris Douglas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127784#comment-13127784 ]
Chris Douglas commented on MAPREDUCE-3162:
------------------------------------------
bq. Here's a patch which fixes the observed race, by queueing up events in INITTING and NEW states both.
Sorry, a race in the existing code or the race from [earlier|https://issues.apache.org/jira/browse/MAPREDUCE-3162?focusedCommentId=13124678&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13124678]?
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3162:
-----------------------------------------------
Status: Open (was: Patch Available)
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3162:
-----------------------------------------------
Fix Version/s: 0.23.0
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130595#comment-13130595 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #865 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/865/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185988
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124758#comment-13124758 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3162:
----------------------------------------------------
The deletion of app-dirs on each startLocalizer() is only happening in DCE case. And it seems like a copy-paste from the legacy code from TT. I don't see us deleting app-dirs like that in LCE case. The corner case mentioned there is also from the MRV1 TT world ("this will happen on a partial execution of localizeJob."), not sure if it even makes any sense now. Dropping that delete should do the trick for you at MAPREDUCE-3159?
Regarding the fork, in DCE, we are not even forking, we invoke the localizer directly. Your comment about avoiding unnecessary fork still applies to LCE though. But we cannot just avoid the fork directly without other related changes. One problem we've known from before is that since the app-dirs creation etc are done during the localization itself, not forking when there are no resources for a container would mean no app-dirs and so containers cannot even launch! *sigh*
We definitely need to move the creation of app-dirs etc out of resource-localization. That's a known issue but one which I delayed due to the bigger effort needed.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124760#comment-13124760 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3162:
----------------------------------------------------
Didn't see your comment, we are at a commenting race here :)
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-3162:
-----------------------------------
Attachment: mr-3162.txt
This patch depends on MR-3161
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3162:
-----------------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch-0.23. Thanks Todd!
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130569#comment-13130569 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #44 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/44/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
svn merge -c r1185988 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-3162:
-----------------------------------
Attachment: mr-3162.txt
Here's a patch which fixes the observed race, by queueing up events in INITTING and NEW states both.
I was able to get through the map phase of a terasort on 20 nodes with this patch. Reduce phase had some issues, but pretty sure it's unrelated, given this is a yarn patch.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130403#comment-13130403 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #22 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/22/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
svn merge -c r1185988 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127787#comment-13127787 ]
Hadoop QA commented on MAPREDUCE-3162:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499086/mr-3162.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 20 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1022//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1022//console
This message is automatically generated.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130422#comment-13130422 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1127 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1127/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185988
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124678#comment-13124678 ]
Todd Lipcon commented on MAPREDUCE-3162:
----------------------------------------
there's a bit of a bug in this right now: startContainer can be called concurrently from several threads, apparently. So one thread can insert the ApplicationImpl in the concurrent applications map, then another thread can send the INIT_CONTAINER event before the INIT_APPLICATION event has been sent. Thinking about a good fix.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124688#comment-13124688 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3162:
----------------------------------------------------
Yes, that was the reason why Chris/I originally made INIT_APPLICATION idempotent and handled it like this. This is a manifestation of the fact that we don't have a separate {{startApplication()}} API along with {{startContainer()}}.
I can understand the clarity of separate events, but not sure if there is any way out. Will let you reach to the conclusion ;)
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124755#comment-13124755 ]
Todd Lipcon commented on MAPREDUCE-3162:
----------------------------------------
Right. The issue here is that "startLocalizer" not only handles localizing resources, but also mkdirs/chowns the appCachedir, log dirs, etc. So we need to have some way of making sure that stuff happens once (preferably exactly once), in addition to being lazy about launching the actual localizer subprocess. Does that make any sense?
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130603#comment-13130603 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #56 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/56/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
svn merge -c r1185988 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130399#comment-13130399 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1187 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1187/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185988
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Chris Douglas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124696#comment-13124696 ]
Chris Douglas commented on MAPREDUCE-3162:
------------------------------------------
bq. startContainer can be called concurrently from several threads, apparently. So one thread can insert the ApplicationImpl in the concurrent applications map, then another thread can send the INIT_CONTAINER event before the INIT_APPLICATION event has been sent.
That was the intent. It should also be legal for the NM to clean up application state, given a richer set of semantics for the intermediate output specifying retention. Then one would have to re-init the application to start containers there. More of the init used to be per-application, before changing the localization to be per-container. It's not doing much now, but it's cheap.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130429#comment-13130429 ]
Hudson commented on MAPREDUCE-3162:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #23 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/23/])
MAPREDUCE-3162. Separated application-init and container-init event types in NodeManager's Application state machine. Contributed by Todd Lipcon.
svn merge -c r1185988 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationContainerInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/ApplicationInitEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/application/TestApplication.java
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130344#comment-13130344 ]
Hadoop QA commented on MAPREDUCE-3162:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499628/MAPREDUCE-3162-20111019.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1707 javac compiler warnings (more than the trunk's current 1706 warnings).
-1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.TestJobMonitorAndPrint
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1061//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1061//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1061//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1061//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1061//console
This message is automatically generated.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3162:
-----------------------------------------------
Status: Patch Available (was: Open)
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3162-20111019.1.txt, MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3162:
-----------------------------------------------
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
Resubmitting this to Jenkins. The previous javac failure reported was transient.
Will commit this once Jenkins gives its blessings.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: MAPREDUCE-3162-20111019.txt, mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated MAPREDUCE-3162:
-----------------------------------
Status: Patch Available (was: Open)
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Chris Douglas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124732#comment-13124732 ]
Chris Douglas commented on MAPREDUCE-3162:
------------------------------------------
bq. Do you think that's worse? The whole "DuplicateAppInitTransition" thing was very confusing to me as I first looked at the code.
The naming sounds error-ish.
bq. it makes sense that we can avoid the shell-out to the localizer on every container initialization - it's not super expensive but the fork isn't free either. In the case that a container has no new resources associated with it, we shouldn't need to spawn one, right?
That's true, but a better fix would lazily launch the localizer. Even if it's missing resources, it may not need to run if other containers are covering its resource set. Creating the stub is cheap.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3162:
-------------------------------------
Component/s: nodemanager
mrv2
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt, mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3162) Separate application-init and
container-init event types in NM's ApplicationImpl FSM
Posted by "Chris Douglas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124791#comment-13124791 ]
Chris Douglas commented on MAPREDUCE-3162:
------------------------------------------
bq. The issue here is that "startLocalizer" not only handles localizing resources, but also mkdirs/chowns the appCachedir, log dirs, etc. So we need to have some way of making sure that stuff happens once (preferably exactly once), in addition to being lazy about launching the actual localizer subprocess. Does that make any sense?
Sure. We went back and forth on this, but the operations saved by adding a fast path for init are pretty minor (one hopes) given the cost of a container. Asserting every precondition for container launch _on_ every container launch is often redundant (particularly for MR), but making it idempotent seemed cleaner than adding all the logic to avoid it. That section also has a lot of rough patches; it's a minor crime for that neighborhood.
bq. We definitely need to move the creation of app-dirs etc out of resource-localization.
It's a pretty inexpensive operation and pushing the dir creation into the resource localization puts all that into a single fork for the LCE (required root privs to create the userdir). But yea, it's usually waste.
> Separate application-init and container-init event types in NM's ApplicationImpl FSM
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3162
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3162
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Attachments: mr-3162.txt
>
>
> Currently, the ApplicationImpl receives an INIT_APPLICATION event on every container initialization. Only on the first one does it really mean to init the application, whereas all subsequent events are for specific containers. This JIRA is to separate the events into INIT_APPLICATION, sent once and only once per application, and INIT_CONTAINER, which is sent for every container. The first container sends INIT_APPLICATION followed by INIT_CONTAINER.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira