You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2012/07/10 22:38:35 UTC
[jira] [Created] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Bikas Saha created MAPREDUCE-4427:
-------------------------------------
Summary: Enable the RM to work with AM's that are not managed by it
Key: MAPREDUCE-4427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Bikas Saha
Assignee: Bikas Saha
Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414228#comment-13414228 ]
Bikas Saha commented on MAPREDUCE-4427:
---------------------------------------
Actually I will remove the "lfs.mkdir(containerDir, null, true)" change. It is unrelated to this jira. It was a workaround to an existing bug that task launch will fail if there are no cache items associated with it because in that case localization is not triggered.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415028#comment-13415028 ]
Hudson commented on MAPREDUCE-4427:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1105 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1105/])
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease development of new applications. Contributed by Bikas Saha. (Revision 1361813)
Result = FAILURE
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361813
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/YarnApplicationState.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.1.0-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414568#comment-13414568 ]
Bikas Saha commented on MAPREDUCE-4427:
---------------------------------------
As mentioned above the findbugs is from the FairScheduler and unrelated to this change.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-4427:
-------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.1-alpha
Status: Resolved (was: Patch Available)
I just committed this. Thanks Bikas!
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410866#comment-13410866 ]
Arun C Murthy commented on MAPREDUCE-4427:
------------------------------------------
+1, this will be huge for people new to YARN who want to write their own apps etc.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414226#comment-13414226 ]
Bikas Saha commented on MAPREDUCE-4427:
---------------------------------------
The client side patch should exemplify the use case clearly.
Yes, we can implement a check mechanism if needed.
I am guessing that queue checks would depend on actual container resource allocation made to jobs and so would not count resources for these AM's because they are not allocated from cluster resources.
RM will do everything it does for a normal AM except cleanup the AM container. So task containers would be killed and AM unregistered from AppMasterService. If the AM actually continues running then it will get a REBOOT response from the AppMasterService on the next allocate() heartbeat and also will not get any more container assignments. So it will be practically useless.
Sorry, I forgot to remove the comment. Your understanding is correct. I will fix the comment.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420996#comment-13420996 ]
Kihwal Lee commented on MAPREDUCE-4427:
---------------------------------------
Nevermind. Arun fixed it in MAPREDUCE-4440.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.1.0-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413916#comment-13413916 ]
Arun C Murthy commented on MAPREDUCE-4427:
------------------------------------------
bq. Is there a way to make sure folks can't use this if admins chose?
Interesting point. I'd like to think through that requirement before adding it in - what were you thinking Thomas?
Maybe we can add that as an enhancement via a separate jira?
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413875#comment-13413875 ]
Bikas Saha commented on MAPREDUCE-4427:
---------------------------------------
The findbugs are from FairScheduler and unrelated to this patch.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414014#comment-13414014 ]
Thomas Graves commented on MAPREDUCE-4427:
------------------------------------------
note - I agree this is good stuff!
Sorry I meant add it to the client side of the app - which you've answered, thanks.
I was thinking of any multi-tenant cluster, where people could start abusing the option and running things on gateway machines and overloading those gateways or perhaps just causing traffic between outside machines that SEs don't want or expect. But thinking about this more, there are plenty of other ways to cause issues like that so I'm good with leaving this off. If a use case ever comes up we can revisit.
Another question or atleast something to think about - it appears it still goes through all the queue checks when submitting the application. I'm wondering if some of those checks might not apply in this case - for instance max am resources (maxActiveApplications) doesn't really apply because external AM's aren't using queue capacity for the AM itself. That might not be a big issue right now if people use this just for debug, but if this is used say by AM's to launch other AM's in arbitrary containers it might be more of an issue.
What happens when you kill one of these applications? The RM can't really force kill it - so does it just kill all containers its requested and "block" the AM from communicating.
minor nitpicky comments to consider:
Can we just remove the commented out code in the container executor: + //lfs.mkdir(containerDir, null, false);
Could you also clarify what the comment there means "+ // Without this app with no cache files cannot launch tasks"? Is it supposed to be "without this, app with" And then is "this" passing true into lfs.mkdir as last parameter?
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414792#comment-13414792 ]
Hudson commented on MAPREDUCE-4427:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2495 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2495/])
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease development of new applications. Contributed by Bikas Saha. (Revision 1361813)
Result = FAILURE
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361813
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/YarnApplicationState.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed MAPREDUCE-4427.
------------------------------------
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.0.2-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4427:
----------------------------------
Status: Patch Available (was: Open)
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4427:
----------------------------------
Attachment: MAPREDUCE-4427-2.patch
Attaching patch with some clarifying comments removed for final submission.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410984#comment-13410984 ]
Hadoop QA commented on MAPREDUCE-4427:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535920/MAPREDUCE-4427-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2569//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2569//console
This message is automatically generated.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413716#comment-13413716 ]
Thomas Graves commented on MAPREDUCE-4427:
------------------------------------------
I only skimmed the patch to see how to use this but don't see any documentation?'
Is there a way to make sure folks can't use this if admins chose?
Is there another jira to actually add this to an existing app master?
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4427:
----------------------------------
Status: Patch Available (was: Open)
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4427:
----------------------------------
Attachment: MAPREDUCE-4427-1.patch
Attaching initial patch
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4427:
----------------------------------
Status: Open (was: Patch Available)
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4427:
----------------------------------
Status: Open (was: Patch Available)
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Kihwal Lee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420993#comment-13420993 ]
Kihwal Lee commented on MAPREDUCE-4427:
---------------------------------------
TestClientRMService.testGetQueueInfo has been consistently failing since MAPREDUCE-4427.
MAPREDUCE-4471 has been filed.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.1.0-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414298#comment-13414298 ]
Hadoop QA commented on MAPREDUCE-4427:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536488/MAPREDUCE-4427-3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2592//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2592//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2592//console
This message is automatically generated.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415099#comment-13415099 ]
Hudson commented on MAPREDUCE-4427:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1138 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1138/])
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease development of new applications. Contributed by Bikas Saha. (Revision 1361813)
Result = FAILURE
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361813
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/YarnApplicationState.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.1.0-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413525#comment-13413525 ]
Arun C Murthy commented on MAPREDUCE-4427:
------------------------------------------
+1, lgtm! Thanks for the patch, this is very cool!
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4427:
----------------------------------
Status: Patch Available (was: Open)
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4427:
----------------------------------
Attachment: MAPREDUCE-4427-3.patch
Attach new patch based on comments
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413890#comment-13413890 ]
Bikas Saha commented on MAPREDUCE-4427:
---------------------------------------
[~alfthomas]
Documentation will come with the client side code that I will imminently add. Just opened MAPREDUCE-4438 for it.
The idea of queue level enable/disable seemed good but has been kept off for now because there does not seem to be a strong reason to require explicit approval. Perhaps you are thinking of some scenario I havent considered. Its not much work to add it if needed though.
There should be no change needed to an appmaster to be used in an unmanaged mode. It should be the same as running it on the cluster. Thats is key to enable the debug/dev-iteration scenario. The client side work will use distributed shell as an example.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414789#comment-13414789 ]
Hudson commented on MAPREDUCE-4427:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2475 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2475/])
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease development of new applications. Contributed by Bikas Saha. (Revision 1361813)
Result = SUCCESS
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361813
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/YarnApplicationState.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413547#comment-13413547 ]
Hadoop QA commented on MAPREDUCE-4427:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536339/MAPREDUCE-4427-2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 9 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2585//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2585//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2585//console
This message is automatically generated.
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4427) Enable the RM to work with AM's
that are not managed by it
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414790#comment-13414790 ]
Hudson commented on MAPREDUCE-4427:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2540 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2540/])
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease development of new applications. Contributed by Bikas Saha. (Revision 1361813)
Result = SUCCESS
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361813
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/YarnApplicationState.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
> Enable the RM to work with AM's that are not managed by it
> ----------------------------------------------------------
>
> Key: MAPREDUCE-4427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4427
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Labels: mrv2
> Fix For: 2.0.1-alpha
>
> Attachments: MAPREDUCE-4427-1.patch, MAPREDUCE-4427-2.patch, MAPREDUCE-4427-3.patch
>
>
> Currently, the RM itself manages the AM by allocating a container for it and negotiating the launch on the NodeManager and manages the AM lifecycle. Thereafter, the AM negotiates resources with the RM and launches tasks to do the real work.
> It would be a useful improvement to enhance this model by allowing the AM to be launched independently by the client without requiring the RM. These AM's would be launched on a gateway machine that can talk to the cluster. This would open up new use cases such as the following
> 1) Easy debugging of AM, specially during initial development. Having the AM launched on an arbitrary cluster node makes it hard to looks at logs or attach a debugger to the AM. If it can be launched locally then these tasks would be easier.
> 2) Running AM's that need special privileges that may not be available on machines managed by the NodeManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira