You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Milind Bhandarkar (Commented) (JIRA)" <ji...@apache.org> on 2011/10/24 19:56:34 UTC

[jira] [Commented] (MAPREDUCE-3251) JobClient should have an option to only to talk to RM+HistoryServer to get job status

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134296#comment-13134296 ] 

Milind Bhandarkar commented on MAPREDUCE-3251:
----------------------------------------------

This proposal, as stated, breaks the separation of concerns. RM should not be aware of MR jobs. Proper wrappers need to be provided to keep RM agnostic of jobs. Thus, AM -> RM should provide a K-V status message to RM. The RM client (wrapped by JC) should allow fetching such K-V status specifying a key (which wraps jobid), and the MR-specific portion of JC should interpret the value as a job status. In my understanding of the current code, I do not see such abstractions in RM.
                
> JobClient should have an option to only to talk to RM+HistoryServer to get job status
> -------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3251
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3251
>             Project: Hadoop Map/Reduce
>          Issue Type: Task
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Anupam Seth
>            Priority: Blocker
>             Fix For: 0.23.0
>
>
> In 0.20.xxx, the JobClient while polling goes to JT to get the job status. With YARN, AM can be launched on any port and the client will have to have ACL open to that port to talk to AM and get the job status. When the client is within the same grid network access to AM is not a problem. But some applications may have one installation per set of clusters and may launch jobs even across such sets (on job trackers in another set of clusters). For that to work only the JT port needs to be open currently. In case of YARN, all ports will have to be opened up for things to work. That would be a security no-no.
> There are two possible solutions:
>   1) Make the job client only talk to RM (as an option) to get the job status. 
>   2) Limit the range of ports AM can listen on.
> Option 2) may not be favorable as there is no direct OS API to find a free port.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira