You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Varun Saxena (JIRA)" <ji...@apache.org> on 2017/01/17 09:19:27 UTC

[jira] [Comment Edited] (YARN-5378) Accommodate app-id->cluster mapping

    [ https://issues.apache.org/jira/browse/YARN-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15825693#comment-15825693 ] 

Varun Saxena edited comment on YARN-5378 at 1/17/17 9:18 AM:
-------------------------------------------------------------

bq.  So, apart from listing down clusterId's given applicationId i.e YARN-6095, It would be better if there is an API to list down all the ClusterId's. Thoughts?
To be honest initially I had thought UI will only retrieve records within the scope of the cluster its running in. But with Federation, we may want to fetch records across multiple clusters.
Arguably user may have a static list of clusters. And that can be configurable at the client side (UI end). 
But ATS stores historical data and we may change around cluster IDs'. So we can probably have an endpoint listing the cluster IDs'. Probably adopt a similar approach to what we have done for listing entity types in YARN-5739.

Anyways the right place to have this discussion would be YARN-6095.


was (Author: varun_saxena):
bq.  So, apart from listing down clusterId's given applicationId i.e YARN-6095, It would be better if there is an API to list down all the ClusterId's. Thoughts?
To be honest initially I thought UI will only retrieve records within the scope of the cluster its running. But with Federation, we may want to fetch records across multiple clusters.
Arguably user may have a static list of clusters. And that can be configurable at the client side (UI end). 
But ATS stores historical data and we may change around cluster IDs'. So we can probably have an endpoint listing the cluster IDs'. Probably adopt a similar approach to what we have done for listing entity types in YARN-5739.

Anyways the right place to have this discussion would be YARN-6095.

> Accommodate app-id->cluster mapping
> -----------------------------------
>
>                 Key: YARN-5378
>                 URL: https://issues.apache.org/jira/browse/YARN-5378
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Joep Rottinghuis
>            Assignee: Sangjin Lee
>              Labels: yarn-5355-merge-blocker
>         Attachments: YARN-5378-YARN-5355.01.patch, YARN-5378-YARN-5355.02.patch, YARN-5378-YARN-5355.03.patch
>
>
> In discussion with [~sjlee0], [~vrushalic], [~subru], and [~curino] a use-case came up to be able to map from application-id to cluster-id in context of federation for Yarn.
> What happens is that a "random" cluster in the federation is asked to generate an app-id and then potentially a different cluster can be the "home" cluster for the AM. Furthermore, tasks can then run in yet other clusters.
> In order to be able to pull up the logical home cluster on which the application ran, there needs to be a mapping from application-id to cluster-id. This mapping is available in the federated Yarn case only during the active live of the application.
> A similar situation is common in our larger production environment. Somebody will complain about a slow job, some failure or whatever. If we're lucky we have an application-id. When we ask the user which cluster they ran on, they'll typically answer with the machine from where they launched the job (many users are unaware of the underlying physical clusters). This leaves us to spelunk through various RM ui's to find a matching epoch in the application ID. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org