[jira] [Commented] (YARN-1039) Add parameter for YARN resource requests to indicate "long lived"

["Carlo Curino (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310328#comment-14310328 ] 

Carlo Curino commented on YARN-1039:
------------------------------------

Tossing some fire back on duration. I read your concerns of applications ability to provide good values, 
however, I'd rather have the app providing their best duration estimate (and the framework rounding it 
or bucketing it), than the app providing a coarse grained tag-based version in the first place. 

Changing cluster configurations and policies might turn what used to be a "short" task into something 
not that short, which we want to handle differently and so on. In a sense asking for "duration" prevent 
us to rely on what application will judge as "short/long" etc.. 

As another example, based on whatever mechanisms for log aggregation we will have in the future, 
we can change our mind about what are the "cut-points" for short/long etc.. For example, because a
new technique makes it very cheap and we want to provide much more frequent feedback to users.

Bottom line, I find duration a rather "neutral" thing to ask, vs something which is more "opinion-based",
and corner cases like never-ending services are easily handled with -1 or +inf values.

I also agree that there are many other use cases for tags, that emerged in the discussion, which have
a clear value and are by no means covered by duration.



> Add parameter for YARN resource requests to indicate "long lived"
> -----------------------------------------------------------------
>
>                 Key: YARN-1039
>                 URL: https://issues.apache.org/jira/browse/YARN-1039
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>    Affects Versions: 3.0.0, 2.1.1-beta
>            Reporter: Steve Loughran
>            Assignee: Craig Welch
>         Attachments: YARN-1039.1.patch, YARN-1039.2.patch, YARN-1039.3.patch
>
>
> A container request could support a new parameter "long-lived". This could be used by a scheduler that would know not to host the service on a transient (cloud: spot priced) node.
> Schedulers could also decide whether or not to allocate multiple long-lived containers on the same node



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)