[jira] [Commented] (YARN-168) No way to turn off virtual memory limits without turning off physical memory limits

["Krishna Kishore Bonagiri (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13632868#comment-13632868 ] 

Krishna Kishore Bonagiri commented on YARN-168:
-----------------------------------------------

How can I get this fix if I want it now? When would the next release be? I mean, the one having this fix!
                
> No way to turn off virtual memory limits without turning off physical memory limits
> -----------------------------------------------------------------------------------
>
>                 Key: YARN-168
>                 URL: https://issues.apache.org/jira/browse/YARN-168
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 3.0.0
>            Reporter: Harsh J
>
> Asked and reported by a user (Krishna) on ML:
> {quote}
> This is possible to do, but you've hit a bug with the current YARN
> implementation. Ideally you should be able to configure the vmem-pmem
> ratio (or an equivalent config) to be -1, to indicate disabling of
> virtual memory checks completely (and there's indeed checks for this),
> but it seems like we are enforcing the ratio to be at least 1.0 (and
> hence negatives are disallowed).
> You can't workaround by setting the NM's offered resource.mb to -1
> either, as you'll lose out on controlling maximum allocations.
> Please file a YARN bug on JIRA. The code at fault lies under
> ContainersMonitorImpl#init(…).
> On Thu, Oct 18, 2012 at 4:00 PM, Krishna Kishore Bonagiri
> <wr...@gmail.com> wrote:
> > Hi,
> >
> >   Is there a way we can ask the YARN RM for not killing a container when it
> > uses excess virtual memory than the maximum it can use as per the
> > specification in the configuration file yarn-site.xml? We can't always
> > estimate the amount of virtual memory needed for our application running on
> > a container, but we don't want to get it killed in a case it exceeds the
> > maximum limit.
> >
> >   Please suggest as to how can we come across this issue.
> >
> > Thanks,
> > Kishore
> {quote}
> Basically, we're doing:
> {code}
>     // ///////// Virtual memory configuration //////
>     float vmemRatio = conf.getFloat(
>         YarnConfiguration.NM_VMEM_PMEM_RATIO,
>         YarnConfiguration.DEFAULT_NM_VMEM_PMEM_RATIO);
>     Preconditions.checkArgument(vmemRatio > 0.99f,
>         YarnConfiguration.NM_VMEM_PMEM_RATIO +
>         " should be at least 1.0");
>     this.maxVmemAllottedForContainers =
>       (long)(vmemRatio * maxPmemAllottedForContainers);
> {code}
> For virtual memory monitoring to be disabled, maxVmemAllottedForContainers has to be -1. For that to be -1, given the above buggy computation, vmemRatio must be -1 or maxPmemAllottedForContainers must be -1.
> If vmemRatio were -1, we fail the precondition check and exit.
> If maxPmemAllottedForContainers, we also end up disabling physical memory monitoring.
> Or perhaps that makes sense - to disable both physical and virtual memory monitoring, but that way your NM becomes infinite in resource grants, I think.
> We need a way to selectively disable kills done via virtual memory monitoring, which is the base request here.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira