[jira] [Commented] (TWILL-252) Not providing any feedback when size of the container requested can't be allocated

["Terence Yim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TWILL-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295294#comment-16295294 ] 

Terence Yim commented on TWILL-252:
-----------------------------------

Do you expect the application to fail to launch due to unable to acquire container of the desired size? It can be a static check before the application is submitted to YARN.

> Not providing any feedback when size of the container requested can't be allocated
> ----------------------------------------------------------------------------------
>
>                 Key: TWILL-252
>                 URL: https://issues.apache.org/jira/browse/TWILL-252
>             Project: Apache Twill
>          Issue Type: Bug
>            Reporter: Yuliya Feldman
>
> Looks like when YARN is configured with max memory per container (yarn.scheduler.maximum-allocation-mb) less then amount of memory end user allocates for their application and container is allocated with just yarn.scheduler.maximum-allocation-mb value there is no way to know about it until container is allocated.
> We try to divide memory into heap and off-heap and end up with setting up off heap to the value higher then allocated for the container, as application assumes it gets what it asked for or container is not allocated at all.
> Need either ability to fail application in this case or not allocate container with memory less then asked.
> As currently Twill adjusts memory and cpu with only INFO level messages in AppMaster log:
> from Hadoop21YarnAMClient.java
> {code:java}
>  protected Resource adjustCapability(Resource resource) {
>     int cores = resource.getVirtualCores();
>     int updatedCores = Math.min(resource.getVirtualCores(), maxCapability.getVirtualCores());
>     if (cores != updatedCores) {
>       resource.setVirtualCores(updatedCores);
>       LOG.info("Adjust virtual cores requirement from {} to {}.", cores, updatedCores);
>     }
>     int updatedMemory = Math.min(resource.getMemory(), maxCapability.getMemory());
>     if (resource.getMemory() != updatedMemory) {
>       resource.setMemory(updatedMemory);
>       LOG.info("Adjust memory requirement from {} to {} MB.", resource.getMemory(), updatedMemory);
>     }
>     return resource;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)