[jira] [Commented] (YARN-6406) Garbage Collect unused SchedulerRequestKeys

["Arun Suresh (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951845#comment-15951845 ] 

Arun Suresh commented on YARN-6406:
-----------------------------------

Thanks for the review [~leftnoteasy],

bq. Why changes of AppInfo required?
Hmmm.. so the the {{TestRMWebServicesApps}} was complaining since it needed a resource request object. With this patch, if there is no outstanding resource request, The AppInfo will not contain any resource request objects. So I decided to send a dummy resourceRequest Object when none exist. I think the right approach is to fix the test case (which might be harder).. thoughts ?

bq.  In LocalitySchedulingPlacementSet: it calls appSchedulingInfo directly in decrementOutstanding ...
Don't think it is a problem too much (based on existing code paths) .. but yes, maybe we should clean it up, since it could lead some circular references to the same placementset object.
Please feel free to open another JIRA for that (I can help review), but for the timebeing, I think we can remove the schedulerkey as is done in this patch ?

> Garbage Collect unused SchedulerRequestKeys
> -------------------------------------------
>
>                 Key: YARN-6406
>                 URL: https://issues.apache.org/jira/browse/YARN-6406
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>         Attachments: YARN-6406.001.patch
>
>
> YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys from the AppScheduleingInfo. It looks like after YARN-6040, ScedulerRequestKeys are removed only if the Application sends a 0 numContainers requests. While earlier, the outstanding schedulerKeys were also remove as soon as a container is allocated as well.
> An additional optimization we were hoping to include is to remove the ResourceRequests itself once the numContainers == 0, since we see in our clusters that the RM heap space consumption increases drastically due to a large number of ResourceRequests with 0 num containers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org