You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Himanshu Mishra (Jira)" <ji...@apache.org> on 2024/09/27 07:02:00 UTC

[jira] [Commented] (TEZ-4580) Slow preemption of new containers when re-use is enabled

    [ https://issues.apache.org/jira/browse/TEZ-4580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17885238#comment-17885238 ] 

Himanshu Mishra commented on TEZ-4580:
--------------------------------------

[~jeagles] [~abstractdog], can you please take a look.

> Slow preemption of new containers when re-use is enabled
> --------------------------------------------------------
>
>                 Key: TEZ-4580
>                 URL: https://issues.apache.org/jira/browse/TEZ-4580
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Himanshu Mishra
>            Assignee: Himanshu Mishra
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> I observed intermittent high runtime of a TPCDS query with running with YARN async scheduler `{_}yarn.scheduler.capacity.schedule-asynchronously.enable=true` along with container reuse.{_}
>  
> I found that preemption of lower priority containers was taking very long time in such cases. Tez AM log had warning {{{}Expected delayed containers to be empty.{}}}, followed by another {{Held container expected to be not null for a non-AM-released container}} and after this only 1 container was getting released, even when {{tez.am.preemption.percentage}} is high.
> Further investigation lead to following conclusion:
> 1. [Warn log / Assertion error|https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L1335] thrown because in [preemptIfNeeded()|https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L1314], when releasing new containers, the loop counter is being decremented with each `{{{}releaseUnassignedContainers{}}}`, leading to looping only half number of times. By using another counter, assertion passes because of condition method returns with check `{{{}if (numPendingRequestsToService < 1) {{}}}`.
> 2. In [releaseContainer()|https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L1566], the container is not getting removed from `{{{}delayedContainers{}}}` queue and only from `{{{}heldContainers{}}}` map, hence same container is being picked up for release in every iteration till next cycle of `{{{}DelayedContainerManager{}}}` finds out that the container is not in `{{{}heldContainers{}}}` and skips it [with log |https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L2095]`{{{}Skipping delayed container as container is no longer running, containerId=...{}}}`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)