You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "CruiseCheung (Jira)" <ji...@apache.org> on 2020/12/22 07:55:00 UTC
[jira] [Updated] (TEZ-4260) DagAwareYarnTaskScheduler may assign
unfittable container to task
[ https://issues.apache.org/jira/browse/TEZ-4260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
CruiseCheung updated TEZ-4260:
------------------------------
Description:
When tasks in two vertice require differrent memory size containers and tez.am.container.reuse.enabled is true, DagAwareYarnTaskSchedule may assign unfittable containers to tasks, cause out of memory error.
Drill down it, we found [https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java#L1110] should call hc.canFit() method
{code:java}
if (hc.canFit(request.getCapability()) && csig == null ||
signatureMatcher.isSuperSet(csig, request.getContainerSignature()))
{code}
was:
When tasks in two vertice require differrent memory size containers and tez.am.container.reuse.enabled is true, DagAwareYarnTaskSchedule may assign unfittable containers to tasks, cause out of memory error.
Drill down it, we found [https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java#L1110] should call hc.canFit() method
{code:java}
if (hc.canFit(request.getCapability()) && csig == null ||
signatureMatcher.isSuperSet(csig, request.getContainerSignature()))
{code}
> DagAwareYarnTaskScheduler may assign unfittable container to task
> -----------------------------------------------------------------
>
> Key: TEZ-4260
> URL: https://issues.apache.org/jira/browse/TEZ-4260
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.2
> Reporter: CruiseCheung
> Priority: Minor
>
> When tasks in two vertice require differrent memory size containers and tez.am.container.reuse.enabled is true, DagAwareYarnTaskSchedule may assign unfittable containers to tasks, cause out of memory error.
> Drill down it, we found [https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/DagAwareYarnTaskScheduler.java#L1110] should call hc.canFit() method
> {code:java}
> if (hc.canFit(request.getCapability()) && csig == null ||
> signatureMatcher.isSuperSet(csig, request.getContainerSignature()))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)