You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yangze Guo (Jira)" <ji...@apache.org> on 2020/05/25 06:30:00 UTC
[jira] [Updated] (FLINK-17917)
ResourceInformationReflector#getExternalResources should ignore the
external resource with a value of 0
[ https://issues.apache.org/jira/browse/FLINK-17917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yangze Guo updated FLINK-17917:
-------------------------------
Description:
*Background*: In FLINK-17390, we leverage {{WorkerSpecContainerResourceAdapter.InternalContainerResource}} to handle container matching logic. In FLINK-17407, we introduce external resources in {{WorkerSpecContainerResourceAdapter.InternalContainerResource}}.
On containers returned by Yarn, we try to get the corresponding worker specs by:
- Convert the container to {{InternalContainerResource}}
- Get the WorkerResourceSpec from {{containerResourceToWorkerSpecs}} map.
*Problem*: Container mismatch could happen in the below scenario:
- Flink does not allocate any external resources, the {{externalResources}} of {{InternalContainerResource}} is an empty map.
- The returned container contains all the resources (with a value of 0) defined in Yarn's {{resource-types.xml}}. The {{externalResources}} of {{InternalContainerResource}} has one or more entries with a value of 0.
- These two {{InternalContainerResource}} do not match.
To solve this problem, we could ignore all the external resources with a value of 0 in "ResourceInformationReflector#getExternalResources".
cc [~trohrmann] Could you assign this to me.
was:
*Background*: In FLINK-17390, we leverage {{WorkerSpecContainerResourceAdapter.InternalContainerResource}} to handle container matching logic. In FLINK-17407, we introduce external resources in {{WorkerSpecContainerResourceAdapter.InternalContainerResource}}.
On containers returned by Yarn, we try to get the corresponding worker specs by:
- Convert the container to {{InternalContainerResource}}
- Get the WorkerResourceSpec from {{containerResourceToWorkerSpecs}} map.
Container mismatch could happen in the below scenario:
- Flink does not allocate any external resources, the {{externalResources}} of {{InternalContainerResource}} is an empty map.
- The returned container contains all the resources (with a value of 0) defined in Yarn's {{resource-types.xml}}. The {{externalResources}} of {{InternalContainerResource}} has one or more entries with a value of 0.
- These two {{InternalContainerResource}} do not match.
To solve this problem, we could ignore all the external resources with a value of 0 in "ResourceInformationReflector#getExternalResources".
cc [~trohrmann] Could you assign this to me.
> ResourceInformationReflector#getExternalResources should ignore the external resource with a value of 0
> -------------------------------------------------------------------------------------------------------
>
> Key: FLINK-17917
> URL: https://issues.apache.org/jira/browse/FLINK-17917
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.11.0
> Reporter: Yangze Guo
> Priority: Blocker
> Fix For: 1.11.0
>
>
> *Background*: In FLINK-17390, we leverage {{WorkerSpecContainerResourceAdapter.InternalContainerResource}} to handle container matching logic. In FLINK-17407, we introduce external resources in {{WorkerSpecContainerResourceAdapter.InternalContainerResource}}.
> On containers returned by Yarn, we try to get the corresponding worker specs by:
> - Convert the container to {{InternalContainerResource}}
> - Get the WorkerResourceSpec from {{containerResourceToWorkerSpecs}} map.
> *Problem*: Container mismatch could happen in the below scenario:
> - Flink does not allocate any external resources, the {{externalResources}} of {{InternalContainerResource}} is an empty map.
> - The returned container contains all the resources (with a value of 0) defined in Yarn's {{resource-types.xml}}. The {{externalResources}} of {{InternalContainerResource}} has one or more entries with a value of 0.
> - These two {{InternalContainerResource}} do not match.
> To solve this problem, we could ignore all the external resources with a value of 0 in "ResourceInformationReflector#getExternalResources".
> cc [~trohrmann] Could you assign this to me.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)