You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Xintong Song (Jira)" <ji...@apache.org> on 2020/02/28 02:47:00 UTC
[jira] [Updated] (FLINK-16299) Release containers recovered from
previous attempt in which TaskExecutor is not started.
[ https://issues.apache.org/jira/browse/FLINK-16299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xintong Song updated FLINK-16299:
---------------------------------
Description:
As discussed in FLINK-16215, on Yarn deployment, {{YarnResourceManager}} starts a new {{TaskExecutor}} in two steps:
# Request a new container from Yarn
# Starts a {{TaskExecutor}} process in the allocated container
If JM failover happens between the two steps, in the new attempt {{YarnResourceManager}} will not start {{TaskExecutor}} processes in recovered containers. That means such containers are neither used nor released.
A potential fix to this problem is to query for the container status by calling {{NMClientAsync#getContainerStatusAsync}}, and release the containers whose state is {{NEW}}, keeps only those whose state is {{RUNNING}} and waiting for them to register.
was:
As discussed in FLINK-16215, on Yarn deployment, {{YarnResourceManager}} starts a new {{TaskExecutor}} in two steps:
# Request a new container from Yarn
# Starts a {{TaskExecutor}} process in the allocated container
If JM failover happens between the two steps, in the new attempt {{YarnResourceManager}} will not start {{TaskExecutor}} processes in recovered containers. That means such containers are neither used nor released.
A potential fix to this problem, is to query form the container status by calling {{NMClientAsync#getContainerStatusAsync}}, and release the containers whose state is {{NEW}}, keeps only those whose state is {{RUNNING}} and waiting for them to register.
> Release containers recovered from previous attempt in which TaskExecutor is not started.
> ----------------------------------------------------------------------------------------
>
> Key: FLINK-16299
> URL: https://issues.apache.org/jira/browse/FLINK-16299
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Reporter: Xintong Song
> Assignee: Yangze Guo
> Priority: Major
>
> As discussed in FLINK-16215, on Yarn deployment, {{YarnResourceManager}} starts a new {{TaskExecutor}} in two steps:
> # Request a new container from Yarn
> # Starts a {{TaskExecutor}} process in the allocated container
> If JM failover happens between the two steps, in the new attempt {{YarnResourceManager}} will not start {{TaskExecutor}} processes in recovered containers. That means such containers are neither used nor released.
> A potential fix to this problem is to query for the container status by calling {{NMClientAsync#getContainerStatusAsync}}, and release the containers whose state is {{NEW}}, keeps only those whose state is {{RUNNING}} and waiting for them to register.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)