You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "shuwenjun (JIRA)" <ji...@apache.org> on 2019/08/07 10:03:00 UTC
[jira] [Comment Edited] (FLINK-13590) flink-on-yarn sometimes could
create many little files that are xxx-taskmanager-conf.yaml
[ https://issues.apache.org/jira/browse/FLINK-13590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901802#comment-16901802 ]
shuwenjun edited comment on FLINK-13590 at 8/7/19 10:02 AM:
------------------------------------------------------------
Hi Yang，
You are right, it's right that the files are created when launching a new yarn container. And it is better optimized to override taskmanager-conf.yaml. Now, it could cause a more serious problem, that it will produce hundreds of thousands of small files, because the RM keep retrying to apply for taskmanager(container) when the source is enough. And the RM always don't get some new correct taskmanagers.
Thank you for your comment.
There are some logs.
{code:java}
//代码占位符
2019-08-07 16:54:52,729 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : hadoopxxx
2019-08-07 16:54:58,171 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 1.
2019-08-07 16:54:58,171 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 2.
2019-08-07 16:54:58,171 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 3.
2019-08-07 16:55:03,675 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_xxx_01_007684 - Remaining pending container requests: 3
2019-08-07 16:55:03,675 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 2.
2019-08-07 16:55:03,676 INFO org.apache.flink.yarn.YarnResourceManager - Adding keytab hdfs://xxxx.keytab to the AM container local resource bucket
2019-08-07 16:55:03,740 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers
2019-08-07 16:55:03,740 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers
2019-08-07 16:55:03,742 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : hadoopxxx
2019-08-07 16:55:03,748 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_xxx_01_007685 - Remaining pending container requests: 2
2019-08-07 16:55:03,748 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 1.
2019-08-07 16:55:03,748 INFO org.apache.flink.yarn.YarnResourceManager - Adding keytab hdfs://xxxx.keytab to the AM container local resource bucket
2019-08-07 16:55:03,761 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers
2019-08-07 16:55:03,761 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers
2019-08-07 16:55:03,761 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : hadoopxxx
2019-08-07 16:55:03,767 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_xxx_01_007686 - Remaining pending container requests: 1
2019-08-07 16:55:03,767 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 0.
2019-08-07 16:55:03,767 INFO org.apache.flink.yarn.YarnResourceManager - Adding keytab hdfs://xxxx.keytab to the AM container local resource bucket
2019-08-07 16:55:03,814 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers
2019-08-07 16:55:03,814 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers
2019-08-07 16:55:03,814 INFO org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - Opening proxy : hadoopxxx
2019-08-07 16:55:09,178 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 1.
2019-08-07 16:55:09,178 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 2.
2019-08-07 16:55:09,178 INFO org.apache.flink.yarn.YarnResourceManager - Requesting new TaskExecutor container with resources <memory:8192, vCores:2>. Number pending requests 3.
2019-08-07 16:55:14,682 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: container_xxx - Remaining pending container requests: 3
2019-08-07 16:55:14,683 INFO org.apache.flink.yarn.YarnResourceManager - Removing container request Capability[<memory:8192, vCores:2>]Priority[1]. Pending container requests 2.
2019-08-07 16:55:14,683 INFO org.apache.flink.yarn.YarnResourceManager - Adding keytab hdfs://xxxx.keytab to the AM container local resource bucket
2019-08-07 16:55:14,778 INFO org.apache.flink.yarn.YarnResourceManager - Creating container launch context for TaskManagers
2019-08-07 16:55:14,778 INFO org.apache.flink.yarn.YarnResourceManager - Starting TaskManagers
{code}
was (Author: shu_wenjun@qq.com):
Hi Yang，
You are right, it's right that the files are created when launching a new yarn container. And it is better optimized to override taskmanager-conf.yaml. Now, it could cause a more serious problem, that it will produce hundreds of thousands of small files, because the RM keep retrying to apply for taskmanager(container) when the source is enough. And the RM always don't get some new correct taskmanagers.
Thank you for your comment.
> flink-on-yarn sometimes could create many little files that are xxx-taskmanager-conf.yaml
> -----------------------------------------------------------------------------------------
>
> Key: FLINK-13590
> URL: https://issues.apache.org/jira/browse/FLINK-13590
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Reporter: shuwenjun
> Priority: Major
> Attachments: taskmanager-conf-yaml.png
>
>
> Both of 1.7.2 and 1.8.0 are used, but they could create many little files.
> These files are the configuration file of taskmanager and when the flink session try to apply a new container, one of the files will be created. And I don't know why sometimes the flink session apply container again and again? Or when one container has lost, it could delete its taskmanager-conf.yaml
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)