You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Biao Geng (Jira)" <ji...@apache.org> on 2022/04/19 09:12:00 UTC

[jira] [Commented] (FLINK-27303) Flink Operator will create a large amount of temp log config files

    [ https://issues.apache.org/jira/browse/FLINK-27303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524189#comment-17524189 ] 

Biao Geng commented on FLINK-27303:
-----------------------------------

Hi [~gyfora], big +1 for this jira. And I believe besides log config files, there will also be a lot of pod template file if we define it the yaml. Because the {{applyCommonPodTemplate}} will create temporary files for pod template as well.
I think your suggestion makes sense. And one possible solution is to divide the generated configs into 2 types: type 1 is configs needed by observer and reconciler. type 2 is configs that not used by observer or reconciler.

Besides, I am wondering if it is a good idea to maintain a concurrent hashmap to save the effective config of each deployment so that we can reduce some repeated creation of effective configs. It may be the bottleneck of memory usage when there are plenty of deployments in the operator and make the operator less `stateless`. Not sure if it is worthwhile.

> Flink Operator will create a large amount of temp log config files
> ------------------------------------------------------------------
>
>                 Key: FLINK-27303
>                 URL: https://issues.apache.org/jira/browse/FLINK-27303
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kubernetes Operator
>    Affects Versions: kubernetes-operator-1.0.0
>            Reporter: Gyula Fora
>            Priority: Critical
>             Fix For: kubernetes-operator-1.0.0
>
>
> Now we use the configbuilder in multiple different places to generate the effective config including observer, reconciler, validator etc.
> The effective config gerenration logic also creates temporary log config files (if spec logConfiguration is set) which would lead to 3-4 files generated in every reconcile loop for a given job. These files are not cleaned up until the operator restarts leading to a large amount of files.
> I believe we should change the config generation logic and only apply the logconfig generation logic right before flink cluster submission as that is the only thing affected by it.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)