You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Heejong Lee (Jira)" <ji...@apache.org> on 2019/12/06 19:48:00 UTC

[jira] [Comment Edited] (BEAM-8901) add experimental flag for reusing flink local environment

    [ https://issues.apache.org/jira/browse/BEAM-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990113#comment-16990113 ] 

Heejong Lee edited comment on BEAM-8901 at 12/6/19 7:47 PM:
------------------------------------------------------------

Interesting. Does Flink also create a new local environment for every test? I could see Flink local environment did some cleanup job when I reuse the existing environment since there are logs that the mini cluster automatically shuts down and reboots after each pipeline run. I'm still not sure what happens when multiple local environments are created in the single JVM (do they share some backend?) I'll try to test with -XX:MaxMetaspaceSize.


was (Author: heejong):
Interesting. Does Flink also create a new local environment for every test? I could see Flink local environment did some cleanup job when I reuse the existing environment since there are logs that the mini cluster automatically shuts down and reboots after each pipeline run. I'm still not sure what happens when multiple local environments are created (do they share some backend?) I'll try to test with -XX:MaxMetaspaceSize.

> add experimental flag for reusing flink local environment
> ---------------------------------------------------------
>
>                 Key: BEAM-8901
>                 URL: https://issues.apache.org/jira/browse/BEAM-8901
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-flink
>            Reporter: Heejong Lee
>            Assignee: Heejong Lee
>            Priority: Major
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> Flink job server launches a new mini cluster every time we run the pipeline on Flink local environment. To prevent OOM, we need to reuse existing Flink local environment if possible.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)