You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Xintong Song (Jira)" <ji...@apache.org> on 2023/04/13 08:32:00 UTC
[jira] [Closed] (FLINK-27750) The configuration of JobManagerOption.TOTAL_PROCESS_MEMORY(jobmanager.memory.process.size) not work
[ https://issues.apache.org/jira/browse/FLINK-27750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xintong Song closed FLINK-27750.
--------------------------------
Resolution: Not A Bug
Closing the ticket due to:
1. 1.14.4 is no longer supported.
2. User should not directly access the Flink's internal classes.
For the record, the root cause here is that the JM in `flinkConfiguration` will be overwritten by `ClusterSpecification`. The user created a `ClusterSpecification` without specifying proper memory sizes.
> The configuration of JobManagerOption.TOTAL_PROCESS_MEMORY(jobmanager.memory.process.size) not work
> ---------------------------------------------------------------------------------------------------
>
> Key: FLINK-27750
> URL: https://issues.apache.org/jira/browse/FLINK-27750
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
> Affects Versions: 1.14.4
> Reporter: dong
> Priority: Minor
> Labels: TOTAL_PROCESS_MEMORY, auto-deprioritized-major, jobmanager.memory.process.size
> Attachments: image-2022-05-24-14-00-39-255.png, image-2022-05-24-14-18-30-063.png
>
>
> By constructing kubernetesClusterDescriptor and Fabric8FlinkKubeClient to deploy kubernetes application mode of job，The code is shown below.
> {code:java}
> //Initialize flinkConfiguration and set options including TOTAL_PROCESS_MEMORY
> Configuration flinkConfiguration = GlobalConfiguration.loadConfiguration();
> flinkConfiguration.set(DeploymentOptions.TARGET, KubernetesDeploymentTarget.APPLICATION.getName())
> .set(PipelineOptions.JARS, Collections.singletonList(flinkDistJar))
> .set(KubernetesConfigOptions.CLUSTER_ID, "APPLICATION1")
> .set(KubernetesConfigOptions.CONTAINER_IMAGE, "img_url")
> .set(KubernetesConfigOptions.CONTAINER_IMAGE_PULL_POLICY, KubernetesConfigOptions.ImagePullPolicy.Always)
> .set(JobManagerOptions.TOTAL_PROCESS_MEMORY, MemorySize.parse("1024M"))
> .set...;
> //Construct kubernetesClusterDescriptor and Fabric8FlinkKubeClient
> KubernetesClusterDescriptor kubernetesClusterDescriptor = new KubernetesClusterDescriptor(
> flinkConfiguration,
> new Fabric8FlinkKubeClient(
> flinkConfiguration,
> new DefaultKubernetesClient(),
> Executors.newFixedThreadPool(2)
> )
> );
> ApplicationConfiguration applicationConfiguration = new ApplicationConfiguration(execArgs, null);
> //deploy kubernetes application mode of job
> ClusterClient<String> clusterClient = kubernetesClusterDescriptor.deployApplicationCluster(
> new ClusterSpecification.ClusterSpecificationBuilder().createClusterSpecification(),
> applicationConfiguration
> ).getClusterClient();
> String clusterId = clusterClient.getClusterId(); {code}
> As above，I set TOTAL_PROCESS_MEMORY to 1024M，The flink UI displays the following memory configuration，which is clearly correct(448+128+256+192=1024).
> !image-2022-05-24-14-00-39-255.png|width=759,height=255!
> But when I turn to JobManager using {_}Kubectl Describe Deployment{_}, I found that the POD memory of JobManager was always 768M, which should have been equal to TOTAL_PROCESS_MEMORY 1024M. And no matter how I adjust TOTAL_PROCESS_MEMORY parameter it doesn't work.
> !image-2022-05-24-14-18-30-063.png!
> The result is a POD OOMkilled when JobManager memory usage exceeds 768M.
> I expect the JobManager pod to be equal to TOTAL_PROCESS_MEMORY, so I can adjust the memory to suit my needs.
> Is there something WRONG with my configuration, or should JobManager's pod take up the same amount of memory as TOTAL_PROCESS_MEMORY?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)