jvm options for individual jobs / topologies

[Benjamin Cuthbert <cu...@gmail.com>]

Hi all,

How do I give different topologies / jobs different Xmx memory parameters?

Regards

Re: jvm options for individual jobs / topologies

[Kostas Kloudas <k....@data-artisans.com>]

Hi Benjamin,

I do not think you can set per job memory configuration in a shared cluster.
The reason is that if different jobs share the same TM, there are resources that are shared between them, e.g, network buffers.

If you are ok with having a separate cluster per job then this will allow you to set per job memory quotas, because this would make
the total TM memory quota specification correspond to the per job ones.

In [1] you can find the available per job options and on [2] the cluster-wide ones.

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/execution_configuration.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/execution_configuration.html>
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/config.html <https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/config.html>

I hope this helps,
Kostas

> On May 8, 2018, at 11:46 AM, Benjamin Cuthbert <cu...@gmail.com> wrote:
> 
> Hi all,
> 
> How do I give different topologies / jobs different Xmx memory parameters?
> 
> Regards