You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Dylan Adams <dy...@gmail.com> on 2019/05/07 13:28:40 UTC

Flink on YARN: TaskManager heap auto-sizing?

In the Configuration section of the docs
<https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager>,
the description for "taskmanager.heap.size" contains: "On YARN setups, this
value is automatically configured to the size of the TaskManager's YARN
container, minus a certain tolerance value."

Does that functionality exist?

I don't see any documented method to specify the YARN container size for
the TaskManagers, nor could I find any logic in the Flink YARN integration
code that seemed to implement that behavior.

My understanding is that you need to manually calculate and specify
taskmanager.heap.size
(and jobmanager.heap.size) based on your YARN setup.

Thanks,
Dylan

Re: Flink on YARN: TaskManager heap auto-sizing?

Posted by Dylan Adams <dy...@gmail.com>.
Till,

Thanks for the pointer to the code.

Regards,
Dylan

On Wed, May 8, 2019 at 11:18 AM Till Rohrmann <tr...@apache.org> wrote:

> Hi Dylan,
>
> the container's memory will be calculated here [1]. In the case of Yarn,
> the user specifies the container memory size and based on this Flink
> calculates with how much heap memory the JVM is started (container memory
> size - off heap memory - cut off memory).
>
> [1]
> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/ContaineredTaskManagerParameters.java#L160
>
> Cheers,
> Till
>
> On Tue, May 7, 2019 at 3:29 PM Dylan Adams <dy...@gmail.com> wrote:
>
>> In the Configuration section of the docs
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager>,
>> the description for "taskmanager.heap.size" contains: "On YARN setups,
>> this value is automatically configured to the size of the TaskManager's
>> YARN container, minus a certain tolerance value."
>>
>> Does that functionality exist?
>>
>> I don't see any documented method to specify the YARN container size for
>> the TaskManagers, nor could I find any logic in the Flink YARN integration
>> code that seemed to implement that behavior.
>>
>> My understanding is that you need to manually calculate and specify taskmanager.heap.size
>> (and jobmanager.heap.size) based on your YARN setup.
>>
>> Thanks,
>> Dylan
>>
>

Re: Flink on YARN: TaskManager heap auto-sizing?

Posted by Till Rohrmann <tr...@apache.org>.
Hi Dylan,

the container's memory will be calculated here [1]. In the case of Yarn,
the user specifies the container memory size and based on this Flink
calculates with how much heap memory the JVM is started (container memory
size - off heap memory - cut off memory).

[1]
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/ContaineredTaskManagerParameters.java#L160

Cheers,
Till

On Tue, May 7, 2019 at 3:29 PM Dylan Adams <dy...@gmail.com> wrote:

> In the Configuration section of the docs
> <https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/config.html#taskmanager>,
> the description for "taskmanager.heap.size" contains: "On YARN setups,
> this value is automatically configured to the size of the TaskManager's
> YARN container, minus a certain tolerance value."
>
> Does that functionality exist?
>
> I don't see any documented method to specify the YARN container size for
> the TaskManagers, nor could I find any logic in the Flink YARN integration
> code that seemed to implement that behavior.
>
> My understanding is that you need to manually calculate and specify taskmanager.heap.size
> (and jobmanager.heap.size) based on your YARN setup.
>
> Thanks,
> Dylan
>