You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Li Zhao <il...@gmail.com> on 2019/12/25 08:18:29 UTC
question: jvm heap size per task?
Hi,
Greetings, hope this is the proper place to ask questions, apologize if not.
We have a shared flink cluster running with docker, want to set different
heap size per task(some tasks require larger heap size, while most tasks
only need a little), is it feasible?
I've gone through [1], [2] and [3], my current understanding is all task
managers have the same heap size which is set by `taskmanager.heap.size`
and is fixed ever since the task manager is started, and all tasks running
in that task manager will share that heap size.
Am I understanding it correctly? And any approaches to our needs?
Thanks in advance!
[1]:
https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html
[2]:
https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html
[3]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
Thanks,
Li
Re: question: jvm heap size per task?
Posted by Li Zhao <il...@gmail.com>.
Understood, thank you for the quick response!
Thanks,
Li
Xintong Song <to...@gmail.com> 于2019年12月25日周三 下午5:05写道:
> Hi Li,
>
> It is true that currently all the task managers have the same heap size,
> and it's fixed ever since started. Unfortunately, your needs cannot be
> satisfied at the moment.
>
> Heap size of task managers cannot be changed once started, because flink
> task managers run in JVMs and JVM does not support resizing after started.
>
> However, there is an ongoing approach towards your needs. The community is
> working on fine-grained resource management, which in general allows
> specify per task/operator resource requirements and allocate task manager
> resources accordingly. You can refers to FLIP-53 [1] and FLIP-56 [2] for
> more details. Another related effort is pluggable slot manager [3], which
> allows having pluggable resource scheduling strategies such as launch task
> managers with customized resources according to the tasks' requirements.
>
> Thank you~
>
> Xintong Song
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
> [3] https://issues.apache.org/jira/browse/FLINK-14106
>
> On Wed, Dec 25, 2019 at 4:18 PM Li Zhao <il...@gmail.com> wrote:
>
>> Hi,
>>
>> Greetings, hope this is the proper place to ask questions, apologize if
>> not.
>>
>> We have a shared flink cluster running with docker, want to set different
>> heap size per task(some tasks require larger heap size, while most tasks
>> only need a little), is it feasible?
>>
>> I've gone through [1], [2] and [3], my current understanding is all task
>> managers have the same heap size which is set by `taskmanager.heap.size`
>> and is fixed ever since the task manager is started, and all tasks running
>> in that task manager will share that heap size.
>>
>> Am I understanding it correctly? And any approaches to our needs?
>>
>> Thanks in advance!
>>
>> [1]:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html
>> [2]:
>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html
>> [3]:
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>>
>> Thanks,
>> Li
>>
>
Re: question: jvm heap size per task?
Posted by Xintong Song <to...@gmail.com>.
Hi Li,
It is true that currently all the task managers have the same heap size,
and it's fixed ever since started. Unfortunately, your needs cannot be
satisfied at the moment.
Heap size of task managers cannot be changed once started, because flink
task managers run in JVMs and JVM does not support resizing after started.
However, there is an ongoing approach towards your needs. The community is
working on fine-grained resource management, which in general allows
specify per task/operator resource requirements and allocate task manager
resources accordingly. You can refers to FLIP-53 [1] and FLIP-56 [2] for
more details. Another related effort is pluggable slot manager [3], which
allows having pluggable resource scheduling strategies such as launch task
managers with customized resources according to the tasks' requirements.
Thank you~
Xintong Song
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Operator+Resource+Management
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-56%3A+Dynamic+Slot+Allocation
[3] https://issues.apache.org/jira/browse/FLINK-14106
On Wed, Dec 25, 2019 at 4:18 PM Li Zhao <il...@gmail.com> wrote:
> Hi,
>
> Greetings, hope this is the proper place to ask questions, apologize if
> not.
>
> We have a shared flink cluster running with docker, want to set different
> heap size per task(some tasks require larger heap size, while most tasks
> only need a little), is it feasible?
>
> I've gone through [1], [2] and [3], my current understanding is all task
> managers have the same heap size which is set by `taskmanager.heap.size`
> and is fixed ever since the task manager is started, and all tasks running
> in that task manager will share that heap size.
>
> Am I understanding it correctly? And any approaches to our needs?
>
> Thanks in advance!
>
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/cli.html
> [2]:
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/concepts/runtime.html
> [3]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-49%3A+Unified+Memory+Configuration+for+TaskExecutors
>
> Thanks,
> Li
>