You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Yik San Chan <ev...@gmail.com> on 2021/05/07 10:56:18 UTC
How to increase the number of task managers?
Hi community,
According to the [docs](
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/
):
> taskmanager.numberOfTaskSlots: The number of slots that a TaskManager
offers *(default: 1)*. Each slot can take one task or pipeline. Having
multiple slots in a TaskManager can help amortize certain constant
overheads (of the JVM, application libraries, or network connections)
across parallel tasks or pipelines. See the Task Slots and Resources
<https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/concepts/flink-architecture/#task-slots-and-resources>
concepts section for details.
> Running more smaller TaskManagers with one slot each is a good starting
point and leads to the best isolation between tasks. Dedicating the same
resources to fewer larger TaskManagers with more slots can help to increase
resource utilization, at the cost of weaker isolation between the tasks
(more tasks share the same JVM).
We're able to tune slot count by setting taskmanager.numberOfTaskSlots,
that may help parallelize my task.
I wonder if I can tune the number of task managers? Is there a
corresponding config?
Best,
Yik San
Re: How to increase the number of task managers?
Posted by Yik San Chan <ev...@gmail.com>.
Hi Yangze,
Thanks for the answer! That helps.
Best,
Yik San
On Sat, May 8, 2021 at 10:15 AM Yangze Guo <ka...@gmail.com> wrote:
> Hi,
>
> > I wonder if I can tune the number of task managers? Is there a
> corresponding config?
>
> With K8S/Yarn resource provider, the task managers are allocated on
> demand. So, the number of them are depends on the max parallelism and
> the slot sharing group topology of your job.
> In standalone mode, you need to config the "conf/workers" in your
> flink distribution to decide the number of task managers[3].
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/
> [2]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/
> [3]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-and-stopping-a-cluster
>
> Best,
> Yangze Guo
>
> Best,
> Yangze Guo
>
>
> On Fri, May 7, 2021 at 7:34 PM Tamir Sagi <Ta...@niceactimize.com>
> wrote:
> >
> > Hey
> >
> > num of TMs = parallelism / num of slots
> >
> > parallelism.default is another config you should consider.
> >
> > Read also
> >
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/parallel/
> >
> >
> > ________________________________
> > From: Yik San Chan <ev...@gmail.com>
> > Sent: Friday, May 7, 2021 1:56 PM
> > To: user <us...@flink.apache.org>
> > Subject: How to increase the number of task managers?
> >
> >
> > EXTERNAL EMAIL
> >
> >
> >
> > Hi community,
> >
> > According to the [docs](
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/
> ):
> >
> > > taskmanager.numberOfTaskSlots: The number of slots that a TaskManager
> offers (default: 1). Each slot can take one task or pipeline. Having
> multiple slots in a TaskManager can help amortize certain constant
> overheads (of the JVM, application libraries, or network connections)
> across parallel tasks or pipelines. See the Task Slots and Resources
> concepts section for details.
> >
> > > Running more smaller TaskManagers with one slot each is a good
> starting point and leads to the best isolation between tasks. Dedicating
> the same resources to fewer larger TaskManagers with more slots can help to
> increase resource utilization, at the cost of weaker isolation between the
> tasks (more tasks share the same JVM).
> >
> > We're able to tune slot count by setting taskmanager.numberOfTaskSlots,
> that may help parallelize my task.
> >
> > I wonder if I can tune the number of task managers? Is there a
> corresponding config?
> >
> > Best,
> > Yik San
> >
> >
> > Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> > Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> > Viruses: Although we have taken steps toward ensuring that this e-mail
> and attachments are free from any virus, we advise that in keeping with
> good computing practice the recipient should ensure they are actually virus
> free.
>
Re: How to increase the number of task managers?
Posted by Yangze Guo <ka...@gmail.com>.
Hi,
> I wonder if I can tune the number of task managers? Is there a corresponding config?
With K8S/Yarn resource provider, the task managers are allocated on
demand. So, the number of them are depends on the max parallelism and
the slot sharing group topology of your job.
In standalone mode, you need to config the "conf/workers" in your
flink distribution to decide the number of task managers[3].
[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/
[2] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/
[3] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-and-stopping-a-cluster
Best,
Yangze Guo
Best,
Yangze Guo
On Fri, May 7, 2021 at 7:34 PM Tamir Sagi <Ta...@niceactimize.com> wrote:
>
> Hey
>
> num of TMs = parallelism / num of slots
>
> parallelism.default is another config you should consider.
>
> Read also
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/parallel/
>
>
> ________________________________
> From: Yik San Chan <ev...@gmail.com>
> Sent: Friday, May 7, 2021 1:56 PM
> To: user <us...@flink.apache.org>
> Subject: How to increase the number of task managers?
>
>
> EXTERNAL EMAIL
>
>
>
> Hi community,
>
> According to the [docs](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/):
>
> > taskmanager.numberOfTaskSlots: The number of slots that a TaskManager offers (default: 1). Each slot can take one task or pipeline. Having multiple slots in a TaskManager can help amortize certain constant overheads (of the JVM, application libraries, or network connections) across parallel tasks or pipelines. See the Task Slots and Resources concepts section for details.
>
> > Running more smaller TaskManagers with one slot each is a good starting point and leads to the best isolation between tasks. Dedicating the same resources to fewer larger TaskManagers with more slots can help to increase resource utilization, at the cost of weaker isolation between the tasks (more tasks share the same JVM).
>
> We're able to tune slot count by setting taskmanager.numberOfTaskSlots, that may help parallelize my task.
>
> I wonder if I can tune the number of task managers? Is there a corresponding config?
>
> Best,
> Yik San
>
>
> Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
Re: How to increase the number of task managers?
Posted by Tamir Sagi <Ta...@niceactimize.com>.
Hey
num of TMs = parallelism / num of slots
parallelism.default is another config you should consider.
Read also
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/execution/parallel/
[https://my-email-signature.link/signature.gif?u=1088647&e=151783011&v=daa41c995e0f03eb0cd358bef914ad1aaab37cbec8d5045736b62bf3f38f5497]
________________________________
From: Yik San Chan <ev...@gmail.com>
Sent: Friday, May 7, 2021 1:56 PM
To: user <us...@flink.apache.org>
Subject: How to increase the number of task managers?
EXTERNAL EMAIL
Hi community,
According to the [docs](https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/):
> taskmanager.numberOfTaskSlots: The number of slots that a TaskManager offers (default: 1). Each slot can take one task or pipeline. Having multiple slots in a TaskManager can help amortize certain constant overheads (of the JVM, application libraries, or network connections) across parallel tasks or pipelines. See the Task Slots and Resources<https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/concepts/flink-architecture/#task-slots-and-resources> concepts section for details.
> Running more smaller TaskManagers with one slot each is a good starting point and leads to the best isolation between tasks. Dedicating the same resources to fewer larger TaskManagers with more slots can help to increase resource utilization, at the cost of weaker isolation between the tasks (more tasks share the same JVM).
We're able to tune slot count by setting taskmanager.numberOfTaskSlots, that may help parallelize my task.
I wonder if I can tune the number of task managers? Is there a corresponding config?
Best,
Yik San
Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.