You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Rex Fenley <Re...@remind101.com> on 2020/11/05 23:21:19 UTC
Rules of Thumb for Setting Parallelism
Hello,
I'm running a Job on AWS EMR with the TableAPI that does a long series of
Joins, GroupBys, and Aggregates and I'd like to know how to best tune
parallelism.
In my case, I have 8 EMR core nodes setup each with 4vCores and 8Gib of
memory. There's a job we have to run that has ~30 table operators. Given
this, how should I calculate what to set the systems parallelism to?
I also plan on running a second job on the same system, but just with 6
operators. Will this change the calculation for parallelism at all?
Thanks!
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: Rules of Thumb for Setting Parallelism
Posted by Rex Fenley <Re...@remind101.com>.
Awesome, thanks!
On Sat, Nov 7, 2020 at 6:43 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Rex,
>
> You should configure the number of slots per TaskManager to be the number
> of cores of a machine/node. In total you will then have a cluster with
> #slots = #cores per machine x #machines.
>
> If you have a cluster with 4 nodes and 8 slots each, then you have a total
> of 32 slots. Now if you have a job A which you start with a parallelism of
> 20, then you have 12 slots left. Hence, you could make use of these 12
> slots by starting a job B with a parallelism 12.
>
> Cheers,
> Till
>
> On Fri, Nov 6, 2020 at 7:20 PM Rex Fenley <Re...@remind101.com> wrote:
>
>> Great, thanks!
>>
>> So just to confirm, configure # of task slots to # of core nodes x # of
>> vCPUs?
>>
>> I'm not sure what you mean by "distribute them across both jobs (so that
>> the total adds up to 32)". Is it configurable how many task slots a job can
>> receive, so in this case I'd provide ~30/36 * 32 task slots for one job and
>> ~6/36 * 32 for another job, but even them out to sum to 32 task slots?
>>
>> Thanks
>>
>> On Fri, Nov 6, 2020 at 10:01 AM Till Rohrmann <tr...@apache.org>
>> wrote:
>>
>>> Hi Rex,
>>>
>>> as a rule of thumb I recommend configuring your TMs with as many slots
>>> as they have cores. So in your case your cluster would have 32 slots. Then
>>> depending on the workload of your jobs you should distribute them across
>>> both jobs (so that the total adds up to 32). A high number of operators
>>> does not necessarily mean that it needs more slots since operators can
>>> share the same slot. It mostly depends on the workload of your job. If the
>>> job should be too slow, then you would have to increase the cluster
>>> resources.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Fri, Nov 6, 2020 at 12:21 AM Rex Fenley <Re...@remind101.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm running a Job on AWS EMR with the TableAPI that does a long series
>>>> of Joins, GroupBys, and Aggregates and I'd like to know how to best tune
>>>> parallelism.
>>>>
>>>> In my case, I have 8 EMR core nodes setup each with 4vCores and 8Gib of
>>>> memory. There's a job we have to run that has ~30 table operators. Given
>>>> this, how should I calculate what to set the systems parallelism to?
>>>>
>>>> I also plan on running a second job on the same system, but just with 6
>>>> operators. Will this change the calculation for parallelism at all?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>>
>>>> Rex Fenley | Software Engineer - Mobile and Backend
>>>>
>>>>
>>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>>>> <https://www.facebook.com/remindhq>
>>>>
>>>
>>
>> --
>>
>> Rex Fenley | Software Engineer - Mobile and Backend
>>
>>
>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>> <https://www.facebook.com/remindhq>
>>
>
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: Rules of Thumb for Setting Parallelism
Posted by Till Rohrmann <tr...@apache.org>.
Hi Rex,
You should configure the number of slots per TaskManager to be the number
of cores of a machine/node. In total you will then have a cluster with
#slots = #cores per machine x #machines.
If you have a cluster with 4 nodes and 8 slots each, then you have a total
of 32 slots. Now if you have a job A which you start with a parallelism of
20, then you have 12 slots left. Hence, you could make use of these 12
slots by starting a job B with a parallelism 12.
Cheers,
Till
On Fri, Nov 6, 2020 at 7:20 PM Rex Fenley <Re...@remind101.com> wrote:
> Great, thanks!
>
> So just to confirm, configure # of task slots to # of core nodes x # of
> vCPUs?
>
> I'm not sure what you mean by "distribute them across both jobs (so that
> the total adds up to 32)". Is it configurable how many task slots a job can
> receive, so in this case I'd provide ~30/36 * 32 task slots for one job and
> ~6/36 * 32 for another job, but even them out to sum to 32 task slots?
>
> Thanks
>
> On Fri, Nov 6, 2020 at 10:01 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Rex,
>>
>> as a rule of thumb I recommend configuring your TMs with as many slots as
>> they have cores. So in your case your cluster would have 32 slots. Then
>> depending on the workload of your jobs you should distribute them across
>> both jobs (so that the total adds up to 32). A high number of operators
>> does not necessarily mean that it needs more slots since operators can
>> share the same slot. It mostly depends on the workload of your job. If the
>> job should be too slow, then you would have to increase the cluster
>> resources.
>>
>> Cheers,
>> Till
>>
>> On Fri, Nov 6, 2020 at 12:21 AM Rex Fenley <Re...@remind101.com> wrote:
>>
>>> Hello,
>>>
>>> I'm running a Job on AWS EMR with the TableAPI that does a long series
>>> of Joins, GroupBys, and Aggregates and I'd like to know how to best tune
>>> parallelism.
>>>
>>> In my case, I have 8 EMR core nodes setup each with 4vCores and 8Gib of
>>> memory. There's a job we have to run that has ~30 table operators. Given
>>> this, how should I calculate what to set the systems parallelism to?
>>>
>>> I also plan on running a second job on the same system, but just with 6
>>> operators. Will this change the calculation for parallelism at all?
>>>
>>> Thanks!
>>>
>>> --
>>>
>>> Rex Fenley | Software Engineer - Mobile and Backend
>>>
>>>
>>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>>> <https://www.facebook.com/remindhq>
>>>
>>
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> |
> FOLLOW US <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>
Re: Rules of Thumb for Setting Parallelism
Posted by Rex Fenley <Re...@remind101.com>.
Great, thanks!
So just to confirm, configure # of task slots to # of core nodes x # of
vCPUs?
I'm not sure what you mean by "distribute them across both jobs (so that
the total adds up to 32)". Is it configurable how many task slots a job can
receive, so in this case I'd provide ~30/36 * 32 task slots for one job and
~6/36 * 32 for another job, but even them out to sum to 32 task slots?
Thanks
On Fri, Nov 6, 2020 at 10:01 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Rex,
>
> as a rule of thumb I recommend configuring your TMs with as many slots as
> they have cores. So in your case your cluster would have 32 slots. Then
> depending on the workload of your jobs you should distribute them across
> both jobs (so that the total adds up to 32). A high number of operators
> does not necessarily mean that it needs more slots since operators can
> share the same slot. It mostly depends on the workload of your job. If the
> job should be too slow, then you would have to increase the cluster
> resources.
>
> Cheers,
> Till
>
> On Fri, Nov 6, 2020 at 12:21 AM Rex Fenley <Re...@remind101.com> wrote:
>
>> Hello,
>>
>> I'm running a Job on AWS EMR with the TableAPI that does a long series of
>> Joins, GroupBys, and Aggregates and I'd like to know how to best tune
>> parallelism.
>>
>> In my case, I have 8 EMR core nodes setup each with 4vCores and 8Gib of
>> memory. There's a job we have to run that has ~30 table operators. Given
>> this, how should I calculate what to set the systems parallelism to?
>>
>> I also plan on running a second job on the same system, but just with 6
>> operators. Will this change the calculation for parallelism at all?
>>
>> Thanks!
>>
>> --
>>
>> Rex Fenley | Software Engineer - Mobile and Backend
>>
>>
>> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
>> | FOLLOW US <https://twitter.com/remindhq> | LIKE US
>> <https://www.facebook.com/remindhq>
>>
>
--
Rex Fenley | Software Engineer - Mobile and Backend
Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/>
| FOLLOW
US <https://twitter.com/remindhq> | LIKE US
<https://www.facebook.com/remindhq>
Re: Rules of Thumb for Setting Parallelism
Posted by Till Rohrmann <tr...@apache.org>.
Hi Rex,
as a rule of thumb I recommend configuring your TMs with as many slots as
they have cores. So in your case your cluster would have 32 slots. Then
depending on the workload of your jobs you should distribute them across
both jobs (so that the total adds up to 32). A high number of operators
does not necessarily mean that it needs more slots since operators can
share the same slot. It mostly depends on the workload of your job. If the
job should be too slow, then you would have to increase the cluster
resources.
Cheers,
Till
On Fri, Nov 6, 2020 at 12:21 AM Rex Fenley <Re...@remind101.com> wrote:
> Hello,
>
> I'm running a Job on AWS EMR with the TableAPI that does a long series of
> Joins, GroupBys, and Aggregates and I'd like to know how to best tune
> parallelism.
>
> In my case, I have 8 EMR core nodes setup each with 4vCores and 8Gib of
> memory. There's a job we have to run that has ~30 table operators. Given
> this, how should I calculate what to set the systems parallelism to?
>
> I also plan on running a second job on the same system, but just with 6
> operators. Will this change the calculation for parallelism at all?
>
> Thanks!
>
> --
>
> Rex Fenley | Software Engineer - Mobile and Backend
>
>
> Remind.com <https://www.remind.com/> | BLOG <http://blog.remind.com/> |
> FOLLOW US <https://twitter.com/remindhq> | LIKE US
> <https://www.facebook.com/remindhq>
>