You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Ahmed Nader <ah...@gmail.com> on 2016/03/18 13:13:32 UTC

degree of Parallelism

Hi,
How can i setParallelism in a generic way that provide better performance
on any device not only mine? In this case is it better to be set to a
certain value or if i just didn't set it to any value does flink take care
of that generically and provide better execution performance?
Thanks.

Re: degree of Parallelism

Posted by Ahmed Nader <ah...@gmail.com>.

Thanks so much Till and Fabian. So if i were to set it, how can i know the
best degree to be used for my application, is it the number of cores or
what should i set it to?
Thanks

On 18 March 2016 at 13:36, Fabian Hueske <fh...@gmail.com> wrote:

> Hi,
>
> did find the documentation for configuring the parallelism [1]?
> It explains how to set the parallelism on different levels: Cluster, Job,
> Task.
>
> Best, Fabian
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/common/index.html#parallel-execution
>
> 2016-03-18 13:34 GMT+01:00 Till Rohrmann <tr...@apache.org>:
>
>> Hi Ahmed,
>>
>> if you don't set the parallelism in your program then depending on how
>> you execute your program different parallelisms will be used. If you
>> execute it in your IDE, then the number of cores will be used as
>> parallelism. If you submit it to a cluster without specifying the
>> parallelism via the cli, then the default parallelism which you've
>> specified in flink-conf.yaml will be used. If you use the -p option of the
>> cli, then this value will be used as the parallelism.
>>
>> Cheers,
>> Till
>>
>> On Fri, Mar 18, 2016 at 1:13 PM, Ahmed Nader <ah...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> How can i setParallelism in a generic way that provide better
>>> performance on any device not only mine? In this case is it better to be
>>> set to a certain value or if i just didn't set it to any value does flink
>>> take care of that generically and provide better execution performance?
>>> Thanks.
>>>
>>
>>
>

Re: degree of Parallelism

Posted by Fabian Hueske <fh...@gmail.com>.

Hi,

did find the documentation for configuring the parallelism [1]?
It explains how to set the parallelism on different levels: Cluster, Job,
Task.

Best, Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/common/index.html#parallel-execution

2016-03-18 13:34 GMT+01:00 Till Rohrmann <tr...@apache.org>:

> Hi Ahmed,
>
> if you don't set the parallelism in your program then depending on how you
> execute your program different parallelisms will be used. If you execute it
> in your IDE, then the number of cores will be used as parallelism. If you
> submit it to a cluster without specifying the parallelism via the cli, then
> the default parallelism which you've specified in flink-conf.yaml will be
> used. If you use the -p option of the cli, then this value will be used as
> the parallelism.
>
> Cheers,
> Till
>
> On Fri, Mar 18, 2016 at 1:13 PM, Ahmed Nader <ah...@gmail.com>
> wrote:
>
>> Hi,
>> How can i setParallelism in a generic way that provide better performance
>> on any device not only mine? In this case is it better to be set to a
>> certain value or if i just didn't set it to any value does flink take care
>> of that generically and provide better execution performance?
>> Thanks.
>>
>
>

Re: degree of Parallelism

Posted by Till Rohrmann <tr...@apache.org>.

Hi Ahmed,

if you don't set the parallelism in your program then depending on how you
execute your program different parallelisms will be used. If you execute it
in your IDE, then the number of cores will be used as parallelism. If you
submit it to a cluster without specifying the parallelism via the cli, then
the default parallelism which you've specified in flink-conf.yaml will be
used. If you use the -p option of the cli, then this value will be used as
the parallelism.

Cheers,
Till

On Fri, Mar 18, 2016 at 1:13 PM, Ahmed Nader <ah...@gmail.com>
wrote:

> Hi,
> How can i setParallelism in a generic way that provide better performance
> on any device not only mine? In this case is it better to be set to a
> certain value or if i just didn't set it to any value does flink take care
> of that generically and provide better execution performance?
> Thanks.
>