You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Nancy Estrada <na...@gmail.com> on 2017/01/23 10:49:23 UTC

Flink configuration

Hi all,

I have been reading about how to configure Flink when we have a set up
consisting on a couple of VMs with more than 1 vCore.  I am a bit confused
about how to set the degree of parallelism in the
taskmanager.numberOfTaskSlots parameter:

* According to the Flink documentation[1], this value is typically
proportional to the number of/ physical CPU cores/ that the TaskManager’s
machine has.

* However the YARN documentation[2], makes reference to the number of
/Virtual CPU cores/ per TaskManager.

My question is, If my Flink Jobs will be running on VMs (without using
YARN),  the "taskmanager.numberOfTaskSlots" will depend on the number of
vCPU that mi VM has? or must be related to the physical cores?  

Thanks in advance for your help! 
Nancy

[1]https://ci.apache.org/projects/flink/flink-docs-release-0.8/config.html
[2]https://ci.apache.org/projects/flink/flink-docs-release-0.8/yarn_setup.html



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-configuration-tp11210.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Flink configuration

Posted by Greg Hogan <co...@greghogan.com>.

Has anyone reported decreased performance with hyper-threading?

On Tue, Jan 24, 2017 at 11:18 AM, Aljoscha Krettek <al...@apache.org>
wrote:

> Hi,
> that wording is from a time where no-one though about VMs with virtual
> cores. IMHO this maps directly to virtual cores so you should set it
> according to the number of virtual cores of your VMs.
>
> Cheers,
> Aljoscha
>
> On Mon, 23 Jan 2017 at 11:51 Nancy Estrada <na...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I have been reading about how to configure Flink when we have a set up
>> consisting on a couple of VMs with more than 1 vCore.  I am a bit confused
>> about how to set the degree of parallelism in the
>> taskmanager.numberOfTaskSlots parameter:
>>
>> * According to the Flink documentation[1], this value is typically
>> proportional to the number of/ physical CPU cores/ that the TaskManager’s
>> machine has.
>>
>> * However the YARN documentation[2], makes reference to the number of
>> /Virtual CPU cores/ per TaskManager.
>>
>> My question is, If my Flink Jobs will be running on VMs (without using
>> YARN),  the "taskmanager.numberOfTaskSlots" will depend on the number of
>> vCPU that mi VM has? or must be related to the physical cores?
>>
>> Thanks in advance for your help!
>> Nancy
>>
>> [1]https://ci.apache.org/projects/flink/flink-docs-
>> release-0.8/config.html
>> [2]https://ci.apache.org/projects/flink/flink-docs-
>> release-0.8/yarn_setup.html
>>
>>
>>
>> --
>> View this message in context: http://apache-flink-user-
>> mailing-list-archive.2336050.n4.nabble.com/Flink-
>> configuration-tp11210.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>

Re: Flink configuration

Posted by Aljoscha Krettek <al...@apache.org>.

Hi,
that wording is from a time where no-one though about VMs with virtual
cores. IMHO this maps directly to virtual cores so you should set it
according to the number of virtual cores of your VMs.

Cheers,
Aljoscha

On Mon, 23 Jan 2017 at 11:51 Nancy Estrada <na...@gmail.com>
wrote:

> Hi all,
>
> I have been reading about how to configure Flink when we have a set up
> consisting on a couple of VMs with more than 1 vCore.  I am a bit confused
> about how to set the degree of parallelism in the
> taskmanager.numberOfTaskSlots parameter:
>
> * According to the Flink documentation[1], this value is typically
> proportional to the number of/ physical CPU cores/ that the TaskManager’s
> machine has.
>
> * However the YARN documentation[2], makes reference to the number of
> /Virtual CPU cores/ per TaskManager.
>
> My question is, If my Flink Jobs will be running on VMs (without using
> YARN),  the "taskmanager.numberOfTaskSlots" will depend on the number of
> vCPU that mi VM has? or must be related to the physical cores?
>
> Thanks in advance for your help!
> Nancy
>
> [1]https://ci.apache.org/projects/flink/flink-docs-release-0.8/config.html
> [2]
> https://ci.apache.org/projects/flink/flink-docs-release-0.8/yarn_setup.html
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-configuration-tp11210.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>