You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Ken Krugler <kk...@transpac.com> on 2016/04/29 01:58:25 UTC
EMR vCores and slot allocation
Based on what Flink reports in the JobManager GUI, it looks like it thinks that the EC2 instances I’m using for my EMR jobs only have 4 physical cores.
Which would make sense, as Amazon describes these servers as having 8 vCores.
From https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html <https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html>, the recommended configuration would then be 4 slots/TaskManager, yes?
Thanks,
— Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: EMR vCores and slot allocation
Posted by Fabian Hueske <fh...@gmail.com>.
The slot configuration should depend on the complexity of jobs.
Since each slot runs a "slice" of a program, one slot might potentially
execute many concurrent tasks.
For complex jobs you should allocate more than one core for each slot.
2016-05-02 10:12 GMT+02:00 Robert Metzger <rm...@apache.org>:
> Hi Ken,
> sorry for the late response. The number of CPU cores we show in the web
> interface is based on what the JVM tells us from
> "Runtime.getRuntime().availableProcessors();". I'm not sure how tthe
> processor count behaves on Amazon VMs.
>
> Given that each of your servers has 8 vCores, I would set the slot count
> to 8 per Taskmanager.
>
>
> On Fri, Apr 29, 2016 at 1:58 AM, Ken Krugler <kk...@transpac.com>
> wrote:
>
>> Based on what Flink reports in the JobManager GUI, it looks like it
>> thinks that the EC2 instances I’m using for my EMR jobs only have 4
>> physical cores.
>>
>> Which would make sense, as Amazon describes these servers as having 8
>> vCores.
>>
>> From
>> https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html,
>> the recommended configuration would then be 4 slots/TaskManager, yes?
>>
>> Thanks,
>>
>> — Ken
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Cassandra & Solr
>>
>>
>>
>>
>
Re: EMR vCores and slot allocation
Posted by Robert Metzger <rm...@apache.org>.
Hi Ken,
sorry for the late response. The number of CPU cores we show in the web
interface is based on what the JVM tells us from
"Runtime.getRuntime().availableProcessors();". I'm not sure how tthe
processor count behaves on Amazon VMs.
Given that each of your servers has 8 vCores, I would set the slot count to
8 per Taskmanager.
On Fri, Apr 29, 2016 at 1:58 AM, Ken Krugler <kk...@transpac.com>
wrote:
> Based on what Flink reports in the JobManager GUI, it looks like it thinks
> that the EC2 instances I’m using for my EMR jobs only have 4 physical cores.
>
> Which would make sense, as Amazon describes these servers as having 8
> vCores.
>
> From
> https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html,
> the recommended configuration would then be 4 slots/TaskManager, yes?
>
> Thanks,
>
> — Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>
>
>