Re: Utilising EMR's master node

[Gary Yao <ga...@data-artisans.com>]

Hi Averell,

It is up to the YARN scheduler on which hosts the containers are started.

What Flink version are you using? I assume you are using 1.4 or earlier
because you are specifying a fixed number of TMs. If you launch Flink with
-yn
2, you should be only seeing 2 TMs in total (not 4). Are you starting two
clusters?

Beginning with Flink 1.5, -yn is obsolete because resources are acquired
dynamically, and it is not well-defined in what order TM slots are exhausted
[1].

Best,
Gary

[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-5-job-distribution-over-cluster-nodes-td23364.html

On Wed, Sep 26, 2018 at 9:25 AM Averell <lv...@gmail.com> wrote:

> Thank you Gary.
> Regarding your previous suggestion to to change the configuration regarding
> to the number of vcores on the EMR master node, I tried and found one
> funny/bad behaviour as following:
>  * hardware onfiguration: master node: 4vcores + 8GB ram, 2x executors with
> 16vcores + 32GB ram each.
>  * Flink launch parameters: -yn 2 -ys 16 -ytm 4g...
> 4 TMs were created, with 2 of them were used (0 free slots) and two others
> not used (16 free slots). The bad thing is most of the time 2 free TMs are
> on a same machine, and two occupied ones are on the other machine.
> If I dont change the Hadoop configurations then still 4 TMs created, but
> the
> occupied ones are always on two different servers.
>
> I'm not sure whether that's EMR's issue, or YARN's or Flink's.
>
> Thanks and regards,
> Averell
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

Re: Utilising EMR's master node

[Averell <lv...@gmail.com>]

Hi Gary,

Thanks for the information. I didn't know that -yn is obsolete :( I am using
Flink 1.6.
Not sure whether that's a bug when I tried to set -yn explicitly, but I
started only 1 cluster.

Thanks and regards,
Averell



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/