You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Andrea Spina <an...@radicalbit.io> on 2019/03/18 12:23:52 UTC
Re: Flink 1.7.2: All jobs are getting deployed on the same task manager
Hi everybody. We're currently experimenting the same characteristic on
flink-1.6.2.
I've been reading that Flink treats all the slot as equals, it doesn't even
know where these slots reside
https://stackoverflow.com/questions/54980104/uneven-assignment-of-tasks-to-workers-in-flink.
So it should not be an issue; thus, the fact that it runs all the slots of
a machine before moving to a new one should be just a rough coincidence.
Given that, I'm pretty sure that I've never been recording this feature
using previous majors (I recall flink-1.3 for sure).
Moreover, this is damaging because you can get resources exhausted (e.g.
memory, disk).
Hope we might find a solution on this.
Sincerely,
Andrea
Il giorno lun 18 mar 2019 alle ore 11:53 Kumar Bolar, Harshith <
hkumv@arity.com> ha scritto:
> Hi all,
>
>
>
> We're running a Flink on a five node standalone cluster with three task
> manager (TM1, TM2, TM3) and two job managers.
>
>
>
> Whenever I submit a new job, the job gets deployed on only TM3. When the
> number of slots in TM3 get exhausted, the jobs start getting deployed on
> TM2 and so on. How do I ensure that the jobs get distributed evenly across
> all 3 task managers?
>
>
>
> Thanks,
>
> Harshith
>
>
>
--
*Andrea Spina*
Software Engineer @ Radicalbit Srl
Via Borsieri 41, 20159, Milano - IT
Re: Re: Flink 1.7.2: All jobs are getting deployed on the same task manager
Posted by Andrey Zagrebin <an...@ververica.com>.
Hi Kumar and Andrea,
this is a known change in Flink behaviour from 1.4 to 1.5 (after FLIP-6).
There is an issue to track progress on more fine-grained task distribution
[1].
Best,
Andrey
[1] https://issues.apache.org/jira/browse/FLINK-11815
On Mon, Mar 18, 2019 at 1:28 PM Kumar Bolar, Harshith <hk...@arity.com>
wrote:
> We recently upgraded to 1.7.2. Previously in 1.4.2 the jobs were getting
> deployed in a round robin manner. First job goes to TM1, second job goes to
> TM2 and so on. But after upgrading to 1.7.2, all jobs are going to the same
> task manager until the slots there run out. Did something change in 1.7?
>
>
>
> Thanks,
>
> Harshith
>
>
>
> *From: *Andrea Spina <an...@radicalbit.io>
> *Date: *Monday, 18 March 2019 at 5:54 PM
> *To: *Harshith Kumar Bolar <hk...@arity.com>
> *Cc: *user <us...@flink.apache.org>
> *Subject: *[External] Re: Flink 1.7.2: All jobs are getting deployed on
> the same task manager
>
>
>
> Hi everybody. We're currently experimenting the same characteristic on
> flink-1.6.2.
>
> I've been reading that Flink treats all the slot as equals, it doesn't
> even know where these slots reside
> https://stackoverflow.com/questions/54980104/uneven-assignment-of-tasks-to-workers-in-flink
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_54980104_uneven-2Dassignment-2Dof-2Dtasks-2Dto-2Dworkers-2Din-2Dflink&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=x7Ae5P9d7Ni3EkdbM3aw6gaRzqYcCoFlSAmXCz-8n-A&s=mLWGWKLiWiUyx1n_16tJQt3E3iAbaoi4GMH40WordpA&e=>.
>
> So it should not be an issue; thus, the fact that it runs all the slots of
> a machine before moving to a new one should be just a rough coincidence.
>
> Given that, I'm pretty sure that I've never been recording this feature
> using previous majors (I recall flink-1.3 for sure).
> Moreover, this is damaging because you can get resources exhausted (e.g.
> memory, disk).
>
> Hope we might find a solution on this.
> Sincerely,
>
> Andrea
>
>
>
> Il giorno lun 18 mar 2019 alle ore 11:53 Kumar Bolar, Harshith <
> hkumv@arity.com> ha scritto:
>
> Hi all,
>
>
>
> We're running a Flink on a five node standalone cluster with three task
> manager (TM1, TM2, TM3) and two job managers.
>
>
>
> Whenever I submit a new job, the job gets deployed on only TM3. When the
> number of slots in TM3 get exhausted, the jobs start getting deployed on
> TM2 and so on. How do I ensure that the jobs get distributed evenly across
> all 3 task managers?
>
>
>
> Thanks,
>
> Harshith
>
>
>
>
>
>
> --
>
> *Andrea Spina*
> Software Engineer @ Radicalbit Srl
> Via Borsieri 41, 20159, Milano - IT
>
Re: Re: Flink 1.7.2: All jobs are getting deployed on the same task
manager
Posted by "Kumar Bolar, Harshith" <hk...@arity.com>.
We recently upgraded to 1.7.2. Previously in 1.4.2 the jobs were getting deployed in a round robin manner. First job goes to TM1, second job goes to TM2 and so on. But after upgrading to 1.7.2, all jobs are going to the same task manager until the slots there run out. Did something change in 1.7?
Thanks,
Harshith
From: Andrea Spina <an...@radicalbit.io>
Date: Monday, 18 March 2019 at 5:54 PM
To: Harshith Kumar Bolar <hk...@arity.com>
Cc: user <us...@flink.apache.org>
Subject: [External] Re: Flink 1.7.2: All jobs are getting deployed on the same task manager
Hi everybody. We're currently experimenting the same characteristic on flink-1.6.2.
I've been reading that Flink treats all the slot as equals, it doesn't even know where these slots reside https://stackoverflow.com/questions/54980104/uneven-assignment-of-tasks-to-workers-in-flink<https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_54980104_uneven-2Dassignment-2Dof-2Dtasks-2Dto-2Dworkers-2Din-2Dflink&d=DwMFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=x7Ae5P9d7Ni3EkdbM3aw6gaRzqYcCoFlSAmXCz-8n-A&s=mLWGWKLiWiUyx1n_16tJQt3E3iAbaoi4GMH40WordpA&e=>.
So it should not be an issue; thus, the fact that it runs all the slots of a machine before moving to a new one should be just a rough coincidence.
Given that, I'm pretty sure that I've never been recording this feature using previous majors (I recall flink-1.3 for sure).
Moreover, this is damaging because you can get resources exhausted (e.g. memory, disk).
Hope we might find a solution on this.
Sincerely,
Andrea
Il giorno lun 18 mar 2019 alle ore 11:53 Kumar Bolar, Harshith <hk...@arity.com>> ha scritto:
Hi all,
We're running a Flink on a five node standalone cluster with three task manager (TM1, TM2, TM3) and two job managers.
Whenever I submit a new job, the job gets deployed on only TM3. When the number of slots in TM3 get exhausted, the jobs start getting deployed on TM2 and so on. How do I ensure that the jobs get distributed evenly across all 3 task managers?
Thanks,
Harshith
--
Andrea Spina
Software Engineer @ Radicalbit Srl
Via Borsieri 41, 20159, Milano - IT