You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by PedroMrChaves <pe...@gmail.com> on 2017/03/08 09:44:44 UTC
Isolate Tasks - Run Distinct Tasks in Different Task Managers
Hello,
Assuming that I have the following Job Graph,
(Source) -> (map) -> (KeyBy | Window | apply) -> (Sink)
Is there a way to assure that the map operator (and all its subtasks) run on
a different
task manager than the operator (map | window | apply)?
This would allow JVM memory isolation without using YARN.
Regards,
Pedro
-----
Best Regards,
Pedro Chaves
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-Run-Distinct-Tasks-in-Different-Task-Managers-tp12104.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers
Posted by Stephan Ewen <se...@apache.org>.
Yes, simply set the number of slots per TaskManager in YARN to 1. That
gives you the isolation.
On Tue, Mar 14, 2017 at 11:58 AM, PedroMrChaves <pe...@gmail.com>
wrote:
> Can YARN provide task isolation?
>
>
>
>
> -----
> Best Regards,
> Pedro Chaves
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-
> Run-Distinct-Tasks-in-Different-Task-Managers-tp12104p12201.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers
Posted by PedroMrChaves <pe...@gmail.com>.
Can YARN provide task isolation?
-----
Best Regards,
Pedro Chaves
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-Run-Distinct-Tasks-in-Different-Task-Managers-tp12104p12201.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers
Posted by Robert Metzger <rm...@apache.org>.
There is currently no way in Flink to define such scheduling constraints.
On Wed, Mar 8, 2017 at 5:00 PM, PedroMrChaves <pe...@gmail.com>
wrote:
> Thanks for the response.
>
> I would like to assure that the map operator is not in the same task
> manager
> as the window/apply operator, regardless of the number of slots of each
> task
> manager.
>
>
>
> -----
> Best Regards,
> Pedro Chaves
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-
> Run-Distinct-Tasks-in-Different-Task-Managers-tp12104p12118.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers
Posted by PedroMrChaves <pe...@gmail.com>.
Thanks for the response.
I would like to assure that the map operator is not in the same task manager
as the window/apply operator, regardless of the number of slots of each task
manager.
-----
Best Regards,
Pedro Chaves
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-Run-Distinct-Tasks-in-Different-Task-Managers-tp12104p12118.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Isolate Tasks - Run Distinct Tasks in Different Task Managers
Posted by Ufuk Celebi <uc...@apache.org>.
Internally, Flink defines through SlotSharingGroup which tasks may
share a task manager slot. By configuring each TaskManager to have a
single slot and configuring the slot sharing groups accordingly, you
can get the desired behaviour.
You can specify the slot sharing group for an operator like map by
calling slotSharingGroup(String). You would have you set a different
slot sharing group for the window/apply part.
Does this help?
On Wed, Mar 8, 2017 at 10:44 AM, PedroMrChaves
<pe...@gmail.com> wrote:
> Hello,
>
> Assuming that I have the following Job Graph,
>
> (Source) -> (map) -> (KeyBy | Window | apply) -> (Sink)
>
> Is there a way to assure that the map operator (and all its subtasks) run on
> a different
> task manager than the operator (map | window | apply)?
>
> This would allow JVM memory isolation without using YARN.
>
> Regards,
> Pedro
>
>
>
>
>
> -----
> Best Regards,
> Pedro Chaves
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Isolate-Tasks-Run-Distinct-Tasks-in-Different-Task-Managers-tp12104.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.