Extending Flink Slots when running on Yarn

[Julio Biason <ju...@azion.com>]

Hi,

I'm wodering, is there a way to make Flink understand that I added a new
worker machine on yarn to increase the tasks slots?

Right now, we are just exploring how to run Flink on Yarn and, so far, we
managed to create a small Hadoop/Yarn cluster (3 machines, 1 master and 2
workers) and start Flink on it.

But say I want to extend this cluster to add more machines; in this case,
simply adding more machines to the Hadoop cluster will work?

-- 
*Julio Biason*, Sofware Engineer
*AZION*  |  Deliver. Accelerate. Protect.
Office: +55 51 3083 8101 <callto:+555130838101>  |  Mobile: +55 51
<callto:+5551996209291>*99907 0554*

Re: Extending Flink Slots when running on Yarn

[Gary Yao <ga...@data-artisans.com>]

Hi Julio,

When you start the Flink YARN session, you have to specify the number of
TaskManagers and number of slots per TaskManager. There is currently no
officially supported way to add more TaskManagers to a long running YARN
session. We are aware of this limitation, and there are ongoing development
efforts to make the Flink Cluster dynamically allocate new YARN containers
when
needed (FLIP-6) [1]. We are aiming at shipping this feature with the Flink
1.5
release [2]. As a workaround, you can restore your jobs from savepoints on a
bigger cluster.

In a standalone cluster it is possible to add TaskManagers to a running
cluster
[3].

Best,
Gary

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077
[2]
https://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html
[3]
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/cluster_setup.html#adding-jobmanagertaskmanager-instances-to-a-cluster

On Thu, Feb 1, 2018 at 5:09 PM, Julio Biason <ju...@azion.com> wrote:

> Hi,
>
> I'm wodering, is there a way to make Flink understand that I added a new
> worker machine on yarn to increase the tasks slots?
>
> Right now, we are just exploring how to run Flink on Yarn and, so far, we
> managed to create a small Hadoop/Yarn cluster (3 machines, 1 master and 2
> workers) and start Flink on it.
>
> But say I want to extend this cluster to add more machines; in this case,
> simply adding more machines to the Hadoop cluster will work?
>
> --
> *Julio Biason*, Sofware Engineer
> *AZION*  |  Deliver. Accelerate. Protect.
> Office: +55 51 3083 8101 <callto:+555130838101>  |  Mobile: +55 51
> <callto:+5551996209291>*99907 0554*
>