You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Sourav Mazumder <so...@gmail.com> on 2016/01/11 17:18:46 UTC

Flink with Yarn

I am going through the documentation of integrating Flink with YARN.

However not sure whether Flink can be run on YARN in two modes (like
Spark). In one mode the driver/client program of Flink is also managed by
YARN. In the second mode where the client program is outside the control of
YARN. Is the running Flinkon behind Firewalls is like the second mode

Any clarification on this ?

Regards,
Sourav

Re: Flink with Yarn

Posted by Sourav Mazumder <so...@gmail.com>.

Hi Gordon,

Thanks for the explanation. It is much clear now. Looks like a much cleaner
approach. In that way the driver program can run in a machine which does
not need connectivity to all worker nodes.

Regards,
Sourav

On Mon, Jan 11, 2016 at 9:22 AM, Tzu-Li (Gordon) Tai <tz...@gmail.com>
wrote:

> Hi Sourav,
>
> A little help with more clarification on your last comment.
>
> In sense of "where" the driver program is executed, then yes the Flink
> driver program runs in a mode similar to Spark's YARN-client.
>
> However, the "role" of the driver program and the work that it is
> responsible of is quite different between Flink and Spark. In Spark, the
> driver program is in charge of coordinating Spark workers (executors) and
> must listen for and accept incoming connections from the workers throughout
> the job's lifetime. Therefore, in Spark's YARN-client mode, you must keep
> the driver program process alive otherwise the job will be shutdown.
>
> However, in Flink, the coordination of Flink TaskManagers to complete a job
> is handled by Flink's JobManager once the client at the driver program
> submits the job to the JobManager. The driver program is solely used for
> the
> job submission and can disconnect afterwards.
>
> Like what Stephan explained, if the user-defined dataflow defines any
> intermediate results to be retrieved via collect() or print(), the results
> are transmitted through the JobManager. Only then does the driver program
> need to stay connected. Note that this connection still does not need to
> have any connections with the workers (Flink TaskManagers), only the
> JobManager.
>
> Cheers,
> Gordon
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-with-Yarn-tp4224p4227.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Re: Flink with Yarn

Posted by "Tzu-Li (Gordon) Tai" <tz...@gmail.com>.

Hi Sourav,

A little help with more clarification on your last comment.

In sense of "where" the driver program is executed, then yes the Flink
driver program runs in a mode similar to Spark's YARN-client.

However, the "role" of the driver program and the work that it is
responsible of is quite different between Flink and Spark. In Spark, the
driver program is in charge of coordinating Spark workers (executors) and
must listen for and accept incoming connections from the workers throughout
the job's lifetime. Therefore, in Spark's YARN-client mode, you must keep
the driver program process alive otherwise the job will be shutdown.

However, in Flink, the coordination of Flink TaskManagers to complete a job
is handled by Flink's JobManager once the client at the driver program
submits the job to the JobManager. The driver program is solely used for the
job submission and can disconnect afterwards.

Like what Stephan explained, if the user-defined dataflow defines any
intermediate results to be retrieved via collect() or print(), the results
are transmitted through the JobManager. Only then does the driver program
need to stay connected. Note that this connection still does not need to
have any connections with the workers (Flink TaskManagers), only the
JobManager.

Cheers,
Gordon

--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-with-Yarn-tp4224p4227.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: Flink with Yarn

Posted by Sourav Mazumder <so...@gmail.com>.

Hi Stephan,

Thanks for the explanation.

>From your explanation it looks like Flink runs in a mode similar to Spark's
YARN-Client mode.

Regards,
Sourav

On Mon, Jan 11, 2016 at 8:27 AM, Stephan Ewen <se...@apache.org> wrote:

> Hi!
>
> Flink is different than Spark in that respect. The driver program in Flink
> can submit a program to the master (in YARN Application Master) and
> disconnect then. It is not a part of the distributed execution - that is
> coordinated only by the master (JobManager).
> The driver can stay connected to receive progress updates, though.
>
> For programs that do consist of multiple parallel executions (that have
> count() or collect() statements), the driver needs to stay connected,
> because it needs to pull the intermediate results. However, they are all
> pulled/proxied through the master (JobManager), so the driver needs not be
> able to connect to the workers. The only requirement for firewalled
> clusters is to have two ports from the master node reachable by the client.
>
> Greetings,
> Stephan
>
>
> On Mon, Jan 11, 2016 at 5:18 PM, Sourav Mazumder <
> sourav.mazumder00@gmail.com> wrote:
>
>> I am going through the documentation of integrating Flink with YARN.
>>
>> However not sure whether Flink can be run on YARN in two modes (like
>> Spark). In one mode the driver/client program of Flink is also managed by
>> YARN. In the second mode where the client program is outside the control of
>> YARN. Is the running Flinkon behind Firewalls is like the second mode
>>
>> Any clarification on this ?
>>
>> Regards,
>> Sourav
>>
>
>

Re: Flink with Yarn

Posted by Stephan Ewen <se...@apache.org>.

Hi!

Flink is different than Spark in that respect. The driver program in Flink
can submit a program to the master (in YARN Application Master) and
disconnect then. It is not a part of the distributed execution - that is
coordinated only by the master (JobManager).
The driver can stay connected to receive progress updates, though.

For programs that do consist of multiple parallel executions (that have
count() or collect() statements), the driver needs to stay connected,
because it needs to pull the intermediate results. However, they are all
pulled/proxied through the master (JobManager), so the driver needs not be
able to connect to the workers. The only requirement for firewalled
clusters is to have two ports from the master node reachable by the client.

Greetings,
Stephan

On Mon, Jan 11, 2016 at 5:18 PM, Sourav Mazumder <
sourav.mazumder00@gmail.com> wrote:

> I am going through the documentation of integrating Flink with YARN.
>
> However not sure whether Flink can be run on YARN in two modes (like
> Spark). In one mode the driver/client program of Flink is also managed by
> YARN. In the second mode where the client program is outside the control of
> YARN. Is the running Flinkon behind Firewalls is like the second mode
>
> Any clarification on this ?
>
> Regards,
> Sourav
>