Job manager - Zookeeper HA

[Sampath Bhat <sa...@gmail.com>]

Hello

I'm configuring flink cluster with Zookeeper for high availability for Job
managers.
These are the HA related configurations-

high-availability: zookeeper
high-availability.cluster-id: flink
high-availability.zookeeper.quorum: localhost:2181
high-availability.zookeeper.storageDir: file:///home/HA/

My concern is that high-availability.zookeeper.storageDir file location is
accessed by only job managers or even the task managers will access it.

My understanding was that the shared file location is used for storing of
job manager meta data such as execution states and there is no need for
task managers to access it.
Correct me if I'm wrong.

Re: Job manager - Zookeeper HA

[Till Rohrmann <tr...@apache.org>]

Hi Sampath,

it is correct that `high-availability.zookeeper.storageDir` is used to
persist job data which is needed to recover a job. This is the JobGraph
itself, its jars as well as checkpoint meta data.

The TaskManager does not need access to this directory, only the
JobManagers do. However, there is one optimization and one special case.

The optimization is when transferring blobs around in your cluster. Instead
of opening a Java socket and sending around the job jars, TaskManager check
first whether they can access the directory in which the job jars are
stored. Usually this is a DFS to which they also have access. If this is
the case, then they will access the file directly via the DFS instead of
taking a detour via the JobManager.

The special case is when using the MemoryStateBackend. When using this
state backend and if you haven't specified a checkpointing directory, then
it will use this directory to create a random folder in which it stores the
checkpoints. This assumes that the ZooKeeper storage directory is
accessible by every Flink component.

Cheers,
Till

On Mon, Mar 19, 2018 at 2:36 PM, Sampath Bhat <sa...@gmail.com>
wrote:

> Hello
>
> I'm configuring flink cluster with Zookeeper for high availability for Job
> managers.
> These are the HA related configurations-
>
> high-availability: zookeeper
> high-availability.cluster-id: flink
> high-availability.zookeeper.quorum: localhost:2181
> high-availability.zookeeper.storageDir: file:///home/HA/
>
> My concern is that high-availability.zookeeper.storageDir file location is
> accessed by only job managers or even the task managers will access it.
>
> My understanding was that the shared file location is used for storing of
> job manager meta data such as execution states and there is no need for
> task managers to access it.
> Correct me if I'm wrong.
>

Re: Job manager - Zookeeper HA

[Liangfei Su <su...@gmail.com>]

You're right. It's job-manager read and distribute jar files. If you are
using HDFS, you should be able to check no connection from TM to hadoop.

On Mon, Mar 19, 2018 at 9:36 PM, Sampath Bhat <sa...@gmail.com>
wrote:

> Hello
>
> I'm configuring flink cluster with Zookeeper for high availability for Job
> managers.
> These are the HA related configurations-
>
> high-availability: zookeeper
> high-availability.cluster-id: flink
> high-availability.zookeeper.quorum: localhost:2181
> high-availability.zookeeper.storageDir: file:///home/HA/
>
> My concern is that high-availability.zookeeper.storageDir file location is
> accessed by only job managers or even the task managers will access it.
>
> My understanding was that the shared file location is used for storing of
> job manager meta data such as execution states and there is no need for
> task managers to access it.
> Correct me if I'm wrong.
>