You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sampath Bhat <sa...@gmail.com> on 2018/07/13 09:48:47 UTC
Flink CLI properties with HA
Hello
When HA is enabled in the flink cluster and if I've to submit job via flink
CLI then in the flink-conf.yaml of flink CLI should contain this properties
-
high-availability: zookeeper
high-availability.cluster-id: flink
high-availability.zookeeper.path.root: flink
high-availability.storageDir: <some path>
high-availability.zookeeper.quorum: <zookeeper IP:port>
What is the need of high-availability.storageDir for flink CLI. Does this
mean that even flink client should be able to access the mentioned path or
is it some check being done on the property name?
Without these properties flink cli will not be able to submit job to flink
cluster when HA is enabled.
Re: Flink CLI properties with HA
Posted by Sampath Bhat <sa...@gmail.com>.
Vino, I'm not getting any error but my suspicion was that if I dont specify
this `high-availability.storageDir` property in flink CLI side then the CLI
will not be able to submit job to flink cluster(HA enabled). But if provide
this property in CLI side the job submission will be successful even though
the CLI cannot access the path mentioned in `high-availability.storageDir`.
So I wanted to understand the underlying implementation.
Till, Thank you for the reply. It clarified my doubt.
On Tue, Jul 17, 2018 at 6:03 PM, Till Rohrmann <tr...@apache.org> wrote:
> Hi Sampath,
>
> technically the client does not need to know the
> `high-availability.storageDir` to submit a job. However, due to how we
> construct the ZooKeeperHaServices it is still needed. The reason behind
> this is that we use the same services for the server and the client. Thus,
> the implementation needs to know the storageDir in both cases. The way it
> should be done is to split the HighAvailabilityServices up into client and
> server services. The former would then not depend on
> `high-availability.storageDir`.
>
> Cheers,
> Till
>
> On Tue, Jul 17, 2018 at 1:31 PM vino yang <ya...@gmail.com> wrote:
>
>> Hi Sampath,
>>
>> It seems Flink CLI for standalone would not access
>> *high-availability.storageDir.*
>>
>> What's the exception stack trace in your environment?
>>
>> Thanks, vino.
>>
>> 2018-07-17 15:08 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>>
>>> Hi vino
>>>
>>> Should the flink CLI have access to the path mentioned in
>>> *high-availability.storageDir*?
>>> If my flink cluster is on set of machines and i submit my job from flink
>>> CLI from another independent machine by giving necessary details will the
>>> CLI try to access *high-availability.storageDir *path?
>>>
>>> I'm aware of the fact that flink client will connect to zookeeper to get
>>> leader address and necessary information for job submission but my
>>> confusion is with *high-availability.storageDir* and its necessity in
>>> flink CLI configuration.
>>>
>>> On Mon, Jul 16, 2018 at 2:44 PM, vino yang <ya...@gmail.com>
>>> wrote:
>>>
>>>> Hi Sampath,
>>>>
>>>> Flink CLI need to retrieve the JobManager leader address, so it need
>>>> to access the HA specific configuration. Because if based on Zookeeper to
>>>> implement the HA, the leader address information will fetch from Zookeeper.
>>>>
>>>> The main use of config item *high-availability.storageDir* is storage
>>>> (Job graph, checkpoint and so on). Actually, the real data is stored under
>>>> this path which used to recover purpose, zookeeper just store a state
>>>> handle.
>>>>
>>>> ---
>>>> Thanks.
>>>> vino.
>>>>
>>>>
>>>> 2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Sampath Bhat <sa...@gmail.com>
>>>>> Date: Fri, Jul 13, 2018 at 3:18 PM
>>>>> Subject: Flink CLI properties with HA
>>>>> To: user <us...@flink.apache.org>
>>>>>
>>>>>
>>>>> Hello
>>>>>
>>>>> When HA is enabled in the flink cluster and if I've to submit job via
>>>>> flink CLI then in the flink-conf.yaml of flink CLI should contain this
>>>>> properties -
>>>>> high-availability: zookeeper
>>>>> high-availability.cluster-id: flink
>>>>> high-availability.zookeeper.path.root: flink
>>>>> high-availability.storageDir: <some path>
>>>>> high-availability.zookeeper.quorum: <zookeeper IP:port>
>>>>>
>>>>> What is the need of high-availability.storageDir for flink CLI. Does
>>>>> this mean that even flink client should be able to access the mentioned
>>>>> path or is it some check being done on the property name?
>>>>>
>>>>> Without these properties flink cli will not be able to submit job to
>>>>> flink cluster when HA is enabled.
>>>>>
>>>>>
>>>>
>>>
>>
Re: Flink CLI properties with HA
Posted by Sampath Bhat <sa...@gmail.com>.
Vino, I'm not getting any error but my suspicion was that if I dont specify
this `high-availability.storageDir` property in flink CLI side then the CLI
will not be able to submit job to flink cluster(HA enabled). But if provide
this property in CLI side the job submission will be successful even though
the CLI cannot access the path mentioned in `high-availability.storageDir`.
So I wanted to understand the underlying implementation.
Till, Thank you for the reply. It clarified my doubt.
On Tue, Jul 17, 2018 at 6:03 PM, Till Rohrmann <tr...@apache.org> wrote:
> Hi Sampath,
>
> technically the client does not need to know the
> `high-availability.storageDir` to submit a job. However, due to how we
> construct the ZooKeeperHaServices it is still needed. The reason behind
> this is that we use the same services for the server and the client. Thus,
> the implementation needs to know the storageDir in both cases. The way it
> should be done is to split the HighAvailabilityServices up into client and
> server services. The former would then not depend on
> `high-availability.storageDir`.
>
> Cheers,
> Till
>
> On Tue, Jul 17, 2018 at 1:31 PM vino yang <ya...@gmail.com> wrote:
>
>> Hi Sampath,
>>
>> It seems Flink CLI for standalone would not access
>> *high-availability.storageDir.*
>>
>> What's the exception stack trace in your environment?
>>
>> Thanks, vino.
>>
>> 2018-07-17 15:08 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>>
>>> Hi vino
>>>
>>> Should the flink CLI have access to the path mentioned in
>>> *high-availability.storageDir*?
>>> If my flink cluster is on set of machines and i submit my job from flink
>>> CLI from another independent machine by giving necessary details will the
>>> CLI try to access *high-availability.storageDir *path?
>>>
>>> I'm aware of the fact that flink client will connect to zookeeper to get
>>> leader address and necessary information for job submission but my
>>> confusion is with *high-availability.storageDir* and its necessity in
>>> flink CLI configuration.
>>>
>>> On Mon, Jul 16, 2018 at 2:44 PM, vino yang <ya...@gmail.com>
>>> wrote:
>>>
>>>> Hi Sampath,
>>>>
>>>> Flink CLI need to retrieve the JobManager leader address, so it need
>>>> to access the HA specific configuration. Because if based on Zookeeper to
>>>> implement the HA, the leader address information will fetch from Zookeeper.
>>>>
>>>> The main use of config item *high-availability.storageDir* is storage
>>>> (Job graph, checkpoint and so on). Actually, the real data is stored under
>>>> this path which used to recover purpose, zookeeper just store a state
>>>> handle.
>>>>
>>>> ---
>>>> Thanks.
>>>> vino.
>>>>
>>>>
>>>> 2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Sampath Bhat <sa...@gmail.com>
>>>>> Date: Fri, Jul 13, 2018 at 3:18 PM
>>>>> Subject: Flink CLI properties with HA
>>>>> To: user <us...@flink.apache.org>
>>>>>
>>>>>
>>>>> Hello
>>>>>
>>>>> When HA is enabled in the flink cluster and if I've to submit job via
>>>>> flink CLI then in the flink-conf.yaml of flink CLI should contain this
>>>>> properties -
>>>>> high-availability: zookeeper
>>>>> high-availability.cluster-id: flink
>>>>> high-availability.zookeeper.path.root: flink
>>>>> high-availability.storageDir: <some path>
>>>>> high-availability.zookeeper.quorum: <zookeeper IP:port>
>>>>>
>>>>> What is the need of high-availability.storageDir for flink CLI. Does
>>>>> this mean that even flink client should be able to access the mentioned
>>>>> path or is it some check being done on the property name?
>>>>>
>>>>> Without these properties flink cli will not be able to submit job to
>>>>> flink cluster when HA is enabled.
>>>>>
>>>>>
>>>>
>>>
>>
Re: Flink CLI properties with HA
Posted by Till Rohrmann <tr...@apache.org>.
Hi Sampath,
technically the client does not need to know the
`high-availability.storageDir` to submit a job. However, due to how we
construct the ZooKeeperHaServices it is still needed. The reason behind
this is that we use the same services for the server and the client. Thus,
the implementation needs to know the storageDir in both cases. The way it
should be done is to split the HighAvailabilityServices up into client and
server services. The former would then not depend on
`high-availability.storageDir`.
Cheers,
Till
On Tue, Jul 17, 2018 at 1:31 PM vino yang <ya...@gmail.com> wrote:
> Hi Sampath,
>
> It seems Flink CLI for standalone would not access
> *high-availability.storageDir.*
>
> What's the exception stack trace in your environment?
>
> Thanks, vino.
>
> 2018-07-17 15:08 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>
>> Hi vino
>>
>> Should the flink CLI have access to the path mentioned in
>> *high-availability.storageDir*?
>> If my flink cluster is on set of machines and i submit my job from flink
>> CLI from another independent machine by giving necessary details will the
>> CLI try to access *high-availability.storageDir *path?
>>
>> I'm aware of the fact that flink client will connect to zookeeper to get
>> leader address and necessary information for job submission but my
>> confusion is with *high-availability.storageDir* and its necessity in
>> flink CLI configuration.
>>
>> On Mon, Jul 16, 2018 at 2:44 PM, vino yang <ya...@gmail.com> wrote:
>>
>>> Hi Sampath,
>>>
>>> Flink CLI need to retrieve the JobManager leader address, so it need to
>>> access the HA specific configuration. Because if based on Zookeeper to
>>> implement the HA, the leader address information will fetch from Zookeeper.
>>>
>>> The main use of config item *high-availability.storageDir* is storage
>>> (Job graph, checkpoint and so on). Actually, the real data is stored under
>>> this path which used to recover purpose, zookeeper just store a state
>>> handle.
>>>
>>> ---
>>> Thanks.
>>> vino.
>>>
>>>
>>> 2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Sampath Bhat <sa...@gmail.com>
>>>> Date: Fri, Jul 13, 2018 at 3:18 PM
>>>> Subject: Flink CLI properties with HA
>>>> To: user <us...@flink.apache.org>
>>>>
>>>>
>>>> Hello
>>>>
>>>> When HA is enabled in the flink cluster and if I've to submit job via
>>>> flink CLI then in the flink-conf.yaml of flink CLI should contain this
>>>> properties -
>>>> high-availability: zookeeper
>>>> high-availability.cluster-id: flink
>>>> high-availability.zookeeper.path.root: flink
>>>> high-availability.storageDir: <some path>
>>>> high-availability.zookeeper.quorum: <zookeeper IP:port>
>>>>
>>>> What is the need of high-availability.storageDir for flink CLI. Does
>>>> this mean that even flink client should be able to access the mentioned
>>>> path or is it some check being done on the property name?
>>>>
>>>> Without these properties flink cli will not be able to submit job to
>>>> flink cluster when HA is enabled.
>>>>
>>>>
>>>
>>
>
Re: Flink CLI properties with HA
Posted by Till Rohrmann <tr...@apache.org>.
Hi Sampath,
technically the client does not need to know the
`high-availability.storageDir` to submit a job. However, due to how we
construct the ZooKeeperHaServices it is still needed. The reason behind
this is that we use the same services for the server and the client. Thus,
the implementation needs to know the storageDir in both cases. The way it
should be done is to split the HighAvailabilityServices up into client and
server services. The former would then not depend on
`high-availability.storageDir`.
Cheers,
Till
On Tue, Jul 17, 2018 at 1:31 PM vino yang <ya...@gmail.com> wrote:
> Hi Sampath,
>
> It seems Flink CLI for standalone would not access
> *high-availability.storageDir.*
>
> What's the exception stack trace in your environment?
>
> Thanks, vino.
>
> 2018-07-17 15:08 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>
>> Hi vino
>>
>> Should the flink CLI have access to the path mentioned in
>> *high-availability.storageDir*?
>> If my flink cluster is on set of machines and i submit my job from flink
>> CLI from another independent machine by giving necessary details will the
>> CLI try to access *high-availability.storageDir *path?
>>
>> I'm aware of the fact that flink client will connect to zookeeper to get
>> leader address and necessary information for job submission but my
>> confusion is with *high-availability.storageDir* and its necessity in
>> flink CLI configuration.
>>
>> On Mon, Jul 16, 2018 at 2:44 PM, vino yang <ya...@gmail.com> wrote:
>>
>>> Hi Sampath,
>>>
>>> Flink CLI need to retrieve the JobManager leader address, so it need to
>>> access the HA specific configuration. Because if based on Zookeeper to
>>> implement the HA, the leader address information will fetch from Zookeeper.
>>>
>>> The main use of config item *high-availability.storageDir* is storage
>>> (Job graph, checkpoint and so on). Actually, the real data is stored under
>>> this path which used to recover purpose, zookeeper just store a state
>>> handle.
>>>
>>> ---
>>> Thanks.
>>> vino.
>>>
>>>
>>> 2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Sampath Bhat <sa...@gmail.com>
>>>> Date: Fri, Jul 13, 2018 at 3:18 PM
>>>> Subject: Flink CLI properties with HA
>>>> To: user <us...@flink.apache.org>
>>>>
>>>>
>>>> Hello
>>>>
>>>> When HA is enabled in the flink cluster and if I've to submit job via
>>>> flink CLI then in the flink-conf.yaml of flink CLI should contain this
>>>> properties -
>>>> high-availability: zookeeper
>>>> high-availability.cluster-id: flink
>>>> high-availability.zookeeper.path.root: flink
>>>> high-availability.storageDir: <some path>
>>>> high-availability.zookeeper.quorum: <zookeeper IP:port>
>>>>
>>>> What is the need of high-availability.storageDir for flink CLI. Does
>>>> this mean that even flink client should be able to access the mentioned
>>>> path or is it some check being done on the property name?
>>>>
>>>> Without these properties flink cli will not be able to submit job to
>>>> flink cluster when HA is enabled.
>>>>
>>>>
>>>
>>
>
Re: Flink CLI properties with HA
Posted by vino yang <ya...@gmail.com>.
Hi Sampath,
It seems Flink CLI for standalone would not access
*high-availability.storageDir.*
What's the exception stack trace in your environment?
Thanks, vino.
2018-07-17 15:08 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
> Hi vino
>
> Should the flink CLI have access to the path mentioned in
> *high-availability.storageDir*?
> If my flink cluster is on set of machines and i submit my job from flink
> CLI from another independent machine by giving necessary details will the
> CLI try to access *high-availability.storageDir *path?
>
> I'm aware of the fact that flink client will connect to zookeeper to get
> leader address and necessary information for job submission but my
> confusion is with *high-availability.storageDir* and its necessity in
> flink CLI configuration.
>
> On Mon, Jul 16, 2018 at 2:44 PM, vino yang <ya...@gmail.com> wrote:
>
>> Hi Sampath,
>>
>> Flink CLI need to retrieve the JobManager leader address, so it need to
>> access the HA specific configuration. Because if based on Zookeeper to
>> implement the HA, the leader address information will fetch from Zookeeper.
>>
>> The main use of config item *high-availability.storageDir* is storage
>> (Job graph, checkpoint and so on). Actually, the real data is stored under
>> this path which used to recover purpose, zookeeper just store a state
>> handle.
>>
>> ---
>> Thanks.
>> vino.
>>
>>
>> 2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Sampath Bhat <sa...@gmail.com>
>>> Date: Fri, Jul 13, 2018 at 3:18 PM
>>> Subject: Flink CLI properties with HA
>>> To: user <us...@flink.apache.org>
>>>
>>>
>>> Hello
>>>
>>> When HA is enabled in the flink cluster and if I've to submit job via
>>> flink CLI then in the flink-conf.yaml of flink CLI should contain this
>>> properties -
>>> high-availability: zookeeper
>>> high-availability.cluster-id: flink
>>> high-availability.zookeeper.path.root: flink
>>> high-availability.storageDir: <some path>
>>> high-availability.zookeeper.quorum: <zookeeper IP:port>
>>>
>>> What is the need of high-availability.storageDir for flink CLI. Does
>>> this mean that even flink client should be able to access the mentioned
>>> path or is it some check being done on the property name?
>>>
>>> Without these properties flink cli will not be able to submit job to
>>> flink cluster when HA is enabled.
>>>
>>>
>>
>
Re: Flink CLI properties with HA
Posted by vino yang <ya...@gmail.com>.
Hi Sampath,
It seems Flink CLI for standalone would not access
*high-availability.storageDir.*
What's the exception stack trace in your environment?
Thanks, vino.
2018-07-17 15:08 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
> Hi vino
>
> Should the flink CLI have access to the path mentioned in
> *high-availability.storageDir*?
> If my flink cluster is on set of machines and i submit my job from flink
> CLI from another independent machine by giving necessary details will the
> CLI try to access *high-availability.storageDir *path?
>
> I'm aware of the fact that flink client will connect to zookeeper to get
> leader address and necessary information for job submission but my
> confusion is with *high-availability.storageDir* and its necessity in
> flink CLI configuration.
>
> On Mon, Jul 16, 2018 at 2:44 PM, vino yang <ya...@gmail.com> wrote:
>
>> Hi Sampath,
>>
>> Flink CLI need to retrieve the JobManager leader address, so it need to
>> access the HA specific configuration. Because if based on Zookeeper to
>> implement the HA, the leader address information will fetch from Zookeeper.
>>
>> The main use of config item *high-availability.storageDir* is storage
>> (Job graph, checkpoint and so on). Actually, the real data is stored under
>> this path which used to recover purpose, zookeeper just store a state
>> handle.
>>
>> ---
>> Thanks.
>> vino.
>>
>>
>> 2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>>
>>>
>>> ---------- Forwarded message ----------
>>> From: Sampath Bhat <sa...@gmail.com>
>>> Date: Fri, Jul 13, 2018 at 3:18 PM
>>> Subject: Flink CLI properties with HA
>>> To: user <us...@flink.apache.org>
>>>
>>>
>>> Hello
>>>
>>> When HA is enabled in the flink cluster and if I've to submit job via
>>> flink CLI then in the flink-conf.yaml of flink CLI should contain this
>>> properties -
>>> high-availability: zookeeper
>>> high-availability.cluster-id: flink
>>> high-availability.zookeeper.path.root: flink
>>> high-availability.storageDir: <some path>
>>> high-availability.zookeeper.quorum: <zookeeper IP:port>
>>>
>>> What is the need of high-availability.storageDir for flink CLI. Does
>>> this mean that even flink client should be able to access the mentioned
>>> path or is it some check being done on the property name?
>>>
>>> Without these properties flink cli will not be able to submit job to
>>> flink cluster when HA is enabled.
>>>
>>>
>>
>
Re: Flink CLI properties with HA
Posted by Sampath Bhat <sa...@gmail.com>.
Hi vino
Should the flink CLI have access to the path mentioned in
*high-availability.storageDir*?
If my flink cluster is on set of machines and i submit my job from flink
CLI from another independent machine by giving necessary details will the
CLI try to access *high-availability.storageDir *path?
I'm aware of the fact that flink client will connect to zookeeper to get
leader address and necessary information for job submission but my
confusion is with *high-availability.storageDir* and its necessity in flink
CLI configuration.
On Mon, Jul 16, 2018 at 2:44 PM, vino yang <ya...@gmail.com> wrote:
> Hi Sampath,
>
> Flink CLI need to retrieve the JobManager leader address, so it need to
> access the HA specific configuration. Because if based on Zookeeper to
> implement the HA, the leader address information will fetch from Zookeeper.
>
> The main use of config item *high-availability.storageDir* is storage
> (Job graph, checkpoint and so on). Actually, the real data is stored under
> this path which used to recover purpose, zookeeper just store a state
> handle.
>
> ---
> Thanks.
> vino.
>
>
> 2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>
>>
>> ---------- Forwarded message ----------
>> From: Sampath Bhat <sa...@gmail.com>
>> Date: Fri, Jul 13, 2018 at 3:18 PM
>> Subject: Flink CLI properties with HA
>> To: user <us...@flink.apache.org>
>>
>>
>> Hello
>>
>> When HA is enabled in the flink cluster and if I've to submit job via
>> flink CLI then in the flink-conf.yaml of flink CLI should contain this
>> properties -
>> high-availability: zookeeper
>> high-availability.cluster-id: flink
>> high-availability.zookeeper.path.root: flink
>> high-availability.storageDir: <some path>
>> high-availability.zookeeper.quorum: <zookeeper IP:port>
>>
>> What is the need of high-availability.storageDir for flink CLI. Does this
>> mean that even flink client should be able to access the mentioned path or
>> is it some check being done on the property name?
>>
>> Without these properties flink cli will not be able to submit job to
>> flink cluster when HA is enabled.
>>
>>
>
Re: Flink CLI properties with HA
Posted by Sampath Bhat <sa...@gmail.com>.
Hi vino
Should the flink CLI have access to the path mentioned in
*high-availability.storageDir*?
If my flink cluster is on set of machines and i submit my job from flink
CLI from another independent machine by giving necessary details will the
CLI try to access *high-availability.storageDir *path?
I'm aware of the fact that flink client will connect to zookeeper to get
leader address and necessary information for job submission but my
confusion is with *high-availability.storageDir* and its necessity in flink
CLI configuration.
On Mon, Jul 16, 2018 at 2:44 PM, vino yang <ya...@gmail.com> wrote:
> Hi Sampath,
>
> Flink CLI need to retrieve the JobManager leader address, so it need to
> access the HA specific configuration. Because if based on Zookeeper to
> implement the HA, the leader address information will fetch from Zookeeper.
>
> The main use of config item *high-availability.storageDir* is storage
> (Job graph, checkpoint and so on). Actually, the real data is stored under
> this path which used to recover purpose, zookeeper just store a state
> handle.
>
> ---
> Thanks.
> vino.
>
>
> 2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>
>>
>> ---------- Forwarded message ----------
>> From: Sampath Bhat <sa...@gmail.com>
>> Date: Fri, Jul 13, 2018 at 3:18 PM
>> Subject: Flink CLI properties with HA
>> To: user <us...@flink.apache.org>
>>
>>
>> Hello
>>
>> When HA is enabled in the flink cluster and if I've to submit job via
>> flink CLI then in the flink-conf.yaml of flink CLI should contain this
>> properties -
>> high-availability: zookeeper
>> high-availability.cluster-id: flink
>> high-availability.zookeeper.path.root: flink
>> high-availability.storageDir: <some path>
>> high-availability.zookeeper.quorum: <zookeeper IP:port>
>>
>> What is the need of high-availability.storageDir for flink CLI. Does this
>> mean that even flink client should be able to access the mentioned path or
>> is it some check being done on the property name?
>>
>> Without these properties flink cli will not be able to submit job to
>> flink cluster when HA is enabled.
>>
>>
>
Re: Flink CLI properties with HA
Posted by vino yang <ya...@gmail.com>.
Hi Sampath,
Flink CLI need to retrieve the JobManager leader address, so it need to
access the HA specific configuration. Because if based on Zookeeper to
implement the HA, the leader address information will fetch from Zookeeper.
The main use of config item *high-availability.storageDir* is storage (Job
graph, checkpoint and so on). Actually, the real data is stored under this
path which used to recover purpose, zookeeper just store a state handle.
---
Thanks.
vino.
2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>
> ---------- Forwarded message ----------
> From: Sampath Bhat <sa...@gmail.com>
> Date: Fri, Jul 13, 2018 at 3:18 PM
> Subject: Flink CLI properties with HA
> To: user <us...@flink.apache.org>
>
>
> Hello
>
> When HA is enabled in the flink cluster and if I've to submit job via
> flink CLI then in the flink-conf.yaml of flink CLI should contain this
> properties -
> high-availability: zookeeper
> high-availability.cluster-id: flink
> high-availability.zookeeper.path.root: flink
> high-availability.storageDir: <some path>
> high-availability.zookeeper.quorum: <zookeeper IP:port>
>
> What is the need of high-availability.storageDir for flink CLI. Does this
> mean that even flink client should be able to access the mentioned path or
> is it some check being done on the property name?
>
> Without these properties flink cli will not be able to submit job to flink
> cluster when HA is enabled.
>
>
Re: Flink CLI properties with HA
Posted by vino yang <ya...@gmail.com>.
Hi Sampath,
Flink CLI need to retrieve the JobManager leader address, so it need to
access the HA specific configuration. Because if based on Zookeeper to
implement the HA, the leader address information will fetch from Zookeeper.
The main use of config item *high-availability.storageDir* is storage (Job
graph, checkpoint and so on). Actually, the real data is stored under this
path which used to recover purpose, zookeeper just store a state handle.
---
Thanks.
vino.
2018-07-16 15:28 GMT+08:00 Sampath Bhat <sa...@gmail.com>:
>
> ---------- Forwarded message ----------
> From: Sampath Bhat <sa...@gmail.com>
> Date: Fri, Jul 13, 2018 at 3:18 PM
> Subject: Flink CLI properties with HA
> To: user <us...@flink.apache.org>
>
>
> Hello
>
> When HA is enabled in the flink cluster and if I've to submit job via
> flink CLI then in the flink-conf.yaml of flink CLI should contain this
> properties -
> high-availability: zookeeper
> high-availability.cluster-id: flink
> high-availability.zookeeper.path.root: flink
> high-availability.storageDir: <some path>
> high-availability.zookeeper.quorum: <zookeeper IP:port>
>
> What is the need of high-availability.storageDir for flink CLI. Does this
> mean that even flink client should be able to access the mentioned path or
> is it some check being done on the property name?
>
> Without these properties flink cli will not be able to submit job to flink
> cluster when HA is enabled.
>
>
Fwd: Flink CLI properties with HA
Posted by Sampath Bhat <sa...@gmail.com>.
---------- Forwarded message ----------
From: Sampath Bhat <sa...@gmail.com>
Date: Fri, Jul 13, 2018 at 3:18 PM
Subject: Flink CLI properties with HA
To: user <us...@flink.apache.org>
Hello
When HA is enabled in the flink cluster and if I've to submit job via flink
CLI then in the flink-conf.yaml of flink CLI should contain this properties
-
high-availability: zookeeper
high-availability.cluster-id: flink
high-availability.zookeeper.path.root: flink
high-availability.storageDir: <some path>
high-availability.zookeeper.quorum: <zookeeper IP:port>
What is the need of high-availability.storageDir for flink CLI. Does this
mean that even flink client should be able to access the mentioned path or
is it some check being done on the property name?
Without these properties flink cli will not be able to submit job to flink
cluster when HA is enabled.
Fwd: Flink CLI properties with HA
Posted by Sampath Bhat <sa...@gmail.com>.
---------- Forwarded message ----------
From: Sampath Bhat <sa...@gmail.com>
Date: Fri, Jul 13, 2018 at 3:18 PM
Subject: Flink CLI properties with HA
To: user <us...@flink.apache.org>
Hello
When HA is enabled in the flink cluster and if I've to submit job via flink
CLI then in the flink-conf.yaml of flink CLI should contain this properties
-
high-availability: zookeeper
high-availability.cluster-id: flink
high-availability.zookeeper.path.root: flink
high-availability.storageDir: <some path>
high-availability.zookeeper.quorum: <zookeeper IP:port>
What is the need of high-availability.storageDir for flink CLI. Does this
mean that even flink client should be able to access the mentioned path or
is it some check being done on the property name?
Without these properties flink cli will not be able to submit job to flink
cluster when HA is enabled.