You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Shubham Bansal <s....@gmail.com> on 2023/05/02 12:01:01 UTC
Elastic scaling with HA (zookeeper)
Hello everyone,
Elastic scaling, as of now, has to be started in application mode. I am
having some trouble in making it work with HA.
I have a jobmanager pod and a single taskmanager pod, I use a post install
helm hook to start the jobmanager along with the single job (As it's
required in application mode).
> ./bin/standalone-job.sh start (..removed for brevity)
>
> If due to some reason my job manager goes down, I have two options
1. I start the same job from the required checkpoint, my HA (zookeeper)
also kicks in and submits another job to the jobmanager. So there are 2
jobs running where it should have been 1.
2. If I don't submit the job, I don't have any other way to start the
jobmanager without the job with elastic scaling enabled.
Is there any way to start the jobmanager without the job with elastic
scaling?
Best,
Shubham
Re: Elastic scaling with HA (zookeeper)
Posted by Weihua Hu <hu...@gmail.com>.
Hi,
If I provide the same jobId, will it restore from the HA first, or just
> start anew?
it will restore from HA first, and will not start a new one.
I tried with flink v1.15.2 with the flowing example, but can't reproduce.
Could you provide the full start command?
1. ./bin/standalone-job.sh start --job-classname
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
2. ./bin/taskmanager.sh start
3. kill -9 ${JobManager PID}
4. ./bin/standalone-job.sh start --job-classname
org.apache.flink.streaming.examples.windowing.TopSpeedWindowing
There is only one job running in my cluster after the new jobmanager
started.
Best,
Weihua
On Sat, May 6, 2023 at 4:00 PM Shubham Bansal <s....@gmail.com> wrote:
> Thanks for the response Weihua.
>
> I am using flink v.15.2. I am unable to understand how I should be
> starting the jobmanager without the job so that it can poll HA store and
> start the job.
> Since the current method of running the script standalone-job.sh requires
> a job at start. If I provide the same jobId, will it restore from the HA
> first, or just start anew?
>
> Best,
> Shubham Bansal
>
>
> On Sat, May 6, 2023 at 9:34 AM Weihua Hu <hu...@gmail.com> wrote:
>
>> Hi, Shubham
>>
>> Which Flink version are you using?
>> AFAIK, the JobManager will recover the job from the HA store first, and
>> it won't submit the same job (identify be jobID) again if it has already
>> been recovered.
>>
>> Best,
>> Weihua
>>
>>
>> On Tue, May 2, 2023 at 8:02 PM Shubham Bansal <s....@gmail.com>
>> wrote:
>>
>>> Hello everyone,
>>>
>>> Elastic scaling, as of now, has to be started in application mode. I am
>>> having some trouble in making it work with HA.
>>> I have a jobmanager pod and a single taskmanager pod, I use a post
>>> install helm hook to start the jobmanager along with the single job (As
>>> it's required in application mode).
>>>
>>>> ./bin/standalone-job.sh start (..removed for brevity)
>>>>
>>>> If due to some reason my job manager goes down, I have two options
>>>
>>> 1. I start the same job from the required checkpoint, my HA
>>> (zookeeper) also kicks in and submits another job to the jobmanager. So
>>> there are 2 jobs running where it should have been 1.
>>> 2. If I don't submit the job, I don't have any other way to start
>>> the jobmanager without the job with elastic scaling enabled.
>>>
>>> Is there any way to start the jobmanager without the job with elastic
>>> scaling?
>>>
>>> Best,
>>> Shubham
>>>
>>
Re: Elastic scaling with HA (zookeeper)
Posted by Shubham Bansal <s....@gmail.com>.
Thanks for the response Weihua.
I am using flink v.15.2. I am unable to understand how I should be starting
the jobmanager without the job so that it can poll HA store and start the
job.
Since the current method of running the script standalone-job.sh requires a
job at start. If I provide the same jobId, will it restore from the HA
first, or just start anew?
Best,
Shubham Bansal
On Sat, May 6, 2023 at 9:34 AM Weihua Hu <hu...@gmail.com> wrote:
> Hi, Shubham
>
> Which Flink version are you using?
> AFAIK, the JobManager will recover the job from the HA store first, and it
> won't submit the same job (identify be jobID) again if it has already been
> recovered.
>
> Best,
> Weihua
>
>
> On Tue, May 2, 2023 at 8:02 PM Shubham Bansal <s....@gmail.com>
> wrote:
>
>> Hello everyone,
>>
>> Elastic scaling, as of now, has to be started in application mode. I am
>> having some trouble in making it work with HA.
>> I have a jobmanager pod and a single taskmanager pod, I use a post
>> install helm hook to start the jobmanager along with the single job (As
>> it's required in application mode).
>>
>>> ./bin/standalone-job.sh start (..removed for brevity)
>>>
>>> If due to some reason my job manager goes down, I have two options
>>
>> 1. I start the same job from the required checkpoint, my HA
>> (zookeeper) also kicks in and submits another job to the jobmanager. So
>> there are 2 jobs running where it should have been 1.
>> 2. If I don't submit the job, I don't have any other way to start the
>> jobmanager without the job with elastic scaling enabled.
>>
>> Is there any way to start the jobmanager without the job with elastic
>> scaling?
>>
>> Best,
>> Shubham
>>
>
Re: Elastic scaling with HA (zookeeper)
Posted by Weihua Hu <hu...@gmail.com>.
Hi, Shubham
Which Flink version are you using?
AFAIK, the JobManager will recover the job from the HA store first, and it
won't submit the same job (identify be jobID) again if it has already been
recovered.
Best,
Weihua
On Tue, May 2, 2023 at 8:02 PM Shubham Bansal <s....@gmail.com> wrote:
> Hello everyone,
>
> Elastic scaling, as of now, has to be started in application mode. I am
> having some trouble in making it work with HA.
> I have a jobmanager pod and a single taskmanager pod, I use a post install
> helm hook to start the jobmanager along with the single job (As it's
> required in application mode).
>
>> ./bin/standalone-job.sh start (..removed for brevity)
>>
>> If due to some reason my job manager goes down, I have two options
>
> 1. I start the same job from the required checkpoint, my HA
> (zookeeper) also kicks in and submits another job to the jobmanager. So
> there are 2 jobs running where it should have been 1.
> 2. If I don't submit the job, I don't have any other way to start the
> jobmanager without the job with elastic scaling enabled.
>
> Is there any way to start the jobmanager without the job with elastic
> scaling?
>
> Best,
> Shubham
>