You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by naga sudhakar <na...@gmail.com> on 2023/04/12 02:57:45 UTC
Flink job manager conditional start of flink jobs
Hi Team,
Greetings!!
Just wanted to know when job manager or task manager is being restarted, is
there a way to run the existing flink jobs based on a condition? Same query
when I am starting flink job fresh also.
Please let me know if any more information is required from my side.
Thanks & Regards
Nagasudhakar Sajja.
Re: Flink job manager conditional start of flink jobs
Posted by Hang Ruan <ru...@gmail.com>.
Hi, nage,
I agree to the Shammon's suggestion after reading the context. Maybe you
need a 'Job Management Service' to manage all jobs among different
namespaces. I think the job management is not suitable for implementation
in the Flink engine.
Best,
Hang
Shammon FY <zj...@gmail.com> 于2023年4月13日周四 11:34写道：
> Hi
>
> The job in ns2 has the permission to stop the job in ns1? How about
> managing the relationship in your `Job Submission Service` if it exists.
> The service can check and stop the job in ns1 before it submitting the job
> to ns2, what do you think?
>
> Best,
> Shammon FY
>
>
> On Thu, Apr 13, 2023 at 10:50 AM naga sudhakar <na...@gmail.com>
> wrote:
>
>> Hi,
>> Thanks for your reply.
>> It is slightly different, would be happy to have any suggestion for the
>> scenario you mentioned.
>> My scenario: I have 2 namespaces say ns1,ns2. I have to make sure only
>> one of ns1 or ns2 should run my flink jobs. Say initially ns1 is running
>> flink jobs, later planned to move them to ns2. Now when I start in ns2, I
>> can make an api call to ns1 jobmanager about running jobs and if no jobs
>> then only I should start in ns2. I can introduce this logic inside the
>> flink job java main method where my total streaming logic present. So if I
>> identify then, how can I stop the initiated job?
>>
>> Thanks,
>> Nagasudhakar.
>>
>> On Thu, 13 Apr, 2023, 7:08 am Shammon FY, <zj...@gmail.com> wrote:
>>
>>> Hi naga
>>>
>>> Could you provide a specific description of your scene? It sounds like
>>> your requirement requires a uniqueness check to ensure that there are no
>>> multiple identical jobs running simultaneously, right?
>>>
>>> Best,
>>> Shammon FY
>>>
>>> On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar <na...@gmail.com>
>>> wrote:
>>>
>>>> Thanks for your email.
>>>> I am looking more in terms of running these flinkk jobs in multi
>>>> names pace environment and make sure only one namespace flink jobs are
>>>> running.
>>>> So on the Job manager when i try to start a flink job, it has to check
>>>> if it's allowed to run in this namespace or not and accordingly flink job
>>>> shud turn into running state otherwise it shud cancel by itself
>>>>
>>>>
>>>>
>>>> On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, <lu...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Is the job you want to start running or already finished?
>>>>> If the job is running, this is simply a failover or a JM failover
>>>>> case.
>>>>> While if the job has finished, there's no such feature that can
>>>>> restart the job
>>>>> automatically, AFAIK. The job has to be submitted again.
>>>>>
>>>>> On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar <na...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Team,
>>>>>> Greetings!!
>>>>>> Just wanted to know when job manager or task manager is being
>>>>>> restarted, is there a way to run the existing flink jobs based on a
>>>>>> condition? Same query when I am starting flink job fresh also.
>>>>>>
>>>>>> Please let me know if any more information is required from my side.
>>>>>>
>>>>>> Thanks & Regards
>>>>>> Nagasudhakar Sajja.
>>>>>>
>>>>>
Re: Flink job manager conditional start of flink jobs
Posted by Shammon FY <zj...@gmail.com>.
Hi
The job in ns2 has the permission to stop the job in ns1? How about
managing the relationship in your `Job Submission Service` if it exists.
The service can check and stop the job in ns1 before it submitting the job
to ns2, what do you think?
Best,
Shammon FY
On Thu, Apr 13, 2023 at 10:50 AM naga sudhakar <na...@gmail.com>
wrote:
> Hi,
> Thanks for your reply.
> It is slightly different, would be happy to have any suggestion for the
> scenario you mentioned.
> My scenario: I have 2 namespaces say ns1,ns2. I have to make sure only one
> of ns1 or ns2 should run my flink jobs. Say initially ns1 is running flink
> jobs, later planned to move them to ns2. Now when I start in ns2, I can
> make an api call to ns1 jobmanager about running jobs and if no jobs then
> only I should start in ns2. I can introduce this logic inside the flink job
> java main method where my total streaming logic present. So if I identify
> then, how can I stop the initiated job?
>
> Thanks,
> Nagasudhakar.
>
> On Thu, 13 Apr, 2023, 7:08 am Shammon FY, <zj...@gmail.com> wrote:
>
>> Hi naga
>>
>> Could you provide a specific description of your scene? It sounds like
>> your requirement requires a uniqueness check to ensure that there are no
>> multiple identical jobs running simultaneously, right?
>>
>> Best,
>> Shammon FY
>>
>> On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar <na...@gmail.com>
>> wrote:
>>
>>> Thanks for your email.
>>> I am looking more in terms of running these flinkk jobs in multi names
>>> pace environment and make sure only one namespace flink jobs are running.
>>> So on the Job manager when i try to start a flink job, it has to check
>>> if it's allowed to run in this namespace or not and accordingly flink job
>>> shud turn into running state otherwise it shud cancel by itself
>>>
>>>
>>>
>>> On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, <lu...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Is the job you want to start running or already finished?
>>>> If the job is running, this is simply a failover or a JM failover case.
>>>> While if the job has finished, there's no such feature that can restart
>>>> the job
>>>> automatically, AFAIK. The job has to be submitted again.
>>>>
>>>> On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar <na...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Team,
>>>>> Greetings!!
>>>>> Just wanted to know when job manager or task manager is being
>>>>> restarted, is there a way to run the existing flink jobs based on a
>>>>> condition? Same query when I am starting flink job fresh also.
>>>>>
>>>>> Please let me know if any more information is required from my side.
>>>>>
>>>>> Thanks & Regards
>>>>> Nagasudhakar Sajja.
>>>>>
>>>>
Re: Flink job manager conditional start of flink jobs
Posted by naga sudhakar <na...@gmail.com>.
Hi,
Thanks for your reply.
It is slightly different, would be happy to have any suggestion for the
scenario you mentioned.
My scenario: I have 2 namespaces say ns1,ns2. I have to make sure only one
of ns1 or ns2 should run my flink jobs. Say initially ns1 is running flink
jobs, later planned to move them to ns2. Now when I start in ns2, I can
make an api call to ns1 jobmanager about running jobs and if no jobs then
only I should start in ns2. I can introduce this logic inside the flink job
java main method where my total streaming logic present. So if I identify
then, how can I stop the initiated job?
Thanks,
Nagasudhakar.
On Thu, 13 Apr, 2023, 7:08 am Shammon FY, <zj...@gmail.com> wrote:
> Hi naga
>
> Could you provide a specific description of your scene? It sounds like
> your requirement requires a uniqueness check to ensure that there are no
> multiple identical jobs running simultaneously, right?
>
> Best,
> Shammon FY
>
> On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar <na...@gmail.com>
> wrote:
>
>> Thanks for your email.
>> I am looking more in terms of running these flinkk jobs in multi names
>> pace environment and make sure only one namespace flink jobs are running.
>> So on the Job manager when i try to start a flink job, it has to check if
>> it's allowed to run in this namespace or not and accordingly flink job
>> shud turn into running state otherwise it shud cancel by itself
>>
>>
>>
>> On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, <lu...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Is the job you want to start running or already finished?
>>> If the job is running, this is simply a failover or a JM failover case.
>>> While if the job has finished, there's no such feature that can restart
>>> the job
>>> automatically, AFAIK. The job has to be submitted again.
>>>
>>> On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar <na...@gmail.com>
>>> wrote:
>>>
>>>> Hi Team,
>>>> Greetings!!
>>>> Just wanted to know when job manager or task manager is being
>>>> restarted, is there a way to run the existing flink jobs based on a
>>>> condition? Same query when I am starting flink job fresh also.
>>>>
>>>> Please let me know if any more information is required from my side.
>>>>
>>>> Thanks & Regards
>>>> Nagasudhakar Sajja.
>>>>
>>>
Re: Flink job manager conditional start of flink jobs
Posted by Shammon FY <zj...@gmail.com>.
Hi naga
Could you provide a specific description of your scene? It sounds like your
requirement requires a uniqueness check to ensure that there are no
multiple identical jobs running simultaneously, right?
Best,
Shammon FY
On Wed, Apr 12, 2023 at 4:08 PM naga sudhakar <na...@gmail.com>
wrote:
> Thanks for your email.
> I am looking more in terms of running these flinkk jobs in multi names
> pace environment and make sure only one namespace flink jobs are running.
> So on the Job manager when i try to start a flink job, it has to check if
> it's allowed to run in this namespace or not and accordingly flink job
> shud turn into running state otherwise it shud cancel by itself
>
>
>
> On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, <lu...@gmail.com> wrote:
>
>> Hi,
>>
>> Is the job you want to start running or already finished?
>> If the job is running, this is simply a failover or a JM failover case.
>> While if the job has finished, there's no such feature that can restart
>> the job
>> automatically, AFAIK. The job has to be submitted again.
>>
>> On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar <na...@gmail.com>
>> wrote:
>>
>>> Hi Team,
>>> Greetings!!
>>> Just wanted to know when job manager or task manager is being restarted,
>>> is there a way to run the existing flink jobs based on a condition? Same
>>> query when I am starting flink job fresh also.
>>>
>>> Please let me know if any more information is required from my side.
>>>
>>> Thanks & Regards
>>> Nagasudhakar Sajja.
>>>
>>
Re: Flink job manager conditional start of flink jobs
Posted by naga sudhakar <na...@gmail.com>.
Thanks for your email.
I am looking more in terms of running these flinkk jobs in multi names
pace environment and make sure only one namespace flink jobs are running.
So on the Job manager when i try to start a flink job, it has to check if
it's allowed to run in this namespace or not and accordingly flink job
shud turn into running state otherwise it shud cancel by itself
On Wed, 12 Apr, 2023, 12:06 pm Gen Luo, <lu...@gmail.com> wrote:
> Hi,
>
> Is the job you want to start running or already finished?
> If the job is running, this is simply a failover or a JM failover case.
> While if the job has finished, there's no such feature that can restart
> the job
> automatically, AFAIK. The job has to be submitted again.
>
> On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar <na...@gmail.com>
> wrote:
>
>> Hi Team,
>> Greetings!!
>> Just wanted to know when job manager or task manager is being restarted,
>> is there a way to run the existing flink jobs based on a condition? Same
>> query when I am starting flink job fresh also.
>>
>> Please let me know if any more information is required from my side.
>>
>> Thanks & Regards
>> Nagasudhakar Sajja.
>>
>
Re: Flink job manager conditional start of flink jobs
Posted by Gen Luo <lu...@gmail.com>.
Hi,
Is the job you want to start running or already finished?
If the job is running, this is simply a failover or a JM failover case.
While if the job has finished, there's no such feature that can restart the
job
automatically, AFAIK. The job has to be submitted again.
On Wed, Apr 12, 2023 at 10:58 AM naga sudhakar <na...@gmail.com>
wrote:
> Hi Team,
> Greetings!!
> Just wanted to know when job manager or task manager is being restarted,
> is there a way to run the existing flink jobs based on a condition? Same
> query when I am starting flink job fresh also.
>
> Please let me know if any more information is required from my side.
>
> Thanks & Regards
> Nagasudhakar Sajja.
>