You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by ramkrishna vasudevan <ra...@gmail.com> on 2022/09/26 04:24:22 UTC
Jobmanager fails to come up if the job has an issue
Hi all
I have a simple job where we read for a given path in cloud storage to
watch for new files in a given fodler. While I setup my job there was some
permission issue on the folder. The job is STREAMING job.
The cluster is set in the session mode and is running on Kubernetes.
The job manager since then is failing to come back up and every time it
fails with the permission issue. But the point is how should i recover my
cluster in this case. Since JM is not there the UI is also not working and
how do I remove the bad job from the JM.
Regards
Ram
Re: Jobmanager fails to come up if the job has an issue
Posted by Matthias Pohl via user <us...@flink.apache.org>.
Yes, the JobManager will failover in HA mode and all jobs would be
recovered.
On Mon, Sep 26, 2022 at 2:06 PM ramkrishna vasudevan <
ramvasu.flink@gmail.com> wrote:
> Thanks @Matthias Pohl <ma...@aiven.io> . This is informative. So
> generally in a session cluster if I have more than one job and only one of
> them has this issue, still we will face the same problem?
>
> Regards
> Ram
>
> On Mon, Sep 26, 2022 at 4:32 PM Matthias Pohl <ma...@aiven.io>
> wrote:
>
>> I see. Thanks for sharing the logs. It's related to a FLINK-9097 [1]. In
>> order for the job to not be cleaned up entirely after a failure while
>> submitting the job, the JobManager is failed fatally resulting in a
>> failover. That's what you're experiencing.
>>
>> One solution is to fix the permission issue to make the job recover
>> without problems. If that's not what you want to do, you could delete the
>> entry with the key 'jobGraph-04ae99777ee2ed34c13fe8120e68436e' from the
>> JobGraphStore ConfigMap (based on your logs it should
>> be flink-972ac3d8028e45fcafa9b8b7b7f1dafb-custer-config-map). This will
>> prevent the JobManager from recovering this specific job. Keep in mind that
>> you have to clean up any job-related data by yourself in that case.
>>
>> I hope that helps.
>> Matthias
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-9097
>>
>> On Mon, Sep 26, 2022 at 12:26 PM ramkrishna vasudevan <
>> ramvasu.flink@gmail.com> wrote:
>>
>>> I got some logs and stack traces from our backend storage. This is not
>>> the entire log though. Can this be useful? With these set of logs messages
>>> the job manager kept restarting.
>>>
>>> Regards
>>> Ram
>>>
>>> On Mon, Sep 26, 2022 at 3:11 PM ramkrishna vasudevan <
>>> ramvasu.flink@gmail.com> wrote:
>>>
>>>> Thank you very much for the reply. I have lost the k8s cluster in this
>>>> case before I could capture the logs. I will try to repro this and get back
>>>> to you.
>>>>
>>>> Regards
>>>> Ram
>>>>
>>>> On Mon, Sep 26, 2022 at 12:42 PM Matthias Pohl <ma...@aiven.io>
>>>> wrote:
>>>>
>>>>> Hi Ramkrishna,
>>>>> thanks for reaching out to the Flink community. Could you share the
>>>>> JobManager logs to get a better understanding of what's going on? I'm
>>>>> wondering why the JobManager is failing when the actual problem is that the
>>>>> job is struggling to access a folder. It sounds like there are multiple
>>>>> problems here.
>>>>>
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>> On Mon, Sep 26, 2022 at 6:25 AM ramkrishna vasudevan <
>>>>> ramvasu.flink@gmail.com> wrote:
>>>>>
>>>>>> Hi all
>>>>>>
>>>>>> I have a simple job where we read for a given path in cloud storage
>>>>>> to watch for new files in a given fodler. While I setup my job there was
>>>>>> some permission issue on the folder. The job is STREAMING job.
>>>>>> The cluster is set in the session mode and is running on Kubernetes.
>>>>>> The job manager since then is failing to come back up and every time
>>>>>> it fails with the permission issue. But the point is how should i recover
>>>>>> my cluster in this case. Since JM is not there the UI is also not working
>>>>>> and how do I remove the bad job from the JM.
>>>>>>
>>>>>> Regards
>>>>>> Ram
>>>>>>
>>>>>
Re: Jobmanager fails to come up if the job has an issue
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Thanks @Matthias Pohl <ma...@aiven.io> . This is informative. So
generally in a session cluster if I have more than one job and only one of
them has this issue, still we will face the same problem?
Regards
Ram
On Mon, Sep 26, 2022 at 4:32 PM Matthias Pohl <ma...@aiven.io>
wrote:
> I see. Thanks for sharing the logs. It's related to a FLINK-9097 [1]. In
> order for the job to not be cleaned up entirely after a failure while
> submitting the job, the JobManager is failed fatally resulting in a
> failover. That's what you're experiencing.
>
> One solution is to fix the permission issue to make the job recover
> without problems. If that's not what you want to do, you could delete the
> entry with the key 'jobGraph-04ae99777ee2ed34c13fe8120e68436e' from the
> JobGraphStore ConfigMap (based on your logs it should
> be flink-972ac3d8028e45fcafa9b8b7b7f1dafb-custer-config-map). This will
> prevent the JobManager from recovering this specific job. Keep in mind that
> you have to clean up any job-related data by yourself in that case.
>
> I hope that helps.
> Matthias
>
> [1] https://issues.apache.org/jira/browse/FLINK-9097
>
> On Mon, Sep 26, 2022 at 12:26 PM ramkrishna vasudevan <
> ramvasu.flink@gmail.com> wrote:
>
>> I got some logs and stack traces from our backend storage. This is not
>> the entire log though. Can this be useful? With these set of logs messages
>> the job manager kept restarting.
>>
>> Regards
>> Ram
>>
>> On Mon, Sep 26, 2022 at 3:11 PM ramkrishna vasudevan <
>> ramvasu.flink@gmail.com> wrote:
>>
>>> Thank you very much for the reply. I have lost the k8s cluster in this
>>> case before I could capture the logs. I will try to repro this and get back
>>> to you.
>>>
>>> Regards
>>> Ram
>>>
>>> On Mon, Sep 26, 2022 at 12:42 PM Matthias Pohl <ma...@aiven.io>
>>> wrote:
>>>
>>>> Hi Ramkrishna,
>>>> thanks for reaching out to the Flink community. Could you share the
>>>> JobManager logs to get a better understanding of what's going on? I'm
>>>> wondering why the JobManager is failing when the actual problem is that the
>>>> job is struggling to access a folder. It sounds like there are multiple
>>>> problems here.
>>>>
>>>> Best,
>>>> Matthias
>>>>
>>>> On Mon, Sep 26, 2022 at 6:25 AM ramkrishna vasudevan <
>>>> ramvasu.flink@gmail.com> wrote:
>>>>
>>>>> Hi all
>>>>>
>>>>> I have a simple job where we read for a given path in cloud storage to
>>>>> watch for new files in a given fodler. While I setup my job there was some
>>>>> permission issue on the folder. The job is STREAMING job.
>>>>> The cluster is set in the session mode and is running on Kubernetes.
>>>>> The job manager since then is failing to come back up and every time
>>>>> it fails with the permission issue. But the point is how should i recover
>>>>> my cluster in this case. Since JM is not there the UI is also not working
>>>>> and how do I remove the bad job from the JM.
>>>>>
>>>>> Regards
>>>>> Ram
>>>>>
>>>>
Re: Jobmanager fails to come up if the job has an issue
Posted by Matthias Pohl via user <us...@flink.apache.org>.
I see. Thanks for sharing the logs. It's related to a FLINK-9097 [1]. In
order for the job to not be cleaned up entirely after a failure while
submitting the job, the JobManager is failed fatally resulting in a
failover. That's what you're experiencing.
One solution is to fix the permission issue to make the job recover without
problems. If that's not what you want to do, you could delete the entry
with the key 'jobGraph-04ae99777ee2ed34c13fe8120e68436e' from the
JobGraphStore ConfigMap (based on your logs it should
be flink-972ac3d8028e45fcafa9b8b7b7f1dafb-custer-config-map). This will
prevent the JobManager from recovering this specific job. Keep in mind that
you have to clean up any job-related data by yourself in that case.
I hope that helps.
Matthias
[1] https://issues.apache.org/jira/browse/FLINK-9097
On Mon, Sep 26, 2022 at 12:26 PM ramkrishna vasudevan <
ramvasu.flink@gmail.com> wrote:
> I got some logs and stack traces from our backend storage. This is not the
> entire log though. Can this be useful? With these set of logs messages the
> job manager kept restarting.
>
> Regards
> Ram
>
> On Mon, Sep 26, 2022 at 3:11 PM ramkrishna vasudevan <
> ramvasu.flink@gmail.com> wrote:
>
>> Thank you very much for the reply. I have lost the k8s cluster in this
>> case before I could capture the logs. I will try to repro this and get back
>> to you.
>>
>> Regards
>> Ram
>>
>> On Mon, Sep 26, 2022 at 12:42 PM Matthias Pohl <ma...@aiven.io>
>> wrote:
>>
>>> Hi Ramkrishna,
>>> thanks for reaching out to the Flink community. Could you share the
>>> JobManager logs to get a better understanding of what's going on? I'm
>>> wondering why the JobManager is failing when the actual problem is that the
>>> job is struggling to access a folder. It sounds like there are multiple
>>> problems here.
>>>
>>> Best,
>>> Matthias
>>>
>>> On Mon, Sep 26, 2022 at 6:25 AM ramkrishna vasudevan <
>>> ramvasu.flink@gmail.com> wrote:
>>>
>>>> Hi all
>>>>
>>>> I have a simple job where we read for a given path in cloud storage to
>>>> watch for new files in a given fodler. While I setup my job there was some
>>>> permission issue on the folder. The job is STREAMING job.
>>>> The cluster is set in the session mode and is running on Kubernetes.
>>>> The job manager since then is failing to come back up and every time it
>>>> fails with the permission issue. But the point is how should i recover my
>>>> cluster in this case. Since JM is not there the UI is also not working and
>>>> how do I remove the bad job from the JM.
>>>>
>>>> Regards
>>>> Ram
>>>>
>>>
Re: Jobmanager fails to come up if the job has an issue
Posted by ramkrishna vasudevan <ra...@gmail.com>.
I got some logs and stack traces from our backend storage. This is not the
entire log though. Can this be useful? With these set of logs messages the
job manager kept restarting.
Regards
Ram
On Mon, Sep 26, 2022 at 3:11 PM ramkrishna vasudevan <
ramvasu.flink@gmail.com> wrote:
> Thank you very much for the reply. I have lost the k8s cluster in this
> case before I could capture the logs. I will try to repro this and get back
> to you.
>
> Regards
> Ram
>
> On Mon, Sep 26, 2022 at 12:42 PM Matthias Pohl <ma...@aiven.io>
> wrote:
>
>> Hi Ramkrishna,
>> thanks for reaching out to the Flink community. Could you share the
>> JobManager logs to get a better understanding of what's going on? I'm
>> wondering why the JobManager is failing when the actual problem is that the
>> job is struggling to access a folder. It sounds like there are multiple
>> problems here.
>>
>> Best,
>> Matthias
>>
>> On Mon, Sep 26, 2022 at 6:25 AM ramkrishna vasudevan <
>> ramvasu.flink@gmail.com> wrote:
>>
>>> Hi all
>>>
>>> I have a simple job where we read for a given path in cloud storage to
>>> watch for new files in a given fodler. While I setup my job there was some
>>> permission issue on the folder. The job is STREAMING job.
>>> The cluster is set in the session mode and is running on Kubernetes.
>>> The job manager since then is failing to come back up and every time it
>>> fails with the permission issue. But the point is how should i recover my
>>> cluster in this case. Since JM is not there the UI is also not working and
>>> how do I remove the bad job from the JM.
>>>
>>> Regards
>>> Ram
>>>
>>
Re: Jobmanager fails to come up if the job has an issue
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Thank you very much for the reply. I have lost the k8s cluster in this case
before I could capture the logs. I will try to repro this and get back to
you.
Regards
Ram
On Mon, Sep 26, 2022 at 12:42 PM Matthias Pohl <ma...@aiven.io>
wrote:
> Hi Ramkrishna,
> thanks for reaching out to the Flink community. Could you share the
> JobManager logs to get a better understanding of what's going on? I'm
> wondering why the JobManager is failing when the actual problem is that the
> job is struggling to access a folder. It sounds like there are multiple
> problems here.
>
> Best,
> Matthias
>
> On Mon, Sep 26, 2022 at 6:25 AM ramkrishna vasudevan <
> ramvasu.flink@gmail.com> wrote:
>
>> Hi all
>>
>> I have a simple job where we read for a given path in cloud storage to
>> watch for new files in a given fodler. While I setup my job there was some
>> permission issue on the folder. The job is STREAMING job.
>> The cluster is set in the session mode and is running on Kubernetes.
>> The job manager since then is failing to come back up and every time it
>> fails with the permission issue. But the point is how should i recover my
>> cluster in this case. Since JM is not there the UI is also not working and
>> how do I remove the bad job from the JM.
>>
>> Regards
>> Ram
>>
>
Re: Jobmanager fails to come up if the job has an issue
Posted by Matthias Pohl via user <us...@flink.apache.org>.
Hi Ramkrishna,
thanks for reaching out to the Flink community. Could you share the
JobManager logs to get a better understanding of what's going on? I'm
wondering why the JobManager is failing when the actual problem is that the
job is struggling to access a folder. It sounds like there are multiple
problems here.
Best,
Matthias
On Mon, Sep 26, 2022 at 6:25 AM ramkrishna vasudevan <
ramvasu.flink@gmail.com> wrote:
> Hi all
>
> I have a simple job where we read for a given path in cloud storage to
> watch for new files in a given fodler. While I setup my job there was some
> permission issue on the folder. The job is STREAMING job.
> The cluster is set in the session mode and is running on Kubernetes.
> The job manager since then is failing to come back up and every time it
> fails with the permission issue. But the point is how should i recover my
> cluster in this case. Since JM is not there the UI is also not working and
> how do I remove the bad job from the JM.
>
> Regards
> Ram
>