You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by amenreet sodhi <am...@gmail.com> on 2023/07/07 08:20:15 UTC

Flink in HA mode causing JM Failure

Hi All,
I am deploying Flink cluster on Kubernetes in HA mode. But i noticed,
whenever i deploy Flink cluster for first time on K8s cluster, it is not
able to populate the cluster configmap, and due to which JM fails with the
following exception:

2023-07-06 16:46:11,428 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] -
Fatal error occurred in the cluster entrypoint.
java.util.concurrent.CompletionException:
java.lang.IllegalStateException: The base directory of the
JobResultStore isn't accessible. No dirty JobResults can be restored.
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
~[?:?]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
[?:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.IllegalStateException: The base directory of the
JobResultStore isn't accessible. No dirty JobResults can be restored.
	at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
~[event_executor-1.1.20.jar:?]
	at org.apache.flink.runtime.highavailability.FileSystemJobResultStore.getDirtyResultsInternal(FileSystemJobResultStore.java:182)
~[event_executor-1.1.20.jar:?]
	at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.withReadLock(AbstractThreadsafeJobResultStore.java:118)
~[event_executor-1.1.20.jar:?]
	at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.getDirtyResults(AbstractThreadsafeJobResultStore.java:100)
~[event_executor-1.1.20.jar:?]
	at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:194)
~[event_executor-1.1.20.jar:?]
	at org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198)
~[event_executor-1.1.20.jar:?]
	at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188)
~[event_executor-1.1.20.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
~[?:?]

Once we reinstall/helm upgrade then this exception goes away. How can this
be resolved, any additional configuration required to resolve this?

I am using the following configuration for HA:

 high-availability.storageDir: file:///opt/flink/pm/ha
    kubernetes.cluster-id: {{ include "fullname" . }}-cluster-{{ now |
date "20060102150405" }}
    high-availability.jobmanager.port: 6123
    high-availability.type: kubernetes
    high-availability:
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    kubernetes.namespace: {{ .Release.Namespace }}

Thanks

Regards
Amenreet Singh Sodhi

Re: Flink in HA mode causing JM Failure

Posted by Shammon FY <zj...@gmail.com>.
Hi amenreet,

According to the error message, I think you can log in the jm pod after it
starts, and check access permissions for the directory
`file:///opt/flink/pm/ha`

Best,
Shammon FY


On Fri, Jul 7, 2023 at 6:04 PM amenreet sodhi <am...@gmail.com> wrote:

> Hi Shammon
>
> I am using an external NFS mount which gets mounted at path
> /opt/flink/pm/, and the path that is mentioned there refers to that
> only, so not a local file. Could there be any other configuration issue?
>
> Thanks
> Regard
> Amenreet Singh Sodhi
>
> On Fri, Jul 7, 2023 at 2:00 PM Shammon FY <zj...@gmail.com> wrote:
>
>> Hi amenreet,
>>
>> Maybe you can try to use hdfs or s3 for `high-availability.storageDir`, I
>> found your current job is using a local file which is started with
>> `file:///`.
>>
>> Best,
>> Shammon FY
>>
>>
>> On Fri, Jul 7, 2023 at 4:20 PM amenreet sodhi <am...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>> I am deploying Flink cluster on Kubernetes in HA mode. But i noticed,
>>> whenever i deploy Flink cluster for first time on K8s cluster, it is not
>>> able to populate the cluster configmap, and due to which JM fails with the
>>> following exception:
>>>
>>> 2023-07-06 16:46:11,428 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Fatal error occurred in the cluster entrypoint.
>>> java.util.concurrent.CompletionException: java.lang.IllegalStateException: The base directory of the JobResultStore isn't accessible. No dirty JobResults can be restored.
>>> 	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]
>>> 	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) [?:?]
>>> 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702) [?:?]
>>> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
>>> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
>>> 	at java.lang.Thread.run(Thread.java:834) [?:?]
>>> Caused by: java.lang.IllegalStateException: The base directory of the JobResultStore isn't accessible. No dirty JobResults can be restored.
>>> 	at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193) ~[event_executor-1.1.20.jar:?]
>>> 	at org.apache.flink.runtime.highavailability.FileSystemJobResultStore.getDirtyResultsInternal(FileSystemJobResultStore.java:182) ~[event_executor-1.1.20.jar:?]
>>> 	at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.withReadLock(AbstractThreadsafeJobResultStore.java:118) ~[event_executor-1.1.20.jar:?]
>>> 	at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.getDirtyResults(AbstractThreadsafeJobResultStore.java:100) ~[event_executor-1.1.20.jar:?]
>>> 	at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:194) ~[event_executor-1.1.20.jar:?]
>>> 	at org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198) ~[event_executor-1.1.20.jar:?]
>>> 	at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188) ~[event_executor-1.1.20.jar:?]
>>> 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
>>>
>>> Once we reinstall/helm upgrade then this exception goes away. How can
>>> this be resolved, any additional configuration required to resolve this?
>>>
>>> I am using the following configuration for HA:
>>>
>>>  high-availability.storageDir: file:///opt/flink/pm/ha
>>>     kubernetes.cluster-id: {{ include "fullname" . }}-cluster-{{ now | date "20060102150405" }}
>>>     high-availability.jobmanager.port: 6123
>>>     high-availability.type: kubernetes
>>>     high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>>>     kubernetes.namespace: {{ .Release.Namespace }}
>>>
>>> Thanks
>>>
>>> Regards
>>> Amenreet Singh Sodhi
>>>
>>>

Re: Flink in HA mode causing JM Failure

Posted by Shammon FY <zj...@gmail.com>.
Hi amenreet,

Maybe you can try to use hdfs or s3 for `high-availability.storageDir`, I
found your current job is using a local file which is started with
`file:///`.

Best,
Shammon FY


On Fri, Jul 7, 2023 at 4:20 PM amenreet sodhi <am...@gmail.com> wrote:

> Hi All,
> I am deploying Flink cluster on Kubernetes in HA mode. But i noticed,
> whenever i deploy Flink cluster for first time on K8s cluster, it is not
> able to populate the cluster configmap, and due to which JM fails with the
> following exception:
>
> 2023-07-06 16:46:11,428 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Fatal error occurred in the cluster entrypoint.
> java.util.concurrent.CompletionException: java.lang.IllegalStateException: The base directory of the JobResultStore isn't accessible. No dirty JobResults can be restored.
> 	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]
> 	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) [?:?]
> 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702) [?:?]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> 	at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: java.lang.IllegalStateException: The base directory of the JobResultStore isn't accessible. No dirty JobResults can be restored.
> 	at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193) ~[event_executor-1.1.20.jar:?]
> 	at org.apache.flink.runtime.highavailability.FileSystemJobResultStore.getDirtyResultsInternal(FileSystemJobResultStore.java:182) ~[event_executor-1.1.20.jar:?]
> 	at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.withReadLock(AbstractThreadsafeJobResultStore.java:118) ~[event_executor-1.1.20.jar:?]
> 	at org.apache.flink.runtime.highavailability.AbstractThreadsafeJobResultStore.getDirtyResults(AbstractThreadsafeJobResultStore.java:100) ~[event_executor-1.1.20.jar:?]
> 	at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResults(SessionDispatcherLeaderProcess.java:194) ~[event_executor-1.1.20.jar:?]
> 	at org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess.supplyUnsynchronizedIfRunning(AbstractDispatcherLeaderProcess.java:198) ~[event_executor-1.1.20.jar:?]
> 	at org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess.getDirtyJobResultsIfRunning(SessionDispatcherLeaderProcess.java:188) ~[event_executor-1.1.20.jar:?]
> 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) ~[?:?]
>
> Once we reinstall/helm upgrade then this exception goes away. How can this
> be resolved, any additional configuration required to resolve this?
>
> I am using the following configuration for HA:
>
>  high-availability.storageDir: file:///opt/flink/pm/ha
>     kubernetes.cluster-id: {{ include "fullname" . }}-cluster-{{ now | date "20060102150405" }}
>     high-availability.jobmanager.port: 6123
>     high-availability.type: kubernetes
>     high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>     kubernetes.namespace: {{ .Release.Namespace }}
>
> Thanks
>
> Regards
> Amenreet Singh Sodhi
>
>