You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by jonas eyob <jo...@gmail.com> on 2021/08/25 06:47:27 UTC

NullPointerException when using KubernetesHaServicesFactory

Hey, I've been struggling with this problem now for some days - driving me
crazy.

I have a standalone kubernetes Flink (1.12.5) using an application cluster
mode approach.

*The problem*
I am getting a NullPointerException when specifying the FQN of the
Kubernetes HA Service Factory class
i.e.
*org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*

What other configurations besides the ones specified (here
<https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
may I be missing?

Details:
* we are using a custom image using the flink: 1.12
<https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
as base image

flink-conf.yaml -- thought this may be useful?
flink-conf.yaml: |+
jobmanager.rpc.address: {{ $fullName }}-jobmanager
jobmanager.rpc.port: 6123
jobmanager.memory.process.size: 1600m
taskmanager.numberOfTaskSlots: 2
taskmanager.rpc.port: 6122
taskmanager.memory.process.size: 1728m
blob.server.port: 6124
queryable-state.proxy.ports: 6125
parallelism.default: 2
scheduler-mode: reactive
execution.checkpointing.interval: 10s
restart-strategy: fixed-delay
restart-strategy.fixed-delay.attempts: 10
high-availability:
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.cluster-id: /{{ $fullName }}
high-availability.storageDir: s3://redacted-flink-dev/recovery

*Snippet of Job Manager pod log*
2021-08-25 06:14:20,652 INFO
 org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
StandaloneApplicationClusterEntryPoint down with application status FAILED.
Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
services from the instantiated HighAvailabilityServicesFactory
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
at
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
at
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
at
org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
Caused by: java.lang.NullPointerException
at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
at
org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
at
org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
at
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
at
org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
... 9 more
.

--
Many thanks,
Jonas

Re: NullPointerException when using KubernetesHaServicesFactory

Posted by jonas eyob <jo...@gmail.com>.
So when spinning it up on minikube, and then ssh into one of the JobManager
pods shows following for the commands you mentioned:

flink@local-thoros-jobmanager-6l9lz:~$ id
uid=9999(flink) gid=9999(flink) groups=9999(flink)

flink@local-thoros-jobmanager-6l9lz:~$  ls -la
$FLINK_HOME/plugins/s3-fs-presto/
ls: cannot access '/opt/flink/plugins/s3-fs-presto/': No such file or
directory

It appears, this had to do with minikube using cached version of the docker
image, so the new changes (i.e. adding the plugin) never was reflected.
After pulling down the latest the error stopped :)

Best,
Jonas



Den ons 25 aug. 2021 kl 21:17 skrev Thms Hmm <th...@gmail.com>:

> Can you check what is the output of those commands
>
> $ id
> $ ls -la $FLINK_HOME/plugins/s3-fs-presto/
>
>
> jonas eyob <jo...@gmail.com> schrieb am Mi. 25. Aug. 2021 um 16:17:
>
>> The exception is showing up both in TM and JM
>>
>> This however seemed only to appear when running on my local kubernetes
>> setup.
>> > I'd also recommend setting "kubernetes.namespace" option, unless you're
>> using "default" namespace.
>>
>> Yes, good point - I now see why that was needed.
>>
>>
>> Den ons 25 aug. 2021 kl 11:37 skrev David Morávek <dm...@apache.org>:
>>
>>> Hi Jonas,
>>>
>>> Where does the exception pop-up? In job driver, TM, JM? You need to make
>>> sure that the plugin folder is setup for all of them, because they all may
>>> need to access s3 at some point.
>>>
>>> Best,
>>> D.
>>>
>>> On Wed, Aug 25, 2021 at 11:54 AM jonas eyob <jo...@gmail.com>
>>> wrote:
>>>
>>>> Hey Thms,
>>>>
>>>> tried the s3p:// option as well - same issue.
>>>>
>>>> > Also check if your user that executes the process is able to read the
>>>> jars.
>>>> Not exactly sure how to do that? The user "flink" in the docker image
>>>> is able to read the contents as far I understand. But maybe that's not how
>>>> I would check it?
>>>>
>>>> Den ons 25 aug. 2021 kl 10:12 skrev Thms Hmm <th...@gmail.com>:
>>>>
>>>>> Hey Jonas,
>>>>> you could also try to use the ´s3p://´ scheme to directly specify that
>>>>> presto should be used. Also check if your user that executes the process is
>>>>> able to read the jars.
>>>>>
>>>>> Am Mi., 25. Aug. 2021 um 10:01 Uhr schrieb jonas eyob <
>>>>> jonas.eyob@gmail.com>:
>>>>>
>>>>>> Thanks David for the quick response!
>>>>>>
>>>>>> *face palm* - Thanks a lot, that seems to have addressed the
>>>>>> NullPointerException issue.
>>>>>> May I also suggest that this [1] page be updated, since it says the
>>>>>> key is "high-availability.cluster-id"
>>>>>>
>>>>>> This led me to another issue however: "org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
>>>>>> Could not find a file system implementation for scheme 's3'"
>>>>>>
>>>>>> The section [2] describe how I can either use environment variables
>>>>>> e.g. ENABLE_BUILT_IN_PLUGINS or bake that in to the image by copying the
>>>>>> provided plugins in opt/ under /plugins
>>>>>>
>>>>>> Dockerfile (snippet)
>>>>>> # Configure flink provided plugin for S3 access
>>>>>> RUN mkdir -p $FLINK_HOME/plugins/s3-fs-presto
>>>>>> RUN cp $FLINK_HOME/opt/flink-s3-fs-presto-*.jar
>>>>>> $FLINK_HOME/plugins/s3-fs-presto/
>>>>>>
>>>>>> When bashing into the image:
>>>>>>
>>>>>> flink@dd86717a92a0:~/plugins/s3-fs-presto$ ls
>>>>>> flink-s3-fs-presto-1.12.5.jar
>>>>>>
>>>>>> Any idea?
>>>>>>
>>>>>> [1]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#high-availability
>>>>>> [2]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins
>>>>>>
>>>>>>
>>>>>> Den ons 25 aug. 2021 kl 08:00 skrev David Morávek <dm...@apache.org>:
>>>>>>
>>>>>>> Hi Jonas,
>>>>>>>
>>>>>>> this exception is raised because "kubernetes.cluster-id" [1] is not
>>>>>>> set. I'd also recommend setting "kubernetes.namespace" option, unless
>>>>>>> you're using "default" namespace.
>>>>>>>
>>>>>>> I've filled FLINK-23961 [2] so we provide more descriptive warning
>>>>>>> for this issue next time ;)
>>>>>>>
>>>>>>> [1]
>>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/#example-configuration
>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-23961
>>>>>>>
>>>>>>> Best,
>>>>>>> D.
>>>>>>>
>>>>>>> On Wed, Aug 25, 2021 at 8:48 AM jonas eyob <jo...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey, I've been struggling with this problem now for some days -
>>>>>>>> driving me crazy.
>>>>>>>>
>>>>>>>> I have a standalone kubernetes Flink (1.12.5) using an application
>>>>>>>> cluster mode approach.
>>>>>>>>
>>>>>>>> *The problem*
>>>>>>>> I am getting a NullPointerException when specifying the FQN of the
>>>>>>>> Kubernetes HA Service Factory class
>>>>>>>> i.e.
>>>>>>>> *org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*
>>>>>>>>
>>>>>>>> What other configurations besides the ones specified (here
>>>>>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
>>>>>>>> may I be missing?
>>>>>>>>
>>>>>>>> Details:
>>>>>>>> * we are using a custom image using the flink: 1.12
>>>>>>>> <https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
>>>>>>>> as base image
>>>>>>>>
>>>>>>>> flink-conf.yaml -- thought this may be useful?
>>>>>>>> flink-conf.yaml: |+
>>>>>>>> jobmanager.rpc.address: {{ $fullName }}-jobmanager
>>>>>>>> jobmanager.rpc.port: 6123
>>>>>>>> jobmanager.memory.process.size: 1600m
>>>>>>>> taskmanager.numberOfTaskSlots: 2
>>>>>>>> taskmanager.rpc.port: 6122
>>>>>>>> taskmanager.memory.process.size: 1728m
>>>>>>>> blob.server.port: 6124
>>>>>>>> queryable-state.proxy.ports: 6125
>>>>>>>> parallelism.default: 2
>>>>>>>> scheduler-mode: reactive
>>>>>>>> execution.checkpointing.interval: 10s
>>>>>>>> restart-strategy: fixed-delay
>>>>>>>> restart-strategy.fixed-delay.attempts: 10
>>>>>>>> high-availability:
>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>>>>>>>> high-availability.cluster-id: /{{ $fullName }}
>>>>>>>> high-availability.storageDir: s3://redacted-flink-dev/recovery
>>>>>>>>
>>>>>>>> *Snippet of Job Manager pod log*
>>>>>>>> 2021-08-25 06:14:20,652 INFO
>>>>>>>>  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
>>>>>>>> StandaloneApplicationClusterEntryPoint down with application status FAILED.
>>>>>>>> Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
>>>>>>>> services from the instantiated HighAvailabilityServicesFactory
>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
>>>>>>>> at
>>>>>>>> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
>>>>>>>> Caused by: java.lang.NullPointerException
>>>>>>>> at
>>>>>>>> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
>>>>>>>> at
>>>>>>>> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
>>>>>>>> at
>>>>>>>> org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
>>>>>>>> at
>>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
>>>>>>>> at
>>>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
>>>>>>>> ... 9 more
>>>>>>>> .
>>>>>>>>
>>>>>>>> --
>>>>>>>> Many thanks,
>>>>>>>> Jonas
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Med Vänliga Hälsningar*
>>>>>> *Jonas Eyob*
>>>>>>
>>>>>
>>>>
>>>> --
>>>> *Med Vänliga Hälsningar*
>>>> *Jonas Eyob*
>>>>
>>>
>>
>> --
>> *Med Vänliga Hälsningar*
>> *Jonas Eyob*
>>
>

-- 
*Med Vänliga Hälsningar*
*Jonas Eyob*

Re: NullPointerException when using KubernetesHaServicesFactory

Posted by Thms Hmm <th...@gmail.com>.
Can you check what is the output of those commands

$ id
$ ls -la $FLINK_HOME/plugins/s3-fs-presto/


jonas eyob <jo...@gmail.com> schrieb am Mi. 25. Aug. 2021 um 16:17:

> The exception is showing up both in TM and JM
>
> This however seemed only to appear when running on my local kubernetes
> setup.
> > I'd also recommend setting "kubernetes.namespace" option, unless you're
> using "default" namespace.
>
> Yes, good point - I now see why that was needed.
>
>
> Den ons 25 aug. 2021 kl 11:37 skrev David Morávek <dm...@apache.org>:
>
>> Hi Jonas,
>>
>> Where does the exception pop-up? In job driver, TM, JM? You need to make
>> sure that the plugin folder is setup for all of them, because they all may
>> need to access s3 at some point.
>>
>> Best,
>> D.
>>
>> On Wed, Aug 25, 2021 at 11:54 AM jonas eyob <jo...@gmail.com> wrote:
>>
>>> Hey Thms,
>>>
>>> tried the s3p:// option as well - same issue.
>>>
>>> > Also check if your user that executes the process is able to read the
>>> jars.
>>> Not exactly sure how to do that? The user "flink" in the docker image is
>>> able to read the contents as far I understand. But maybe that's not how I
>>> would check it?
>>>
>>> Den ons 25 aug. 2021 kl 10:12 skrev Thms Hmm <th...@gmail.com>:
>>>
>>>> Hey Jonas,
>>>> you could also try to use the ´s3p://´ scheme to directly specify that
>>>> presto should be used. Also check if your user that executes the process is
>>>> able to read the jars.
>>>>
>>>> Am Mi., 25. Aug. 2021 um 10:01 Uhr schrieb jonas eyob <
>>>> jonas.eyob@gmail.com>:
>>>>
>>>>> Thanks David for the quick response!
>>>>>
>>>>> *face palm* - Thanks a lot, that seems to have addressed the
>>>>> NullPointerException issue.
>>>>> May I also suggest that this [1] page be updated, since it says the
>>>>> key is "high-availability.cluster-id"
>>>>>
>>>>> This led me to another issue however: "org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
>>>>> Could not find a file system implementation for scheme 's3'"
>>>>>
>>>>> The section [2] describe how I can either use environment variables
>>>>> e.g. ENABLE_BUILT_IN_PLUGINS or bake that in to the image by copying the
>>>>> provided plugins in opt/ under /plugins
>>>>>
>>>>> Dockerfile (snippet)
>>>>> # Configure flink provided plugin for S3 access
>>>>> RUN mkdir -p $FLINK_HOME/plugins/s3-fs-presto
>>>>> RUN cp $FLINK_HOME/opt/flink-s3-fs-presto-*.jar
>>>>> $FLINK_HOME/plugins/s3-fs-presto/
>>>>>
>>>>> When bashing into the image:
>>>>>
>>>>> flink@dd86717a92a0:~/plugins/s3-fs-presto$ ls
>>>>> flink-s3-fs-presto-1.12.5.jar
>>>>>
>>>>> Any idea?
>>>>>
>>>>> [1]
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#high-availability
>>>>> [2]
>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins
>>>>>
>>>>>
>>>>> Den ons 25 aug. 2021 kl 08:00 skrev David Morávek <dm...@apache.org>:
>>>>>
>>>>>> Hi Jonas,
>>>>>>
>>>>>> this exception is raised because "kubernetes.cluster-id" [1] is not
>>>>>> set. I'd also recommend setting "kubernetes.namespace" option, unless
>>>>>> you're using "default" namespace.
>>>>>>
>>>>>> I've filled FLINK-23961 [2] so we provide more descriptive warning
>>>>>> for this issue next time ;)
>>>>>>
>>>>>> [1]
>>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/#example-configuration
>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-23961
>>>>>>
>>>>>> Best,
>>>>>> D.
>>>>>>
>>>>>> On Wed, Aug 25, 2021 at 8:48 AM jonas eyob <jo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey, I've been struggling with this problem now for some days -
>>>>>>> driving me crazy.
>>>>>>>
>>>>>>> I have a standalone kubernetes Flink (1.12.5) using an application
>>>>>>> cluster mode approach.
>>>>>>>
>>>>>>> *The problem*
>>>>>>> I am getting a NullPointerException when specifying the FQN of the
>>>>>>> Kubernetes HA Service Factory class
>>>>>>> i.e.
>>>>>>> *org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*
>>>>>>>
>>>>>>> What other configurations besides the ones specified (here
>>>>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
>>>>>>> may I be missing?
>>>>>>>
>>>>>>> Details:
>>>>>>> * we are using a custom image using the flink: 1.12
>>>>>>> <https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
>>>>>>> as base image
>>>>>>>
>>>>>>> flink-conf.yaml -- thought this may be useful?
>>>>>>> flink-conf.yaml: |+
>>>>>>> jobmanager.rpc.address: {{ $fullName }}-jobmanager
>>>>>>> jobmanager.rpc.port: 6123
>>>>>>> jobmanager.memory.process.size: 1600m
>>>>>>> taskmanager.numberOfTaskSlots: 2
>>>>>>> taskmanager.rpc.port: 6122
>>>>>>> taskmanager.memory.process.size: 1728m
>>>>>>> blob.server.port: 6124
>>>>>>> queryable-state.proxy.ports: 6125
>>>>>>> parallelism.default: 2
>>>>>>> scheduler-mode: reactive
>>>>>>> execution.checkpointing.interval: 10s
>>>>>>> restart-strategy: fixed-delay
>>>>>>> restart-strategy.fixed-delay.attempts: 10
>>>>>>> high-availability:
>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>>>>>>> high-availability.cluster-id: /{{ $fullName }}
>>>>>>> high-availability.storageDir: s3://redacted-flink-dev/recovery
>>>>>>>
>>>>>>> *Snippet of Job Manager pod log*
>>>>>>> 2021-08-25 06:14:20,652 INFO
>>>>>>>  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
>>>>>>> StandaloneApplicationClusterEntryPoint down with application status FAILED.
>>>>>>> Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
>>>>>>> services from the instantiated HighAvailabilityServicesFactory
>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
>>>>>>> at
>>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
>>>>>>> at
>>>>>>> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
>>>>>>> Caused by: java.lang.NullPointerException
>>>>>>> at
>>>>>>> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
>>>>>>> at
>>>>>>> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
>>>>>>> at
>>>>>>> org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
>>>>>>> at
>>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
>>>>>>> at
>>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
>>>>>>> ... 9 more
>>>>>>> .
>>>>>>>
>>>>>>> --
>>>>>>> Many thanks,
>>>>>>> Jonas
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> *Med Vänliga Hälsningar*
>>>>> *Jonas Eyob*
>>>>>
>>>>
>>>
>>> --
>>> *Med Vänliga Hälsningar*
>>> *Jonas Eyob*
>>>
>>
>
> --
> *Med Vänliga Hälsningar*
> *Jonas Eyob*
>

Re: NullPointerException when using KubernetesHaServicesFactory

Posted by jonas eyob <jo...@gmail.com>.
The exception is showing up both in TM and JM

This however seemed only to appear when running on my local kubernetes
setup.
> I'd also recommend setting "kubernetes.namespace" option, unless you're
using "default" namespace.

Yes, good point - I now see why that was needed.


Den ons 25 aug. 2021 kl 11:37 skrev David Morávek <dm...@apache.org>:

> Hi Jonas,
>
> Where does the exception pop-up? In job driver, TM, JM? You need to make
> sure that the plugin folder is setup for all of them, because they all may
> need to access s3 at some point.
>
> Best,
> D.
>
> On Wed, Aug 25, 2021 at 11:54 AM jonas eyob <jo...@gmail.com> wrote:
>
>> Hey Thms,
>>
>> tried the s3p:// option as well - same issue.
>>
>> > Also check if your user that executes the process is able to read the
>> jars.
>> Not exactly sure how to do that? The user "flink" in the docker image is
>> able to read the contents as far I understand. But maybe that's not how I
>> would check it?
>>
>> Den ons 25 aug. 2021 kl 10:12 skrev Thms Hmm <th...@gmail.com>:
>>
>>> Hey Jonas,
>>> you could also try to use the ´s3p://´ scheme to directly specify that
>>> presto should be used. Also check if your user that executes the process is
>>> able to read the jars.
>>>
>>> Am Mi., 25. Aug. 2021 um 10:01 Uhr schrieb jonas eyob <
>>> jonas.eyob@gmail.com>:
>>>
>>>> Thanks David for the quick response!
>>>>
>>>> *face palm* - Thanks a lot, that seems to have addressed the
>>>> NullPointerException issue.
>>>> May I also suggest that this [1] page be updated, since it says the key
>>>> is "high-availability.cluster-id"
>>>>
>>>> This led me to another issue however: "org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
>>>> Could not find a file system implementation for scheme 's3'"
>>>>
>>>> The section [2] describe how I can either use environment variables
>>>> e.g. ENABLE_BUILT_IN_PLUGINS or bake that in to the image by copying the
>>>> provided plugins in opt/ under /plugins
>>>>
>>>> Dockerfile (snippet)
>>>> # Configure flink provided plugin for S3 access
>>>> RUN mkdir -p $FLINK_HOME/plugins/s3-fs-presto
>>>> RUN cp $FLINK_HOME/opt/flink-s3-fs-presto-*.jar
>>>> $FLINK_HOME/plugins/s3-fs-presto/
>>>>
>>>> When bashing into the image:
>>>>
>>>> flink@dd86717a92a0:~/plugins/s3-fs-presto$ ls
>>>> flink-s3-fs-presto-1.12.5.jar
>>>>
>>>> Any idea?
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#high-availability
>>>> [2]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins
>>>>
>>>>
>>>> Den ons 25 aug. 2021 kl 08:00 skrev David Morávek <dm...@apache.org>:
>>>>
>>>>> Hi Jonas,
>>>>>
>>>>> this exception is raised because "kubernetes.cluster-id" [1] is not
>>>>> set. I'd also recommend setting "kubernetes.namespace" option, unless
>>>>> you're using "default" namespace.
>>>>>
>>>>> I've filled FLINK-23961 [2] so we provide more descriptive warning for
>>>>> this issue next time ;)
>>>>>
>>>>> [1]
>>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/#example-configuration
>>>>> [2] https://issues.apache.org/jira/browse/FLINK-23961
>>>>>
>>>>> Best,
>>>>> D.
>>>>>
>>>>> On Wed, Aug 25, 2021 at 8:48 AM jonas eyob <jo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hey, I've been struggling with this problem now for some days -
>>>>>> driving me crazy.
>>>>>>
>>>>>> I have a standalone kubernetes Flink (1.12.5) using an application
>>>>>> cluster mode approach.
>>>>>>
>>>>>> *The problem*
>>>>>> I am getting a NullPointerException when specifying the FQN of the
>>>>>> Kubernetes HA Service Factory class
>>>>>> i.e.
>>>>>> *org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*
>>>>>>
>>>>>> What other configurations besides the ones specified (here
>>>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
>>>>>> may I be missing?
>>>>>>
>>>>>> Details:
>>>>>> * we are using a custom image using the flink: 1.12
>>>>>> <https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
>>>>>> as base image
>>>>>>
>>>>>> flink-conf.yaml -- thought this may be useful?
>>>>>> flink-conf.yaml: |+
>>>>>> jobmanager.rpc.address: {{ $fullName }}-jobmanager
>>>>>> jobmanager.rpc.port: 6123
>>>>>> jobmanager.memory.process.size: 1600m
>>>>>> taskmanager.numberOfTaskSlots: 2
>>>>>> taskmanager.rpc.port: 6122
>>>>>> taskmanager.memory.process.size: 1728m
>>>>>> blob.server.port: 6124
>>>>>> queryable-state.proxy.ports: 6125
>>>>>> parallelism.default: 2
>>>>>> scheduler-mode: reactive
>>>>>> execution.checkpointing.interval: 10s
>>>>>> restart-strategy: fixed-delay
>>>>>> restart-strategy.fixed-delay.attempts: 10
>>>>>> high-availability:
>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>>>>>> high-availability.cluster-id: /{{ $fullName }}
>>>>>> high-availability.storageDir: s3://redacted-flink-dev/recovery
>>>>>>
>>>>>> *Snippet of Job Manager pod log*
>>>>>> 2021-08-25 06:14:20,652 INFO
>>>>>>  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
>>>>>> StandaloneApplicationClusterEntryPoint down with application status FAILED.
>>>>>> Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
>>>>>> services from the instantiated HighAvailabilityServicesFactory
>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
>>>>>> at
>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
>>>>>> at
>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
>>>>>> at
>>>>>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
>>>>>> at
>>>>>> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
>>>>>> Caused by: java.lang.NullPointerException
>>>>>> at
>>>>>> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
>>>>>> at
>>>>>> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
>>>>>> at
>>>>>> org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
>>>>>> at
>>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
>>>>>> at
>>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
>>>>>> ... 9 more
>>>>>> .
>>>>>>
>>>>>> --
>>>>>> Many thanks,
>>>>>> Jonas
>>>>>>
>>>>>
>>>>
>>>> --
>>>> *Med Vänliga Hälsningar*
>>>> *Jonas Eyob*
>>>>
>>>
>>
>> --
>> *Med Vänliga Hälsningar*
>> *Jonas Eyob*
>>
>

-- 
*Med Vänliga Hälsningar*
*Jonas Eyob*

Re: NullPointerException when using KubernetesHaServicesFactory

Posted by David Morávek <dm...@apache.org>.
Hi Jonas,

Where does the exception pop-up? In job driver, TM, JM? You need to make
sure that the plugin folder is setup for all of them, because they all may
need to access s3 at some point.

Best,
D.

On Wed, Aug 25, 2021 at 11:54 AM jonas eyob <jo...@gmail.com> wrote:

> Hey Thms,
>
> tried the s3p:// option as well - same issue.
>
> > Also check if your user that executes the process is able to read the
> jars.
> Not exactly sure how to do that? The user "flink" in the docker image is
> able to read the contents as far I understand. But maybe that's not how I
> would check it?
>
> Den ons 25 aug. 2021 kl 10:12 skrev Thms Hmm <th...@gmail.com>:
>
>> Hey Jonas,
>> you could also try to use the ´s3p://´ scheme to directly specify that
>> presto should be used. Also check if your user that executes the process is
>> able to read the jars.
>>
>> Am Mi., 25. Aug. 2021 um 10:01 Uhr schrieb jonas eyob <
>> jonas.eyob@gmail.com>:
>>
>>> Thanks David for the quick response!
>>>
>>> *face palm* - Thanks a lot, that seems to have addressed the
>>> NullPointerException issue.
>>> May I also suggest that this [1] page be updated, since it says the key
>>> is "high-availability.cluster-id"
>>>
>>> This led me to another issue however: "org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
>>> Could not find a file system implementation for scheme 's3'"
>>>
>>> The section [2] describe how I can either use environment variables e.g.
>>> ENABLE_BUILT_IN_PLUGINS or bake that in to the image by copying the
>>> provided plugins in opt/ under /plugins
>>>
>>> Dockerfile (snippet)
>>> # Configure flink provided plugin for S3 access
>>> RUN mkdir -p $FLINK_HOME/plugins/s3-fs-presto
>>> RUN cp $FLINK_HOME/opt/flink-s3-fs-presto-*.jar
>>> $FLINK_HOME/plugins/s3-fs-presto/
>>>
>>> When bashing into the image:
>>>
>>> flink@dd86717a92a0:~/plugins/s3-fs-presto$ ls
>>> flink-s3-fs-presto-1.12.5.jar
>>>
>>> Any idea?
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#high-availability
>>> [2]
>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins
>>>
>>>
>>> Den ons 25 aug. 2021 kl 08:00 skrev David Morávek <dm...@apache.org>:
>>>
>>>> Hi Jonas,
>>>>
>>>> this exception is raised because "kubernetes.cluster-id" [1] is not
>>>> set. I'd also recommend setting "kubernetes.namespace" option, unless
>>>> you're using "default" namespace.
>>>>
>>>> I've filled FLINK-23961 [2] so we provide more descriptive warning for
>>>> this issue next time ;)
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/#example-configuration
>>>> [2] https://issues.apache.org/jira/browse/FLINK-23961
>>>>
>>>> Best,
>>>> D.
>>>>
>>>> On Wed, Aug 25, 2021 at 8:48 AM jonas eyob <jo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hey, I've been struggling with this problem now for some days -
>>>>> driving me crazy.
>>>>>
>>>>> I have a standalone kubernetes Flink (1.12.5) using an application
>>>>> cluster mode approach.
>>>>>
>>>>> *The problem*
>>>>> I am getting a NullPointerException when specifying the FQN of the
>>>>> Kubernetes HA Service Factory class
>>>>> i.e.
>>>>> *org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*
>>>>>
>>>>> What other configurations besides the ones specified (here
>>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
>>>>> may I be missing?
>>>>>
>>>>> Details:
>>>>> * we are using a custom image using the flink: 1.12
>>>>> <https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
>>>>> as base image
>>>>>
>>>>> flink-conf.yaml -- thought this may be useful?
>>>>> flink-conf.yaml: |+
>>>>> jobmanager.rpc.address: {{ $fullName }}-jobmanager
>>>>> jobmanager.rpc.port: 6123
>>>>> jobmanager.memory.process.size: 1600m
>>>>> taskmanager.numberOfTaskSlots: 2
>>>>> taskmanager.rpc.port: 6122
>>>>> taskmanager.memory.process.size: 1728m
>>>>> blob.server.port: 6124
>>>>> queryable-state.proxy.ports: 6125
>>>>> parallelism.default: 2
>>>>> scheduler-mode: reactive
>>>>> execution.checkpointing.interval: 10s
>>>>> restart-strategy: fixed-delay
>>>>> restart-strategy.fixed-delay.attempts: 10
>>>>> high-availability:
>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>>>>> high-availability.cluster-id: /{{ $fullName }}
>>>>> high-availability.storageDir: s3://redacted-flink-dev/recovery
>>>>>
>>>>> *Snippet of Job Manager pod log*
>>>>> 2021-08-25 06:14:20,652 INFO
>>>>>  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
>>>>> StandaloneApplicationClusterEntryPoint down with application status FAILED.
>>>>> Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
>>>>> services from the instantiated HighAvailabilityServicesFactory
>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
>>>>> at
>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
>>>>> at
>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
>>>>> at
>>>>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
>>>>> at
>>>>> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
>>>>> Caused by: java.lang.NullPointerException
>>>>> at
>>>>> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
>>>>> at
>>>>> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
>>>>> at
>>>>> org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
>>>>> at
>>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
>>>>> at
>>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
>>>>> ... 9 more
>>>>> .
>>>>>
>>>>> --
>>>>> Many thanks,
>>>>> Jonas
>>>>>
>>>>
>>>
>>> --
>>> *Med Vänliga Hälsningar*
>>> *Jonas Eyob*
>>>
>>
>
> --
> *Med Vänliga Hälsningar*
> *Jonas Eyob*
>

Re: NullPointerException when using KubernetesHaServicesFactory

Posted by jonas eyob <jo...@gmail.com>.
Hey Thms,

tried the s3p:// option as well - same issue.

> Also check if your user that executes the process is able to read the
jars.
Not exactly sure how to do that? The user "flink" in the docker image is
able to read the contents as far I understand. But maybe that's not how I
would check it?

Den ons 25 aug. 2021 kl 10:12 skrev Thms Hmm <th...@gmail.com>:

> Hey Jonas,
> you could also try to use the ´s3p://´ scheme to directly specify that
> presto should be used. Also check if your user that executes the process is
> able to read the jars.
>
> Am Mi., 25. Aug. 2021 um 10:01 Uhr schrieb jonas eyob <
> jonas.eyob@gmail.com>:
>
>> Thanks David for the quick response!
>>
>> *face palm* - Thanks a lot, that seems to have addressed the
>> NullPointerException issue.
>> May I also suggest that this [1] page be updated, since it says the key
>> is "high-availability.cluster-id"
>>
>> This led me to another issue however: "org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
>> Could not find a file system implementation for scheme 's3'"
>>
>> The section [2] describe how I can either use environment variables e.g.
>> ENABLE_BUILT_IN_PLUGINS or bake that in to the image by copying the
>> provided plugins in opt/ under /plugins
>>
>> Dockerfile (snippet)
>> # Configure flink provided plugin for S3 access
>> RUN mkdir -p $FLINK_HOME/plugins/s3-fs-presto
>> RUN cp $FLINK_HOME/opt/flink-s3-fs-presto-*.jar
>> $FLINK_HOME/plugins/s3-fs-presto/
>>
>> When bashing into the image:
>>
>> flink@dd86717a92a0:~/plugins/s3-fs-presto$ ls
>> flink-s3-fs-presto-1.12.5.jar
>>
>> Any idea?
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#high-availability
>> [2]
>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins
>>
>>
>> Den ons 25 aug. 2021 kl 08:00 skrev David Morávek <dm...@apache.org>:
>>
>>> Hi Jonas,
>>>
>>> this exception is raised because "kubernetes.cluster-id" [1] is not set.
>>> I'd also recommend setting "kubernetes.namespace" option, unless you're
>>> using "default" namespace.
>>>
>>> I've filled FLINK-23961 [2] so we provide more descriptive warning for
>>> this issue next time ;)
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/#example-configuration
>>> [2] https://issues.apache.org/jira/browse/FLINK-23961
>>>
>>> Best,
>>> D.
>>>
>>> On Wed, Aug 25, 2021 at 8:48 AM jonas eyob <jo...@gmail.com> wrote:
>>>
>>>> Hey, I've been struggling with this problem now for some days - driving
>>>> me crazy.
>>>>
>>>> I have a standalone kubernetes Flink (1.12.5) using an application
>>>> cluster mode approach.
>>>>
>>>> *The problem*
>>>> I am getting a NullPointerException when specifying the FQN of the
>>>> Kubernetes HA Service Factory class
>>>> i.e.
>>>> *org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*
>>>>
>>>> What other configurations besides the ones specified (here
>>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
>>>> may I be missing?
>>>>
>>>> Details:
>>>> * we are using a custom image using the flink: 1.12
>>>> <https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
>>>> as base image
>>>>
>>>> flink-conf.yaml -- thought this may be useful?
>>>> flink-conf.yaml: |+
>>>> jobmanager.rpc.address: {{ $fullName }}-jobmanager
>>>> jobmanager.rpc.port: 6123
>>>> jobmanager.memory.process.size: 1600m
>>>> taskmanager.numberOfTaskSlots: 2
>>>> taskmanager.rpc.port: 6122
>>>> taskmanager.memory.process.size: 1728m
>>>> blob.server.port: 6124
>>>> queryable-state.proxy.ports: 6125
>>>> parallelism.default: 2
>>>> scheduler-mode: reactive
>>>> execution.checkpointing.interval: 10s
>>>> restart-strategy: fixed-delay
>>>> restart-strategy.fixed-delay.attempts: 10
>>>> high-availability:
>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>>>> high-availability.cluster-id: /{{ $fullName }}
>>>> high-availability.storageDir: s3://redacted-flink-dev/recovery
>>>>
>>>> *Snippet of Job Manager pod log*
>>>> 2021-08-25 06:14:20,652 INFO
>>>>  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
>>>> StandaloneApplicationClusterEntryPoint down with application status FAILED.
>>>> Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
>>>> services from the instantiated HighAvailabilityServicesFactory
>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
>>>> at
>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
>>>> at
>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
>>>> at
>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
>>>> at
>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
>>>> at
>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
>>>> at
>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
>>>> at
>>>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>>>> at
>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
>>>> at
>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
>>>> at
>>>> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
>>>> Caused by: java.lang.NullPointerException
>>>> at
>>>> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
>>>> at
>>>> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
>>>> at
>>>> org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
>>>> at
>>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
>>>> at
>>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
>>>> ... 9 more
>>>> .
>>>>
>>>> --
>>>> Many thanks,
>>>> Jonas
>>>>
>>>
>>
>> --
>> *Med Vänliga Hälsningar*
>> *Jonas Eyob*
>>
>

-- 
*Med Vänliga Hälsningar*
*Jonas Eyob*

Re: NullPointerException when using KubernetesHaServicesFactory

Posted by Thms Hmm <th...@gmail.com>.
Hey Jonas,
you could also try to use the ´s3p://´ scheme to directly specify that
presto should be used. Also check if your user that executes the process is
able to read the jars.

Am Mi., 25. Aug. 2021 um 10:01 Uhr schrieb jonas eyob <jonas.eyob@gmail.com
>:

> Thanks David for the quick response!
>
> *face palm* - Thanks a lot, that seems to have addressed the
> NullPointerException issue.
> May I also suggest that this [1] page be updated, since it says the key is
> "high-availability.cluster-id"
>
> This led me to another issue however: "org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
> Could not find a file system implementation for scheme 's3'"
>
> The section [2] describe how I can either use environment variables e.g.
> ENABLE_BUILT_IN_PLUGINS or bake that in to the image by copying the
> provided plugins in opt/ under /plugins
>
> Dockerfile (snippet)
> # Configure flink provided plugin for S3 access
> RUN mkdir -p $FLINK_HOME/plugins/s3-fs-presto
> RUN cp $FLINK_HOME/opt/flink-s3-fs-presto-*.jar
> $FLINK_HOME/plugins/s3-fs-presto/
>
> When bashing into the image:
>
> flink@dd86717a92a0:~/plugins/s3-fs-presto$ ls
> flink-s3-fs-presto-1.12.5.jar
>
> Any idea?
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#high-availability
> [2]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins
>
>
> Den ons 25 aug. 2021 kl 08:00 skrev David Morávek <dm...@apache.org>:
>
>> Hi Jonas,
>>
>> this exception is raised because "kubernetes.cluster-id" [1] is not set.
>> I'd also recommend setting "kubernetes.namespace" option, unless you're
>> using "default" namespace.
>>
>> I've filled FLINK-23961 [2] so we provide more descriptive warning for
>> this issue next time ;)
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/#example-configuration
>> [2] https://issues.apache.org/jira/browse/FLINK-23961
>>
>> Best,
>> D.
>>
>> On Wed, Aug 25, 2021 at 8:48 AM jonas eyob <jo...@gmail.com> wrote:
>>
>>> Hey, I've been struggling with this problem now for some days - driving
>>> me crazy.
>>>
>>> I have a standalone kubernetes Flink (1.12.5) using an application
>>> cluster mode approach.
>>>
>>> *The problem*
>>> I am getting a NullPointerException when specifying the FQN of the
>>> Kubernetes HA Service Factory class
>>> i.e.
>>> *org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*
>>>
>>> What other configurations besides the ones specified (here
>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
>>> may I be missing?
>>>
>>> Details:
>>> * we are using a custom image using the flink: 1.12
>>> <https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
>>> as base image
>>>
>>> flink-conf.yaml -- thought this may be useful?
>>> flink-conf.yaml: |+
>>> jobmanager.rpc.address: {{ $fullName }}-jobmanager
>>> jobmanager.rpc.port: 6123
>>> jobmanager.memory.process.size: 1600m
>>> taskmanager.numberOfTaskSlots: 2
>>> taskmanager.rpc.port: 6122
>>> taskmanager.memory.process.size: 1728m
>>> blob.server.port: 6124
>>> queryable-state.proxy.ports: 6125
>>> parallelism.default: 2
>>> scheduler-mode: reactive
>>> execution.checkpointing.interval: 10s
>>> restart-strategy: fixed-delay
>>> restart-strategy.fixed-delay.attempts: 10
>>> high-availability:
>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>>> high-availability.cluster-id: /{{ $fullName }}
>>> high-availability.storageDir: s3://redacted-flink-dev/recovery
>>>
>>> *Snippet of Job Manager pod log*
>>> 2021-08-25 06:14:20,652 INFO
>>>  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
>>> StandaloneApplicationClusterEntryPoint down with application status FAILED.
>>> Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
>>> services from the instantiated HighAvailabilityServicesFactory
>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
>>> at
>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
>>> at
>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
>>> at
>>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
>>> at
>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
>>> at
>>> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
>>> Caused by: java.lang.NullPointerException
>>> at
>>> org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
>>> at
>>> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
>>> at
>>> org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
>>> at
>>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
>>> at
>>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
>>> ... 9 more
>>> .
>>>
>>> --
>>> Many thanks,
>>> Jonas
>>>
>>
>
> --
> *Med Vänliga Hälsningar*
> *Jonas Eyob*
>

Re: NullPointerException when using KubernetesHaServicesFactory

Posted by jonas eyob <jo...@gmail.com>.
Thanks David for the quick response!

*face palm* - Thanks a lot, that seems to have addressed the
NullPointerException issue.
May I also suggest that this [1] page be updated, since it says the key is "
high-availability.cluster-id"

This led me to another issue however:
"org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
Could not find a file system implementation for scheme 's3'"

The section [2] describe how I can either use environment variables e.g.
ENABLE_BUILT_IN_PLUGINS or bake that in to the image by copying the
provided plugins in opt/ under /plugins

Dockerfile (snippet)
# Configure flink provided plugin for S3 access
RUN mkdir -p $FLINK_HOME/plugins/s3-fs-presto
RUN cp $FLINK_HOME/opt/flink-s3-fs-presto-*.jar
$FLINK_HOME/plugins/s3-fs-presto/

When bashing into the image:

flink@dd86717a92a0:~/plugins/s3-fs-presto$ ls
flink-s3-fs-presto-1.12.5.jar

Any idea?

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/config/#high-availability
[2]
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins


Den ons 25 aug. 2021 kl 08:00 skrev David Morávek <dm...@apache.org>:

> Hi Jonas,
>
> this exception is raised because "kubernetes.cluster-id" [1] is not set.
> I'd also recommend setting "kubernetes.namespace" option, unless you're
> using "default" namespace.
>
> I've filled FLINK-23961 [2] so we provide more descriptive warning for
> this issue next time ;)
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/#example-configuration
> [2] https://issues.apache.org/jira/browse/FLINK-23961
>
> Best,
> D.
>
> On Wed, Aug 25, 2021 at 8:48 AM jonas eyob <jo...@gmail.com> wrote:
>
>> Hey, I've been struggling with this problem now for some days - driving
>> me crazy.
>>
>> I have a standalone kubernetes Flink (1.12.5) using an application
>> cluster mode approach.
>>
>> *The problem*
>> I am getting a NullPointerException when specifying the FQN of the
>> Kubernetes HA Service Factory class
>> i.e.
>> *org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*
>>
>> What other configurations besides the ones specified (here
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
>> may I be missing?
>>
>> Details:
>> * we are using a custom image using the flink: 1.12
>> <https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
>> as base image
>>
>> flink-conf.yaml -- thought this may be useful?
>> flink-conf.yaml: |+
>> jobmanager.rpc.address: {{ $fullName }}-jobmanager
>> jobmanager.rpc.port: 6123
>> jobmanager.memory.process.size: 1600m
>> taskmanager.numberOfTaskSlots: 2
>> taskmanager.rpc.port: 6122
>> taskmanager.memory.process.size: 1728m
>> blob.server.port: 6124
>> queryable-state.proxy.ports: 6125
>> parallelism.default: 2
>> scheduler-mode: reactive
>> execution.checkpointing.interval: 10s
>> restart-strategy: fixed-delay
>> restart-strategy.fixed-delay.attempts: 10
>> high-availability:
>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
>> high-availability.cluster-id: /{{ $fullName }}
>> high-availability.storageDir: s3://redacted-flink-dev/recovery
>>
>> *Snippet of Job Manager pod log*
>> 2021-08-25 06:14:20,652 INFO
>>  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
>> StandaloneApplicationClusterEntryPoint down with application status FAILED.
>> Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
>> services from the instantiated HighAvailabilityServicesFactory
>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
>> at
>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
>> at
>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
>> at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
>> at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
>> at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
>> at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
>> at
>> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
>> at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
>> at
>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
>> at
>> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
>> Caused by: java.lang.NullPointerException
>> at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
>> at
>> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
>> at
>> org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
>> at
>> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
>> at
>> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
>> ... 9 more
>> .
>>
>> --
>> Many thanks,
>> Jonas
>>
>

-- 
*Med Vänliga Hälsningar*
*Jonas Eyob*

Re: NullPointerException when using KubernetesHaServicesFactory

Posted by David Morávek <dm...@apache.org>.
Hi Jonas,

this exception is raised because "kubernetes.cluster-id" [1] is not set.
I'd also recommend setting "kubernetes.namespace" option, unless you're
using "default" namespace.

I've filled FLINK-23961 [2] so we provide more descriptive warning for this
issue next time ;)

[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/ha/kubernetes_ha/#example-configuration
[2] https://issues.apache.org/jira/browse/FLINK-23961

Best,
D.

On Wed, Aug 25, 2021 at 8:48 AM jonas eyob <jo...@gmail.com> wrote:

> Hey, I've been struggling with this problem now for some days - driving me
> crazy.
>
> I have a standalone kubernetes Flink (1.12.5) using an application cluster
> mode approach.
>
> *The problem*
> I am getting a NullPointerException when specifying the FQN of the
> Kubernetes HA Service Factory class
> i.e.
> *org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory*
>
> What other configurations besides the ones specified (here
> <https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions>)
> may I be missing?
>
> Details:
> * we are using a custom image using the flink: 1.12
> <https://hub.docker.com/layers/flink/library/flink/1.12/images/sha256-4b4290888e30d27a28517bac3b1678674cd4b17aa7b8329969d1d12fcdf68f02?context=explore>
> as base image
>
> flink-conf.yaml -- thought this may be useful?
> flink-conf.yaml: |+
> jobmanager.rpc.address: {{ $fullName }}-jobmanager
> jobmanager.rpc.port: 6123
> jobmanager.memory.process.size: 1600m
> taskmanager.numberOfTaskSlots: 2
> taskmanager.rpc.port: 6122
> taskmanager.memory.process.size: 1728m
> blob.server.port: 6124
> queryable-state.proxy.ports: 6125
> parallelism.default: 2
> scheduler-mode: reactive
> execution.checkpointing.interval: 10s
> restart-strategy: fixed-delay
> restart-strategy.fixed-delay.attempts: 10
> high-availability:
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
> high-availability.cluster-id: /{{ $fullName }}
> high-availability.storageDir: s3://redacted-flink-dev/recovery
>
> *Snippet of Job Manager pod log*
> 2021-08-25 06:14:20,652 INFO
>  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting
> StandaloneApplicationClusterEntryPoint down with application status FAILED.
> Diagnostics org.apache.flink.util.FlinkException: Could not create the ha
> services from the instantiated HighAvailabilityServicesFactory
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.
> at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:268)
> at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:124)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createHaServices(ClusterEntrypoint.java:338)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:296)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:224)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$1(ClusterEntrypoint.java:178)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:175)
> at
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:585)
> at
> org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85)
> Caused by: java.lang.NullPointerException
> at org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:59)
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.<init>(Fabric8FlinkKubeClient.java:85)
> at
> org.apache.flink.kubernetes.kubeclient.FlinkKubeClientFactory.fromConfiguration(FlinkKubeClientFactory.java:106)
> at
> org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory.createHAServices(KubernetesHaServicesFactory.java:37)
> at
> org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createCustomHAServices(HighAvailabilityServicesUtils.java:265)
> ... 9 more
> .
>
> --
> Many thanks,
> Jonas
>