Classloading issues with Flink Operator / Kubernetes Native

[Yaroslav Tkachenko <ya...@goldsky.com>]

Hey everyone,

I’m migrating a Flink Kubernetes standalone job to the Flink operator (with
Kubernetes native mode).

I have a lot of classloading issues when trying to run with the operator in
native mode. For example, I have a Postgres driver as a dependency (I can
confirm the files are included in the uber jar), but I still get
"java.sql.SQLException: No suitable driver found for jdbc:postgresql:..."
exception.

In the Kubernetes standalone setup my uber jar is placed in the
/opt/flink/lib folder, this is what I specify as "jarURI" in the operator
config. Is this supported? Should I only be using /opt/flink/usrlib?

Thanks for any suggestions.

Re: Classloading issues with Flink Operator / Kubernetes Native

[Javier Vegas <jv...@strava.com>]

Version 1.15.2, there is no /opt/flink/usrlib folder created

El mar, 20 sept 2022 a las 20:53, Yaroslav Tkachenko (<ya...@goldsky.com>)
escribió:

> Interesting, do you see the /opt/flink/usrlib folder created as well?
> Also, what Flink version do you use?
>
> Thanks.
>
> On Tue, Sep 20, 2022 at 4:04 PM Javier Vegas <jv...@strava.com> wrote:
>
>>
>> jarURI: local:///opt/flink/lib/MYJARNAME.jar
>>
>> El mar, 20 sept 2022 a las 0:25, Yaroslav Tkachenko (<
>> yaroslav@goldsky.com>) escribió:
>>
>>> Hi Javier,
>>>
>>> What do you specify as a jarURI?
>>>
>>> On Mon, Sep 19, 2022 at 3:56 PM Javier Vegas <jv...@strava.com> wrote:
>>>
>>>> I am doing the same thing (migrating from standalone to operator in
>>>> native mode) and also have my jar in /opt/flink/lib but for me it works
>>>> fine, no class loading errors on app startup.
>>>>
>>>> El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko (<
>>>> yaroslav@goldsky.com>) escribió:
>>>>
>>>>> Application mode. I've done a bit more research and created
>>>>> https://issues.apache.org/jira/browse/FLINK-29288, planning to work
>>>>> on a PR today.
>>>>>
>>>>> TLDR: currently Flink operator always creates /opt/flink/usrlib folder
>>>>> and forces you to specify the jarURI parameter, which is passed as
>>>>> pipeline.jars / pipeline.classpaths configuration options. This leads to
>>>>> the jar being loaded twice by different classloaders (system and user
>>>>> ones).
>>>>>
>>>>> On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl <ma...@aiven.io>
>>>>> wrote:
>>>>>
>>>>>> Are you deploying the job in session or application mode? Could you
>>>>>> provide the stacktrace. I'm wondering whether that would be helpful to pin
>>>>>> a code location for further investigation.
>>>>>> So far, I couldn't come up with a definite answer about placing the
>>>>>> jar in the lib directory. Initially, I would have thought that it's fine
>>>>>> considering that all dependencies are included and the job jar itself ends
>>>>>> up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer
>>>>>> to that one.
>>>>>>
>>>>>> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko <
>>>>>> yaroslav@goldsky.com> wrote:
>>>>>>
>>>>>>> Hey everyone,
>>>>>>>
>>>>>>> I’m migrating a Flink Kubernetes standalone job to the Flink
>>>>>>> operator (with Kubernetes native mode).
>>>>>>>
>>>>>>> I have a lot of classloading issues when trying to run with
>>>>>>> the operator in native mode. For example, I have a Postgres driver as a
>>>>>>> dependency (I can confirm the files are included in the uber jar), but I
>>>>>>> still get "java.sql.SQLException: No suitable driver found for
>>>>>>> jdbc:postgresql:..." exception.
>>>>>>>
>>>>>>> In the Kubernetes standalone setup my uber jar is placed in the
>>>>>>> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator
>>>>>>> config. Is this supported? Should I only be using /opt/flink/usrlib?
>>>>>>>
>>>>>>> Thanks for any suggestions.
>>>>>>>
>>>>>>

Re: Classloading issues with Flink Operator / Kubernetes Native

[Yaroslav Tkachenko <ya...@goldsky.com>]

Interesting, do you see the /opt/flink/usrlib folder created as well? Also,
what Flink version do you use?

Thanks.

On Tue, Sep 20, 2022 at 4:04 PM Javier Vegas <jv...@strava.com> wrote:

>
> jarURI: local:///opt/flink/lib/MYJARNAME.jar
>
> El mar, 20 sept 2022 a las 0:25, Yaroslav Tkachenko (<ya...@goldsky.com>)
> escribió:
>
>> Hi Javier,
>>
>> What do you specify as a jarURI?
>>
>> On Mon, Sep 19, 2022 at 3:56 PM Javier Vegas <jv...@strava.com> wrote:
>>
>>> I am doing the same thing (migrating from standalone to operator in
>>> native mode) and also have my jar in /opt/flink/lib but for me it works
>>> fine, no class loading errors on app startup.
>>>
>>> El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko (<
>>> yaroslav@goldsky.com>) escribió:
>>>
>>>> Application mode. I've done a bit more research and created
>>>> https://issues.apache.org/jira/browse/FLINK-29288, planning to work on
>>>> a PR today.
>>>>
>>>> TLDR: currently Flink operator always creates /opt/flink/usrlib folder
>>>> and forces you to specify the jarURI parameter, which is passed as
>>>> pipeline.jars / pipeline.classpaths configuration options. This leads to
>>>> the jar being loaded twice by different classloaders (system and user
>>>> ones).
>>>>
>>>> On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl <ma...@aiven.io>
>>>> wrote:
>>>>
>>>>> Are you deploying the job in session or application mode? Could you
>>>>> provide the stacktrace. I'm wondering whether that would be helpful to pin
>>>>> a code location for further investigation.
>>>>> So far, I couldn't come up with a definite answer about placing the
>>>>> jar in the lib directory. Initially, I would have thought that it's fine
>>>>> considering that all dependencies are included and the job jar itself ends
>>>>> up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer
>>>>> to that one.
>>>>>
>>>>> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko <
>>>>> yaroslav@goldsky.com> wrote:
>>>>>
>>>>>> Hey everyone,
>>>>>>
>>>>>> I’m migrating a Flink Kubernetes standalone job to the Flink operator
>>>>>> (with Kubernetes native mode).
>>>>>>
>>>>>> I have a lot of classloading issues when trying to run with
>>>>>> the operator in native mode. For example, I have a Postgres driver as a
>>>>>> dependency (I can confirm the files are included in the uber jar), but I
>>>>>> still get "java.sql.SQLException: No suitable driver found for
>>>>>> jdbc:postgresql:..." exception.
>>>>>>
>>>>>> In the Kubernetes standalone setup my uber jar is placed in the
>>>>>> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator
>>>>>> config. Is this supported? Should I only be using /opt/flink/usrlib?
>>>>>>
>>>>>> Thanks for any suggestions.
>>>>>>
>>>>>

Re: Classloading issues with Flink Operator / Kubernetes Native

[Javier Vegas <jv...@strava.com>]

jarURI: local:///opt/flink/lib/MYJARNAME.jar

El mar, 20 sept 2022 a las 0:25, Yaroslav Tkachenko (<ya...@goldsky.com>)
escribió:

> Hi Javier,
>
> What do you specify as a jarURI?
>
> On Mon, Sep 19, 2022 at 3:56 PM Javier Vegas <jv...@strava.com> wrote:
>
>> I am doing the same thing (migrating from standalone to operator in
>> native mode) and also have my jar in /opt/flink/lib but for me it works
>> fine, no class loading errors on app startup.
>>
>> El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko (<
>> yaroslav@goldsky.com>) escribió:
>>
>>> Application mode. I've done a bit more research and created
>>> https://issues.apache.org/jira/browse/FLINK-29288, planning to work on
>>> a PR today.
>>>
>>> TLDR: currently Flink operator always creates /opt/flink/usrlib folder
>>> and forces you to specify the jarURI parameter, which is passed as
>>> pipeline.jars / pipeline.classpaths configuration options. This leads to
>>> the jar being loaded twice by different classloaders (system and user
>>> ones).
>>>
>>> On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl <ma...@aiven.io>
>>> wrote:
>>>
>>>> Are you deploying the job in session or application mode? Could you
>>>> provide the stacktrace. I'm wondering whether that would be helpful to pin
>>>> a code location for further investigation.
>>>> So far, I couldn't come up with a definite answer about placing the jar
>>>> in the lib directory. Initially, I would have thought that it's fine
>>>> considering that all dependencies are included and the job jar itself ends
>>>> up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer
>>>> to that one.
>>>>
>>>> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko <
>>>> yaroslav@goldsky.com> wrote:
>>>>
>>>>> Hey everyone,
>>>>>
>>>>> I’m migrating a Flink Kubernetes standalone job to the Flink operator
>>>>> (with Kubernetes native mode).
>>>>>
>>>>> I have a lot of classloading issues when trying to run with
>>>>> the operator in native mode. For example, I have a Postgres driver as a
>>>>> dependency (I can confirm the files are included in the uber jar), but I
>>>>> still get "java.sql.SQLException: No suitable driver found for
>>>>> jdbc:postgresql:..." exception.
>>>>>
>>>>> In the Kubernetes standalone setup my uber jar is placed in the
>>>>> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator
>>>>> config. Is this supported? Should I only be using /opt/flink/usrlib?
>>>>>
>>>>> Thanks for any suggestions.
>>>>>
>>>>

Re: Classloading issues with Flink Operator / Kubernetes Native

[Yaroslav Tkachenko <ya...@goldsky.com>]

Hi Javier,

What do you specify as a jarURI?

On Mon, Sep 19, 2022 at 3:56 PM Javier Vegas <jv...@strava.com> wrote:

> I am doing the same thing (migrating from standalone to operator in native
> mode) and also have my jar in /opt/flink/lib but for me it works fine, no
> class loading errors on app startup.
>
> El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko (<ya...@goldsky.com>)
> escribió:
>
>> Application mode. I've done a bit more research and created
>> https://issues.apache.org/jira/browse/FLINK-29288, planning to work on a
>> PR today.
>>
>> TLDR: currently Flink operator always creates /opt/flink/usrlib folder
>> and forces you to specify the jarURI parameter, which is passed as
>> pipeline.jars / pipeline.classpaths configuration options. This leads to
>> the jar being loaded twice by different classloaders (system and user
>> ones).
>>
>> On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl <ma...@aiven.io>
>> wrote:
>>
>>> Are you deploying the job in session or application mode? Could you
>>> provide the stacktrace. I'm wondering whether that would be helpful to pin
>>> a code location for further investigation.
>>> So far, I couldn't come up with a definite answer about placing the jar
>>> in the lib directory. Initially, I would have thought that it's fine
>>> considering that all dependencies are included and the job jar itself ends
>>> up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer
>>> to that one.
>>>
>>> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko <ya...@goldsky.com>
>>> wrote:
>>>
>>>> Hey everyone,
>>>>
>>>> I’m migrating a Flink Kubernetes standalone job to the Flink operator
>>>> (with Kubernetes native mode).
>>>>
>>>> I have a lot of classloading issues when trying to run with
>>>> the operator in native mode. For example, I have a Postgres driver as a
>>>> dependency (I can confirm the files are included in the uber jar), but I
>>>> still get "java.sql.SQLException: No suitable driver found for
>>>> jdbc:postgresql:..." exception.
>>>>
>>>> In the Kubernetes standalone setup my uber jar is placed in the
>>>> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator
>>>> config. Is this supported? Should I only be using /opt/flink/usrlib?
>>>>
>>>> Thanks for any suggestions.
>>>>
>>>

Re: Classloading issues with Flink Operator / Kubernetes Native

[Javier Vegas <jv...@strava.com>]

I am doing the same thing (migrating from standalone to operator in native
mode) and also have my jar in /opt/flink/lib but for me it works fine, no
class loading errors on app startup.

El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko (<ya...@goldsky.com>)
escribió:

> Application mode. I've done a bit more research and created
> https://issues.apache.org/jira/browse/FLINK-29288, planning to work on a
> PR today.
>
> TLDR: currently Flink operator always creates /opt/flink/usrlib folder and
> forces you to specify the jarURI parameter, which is passed as
> pipeline.jars / pipeline.classpaths configuration options. This leads to
> the jar being loaded twice by different classloaders (system and user
> ones).
>
> On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl <ma...@aiven.io>
> wrote:
>
>> Are you deploying the job in session or application mode? Could you
>> provide the stacktrace. I'm wondering whether that would be helpful to pin
>> a code location for further investigation.
>> So far, I couldn't come up with a definite answer about placing the jar
>> in the lib directory. Initially, I would have thought that it's fine
>> considering that all dependencies are included and the job jar itself ends
>> up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer
>> to that one.
>>
>> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko <ya...@goldsky.com>
>> wrote:
>>
>>> Hey everyone,
>>>
>>> I’m migrating a Flink Kubernetes standalone job to the Flink operator
>>> (with Kubernetes native mode).
>>>
>>> I have a lot of classloading issues when trying to run with the operator
>>> in native mode. For example, I have a Postgres driver as a dependency (I
>>> can confirm the files are included in the uber jar), but I still get
>>> "java.sql.SQLException: No suitable driver found for jdbc:postgresql:..."
>>> exception.
>>>
>>> In the Kubernetes standalone setup my uber jar is placed in the
>>> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator
>>> config. Is this supported? Should I only be using /opt/flink/usrlib?
>>>
>>> Thanks for any suggestions.
>>>
>>

Re: Classloading issues with Flink Operator / Kubernetes Native

[Yaroslav Tkachenko <ya...@goldsky.com>]

Application mode. I've done a bit more research and created
https://issues.apache.org/jira/browse/FLINK-29288, planning to work on a PR
today.

TLDR: currently Flink operator always creates /opt/flink/usrlib folder and
forces you to specify the jarURI parameter, which is passed as
pipeline.jars / pipeline.classpaths configuration options. This leads to
the jar being loaded twice by different classloaders (system and user
ones).

On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl <ma...@aiven.io>
wrote:

> Are you deploying the job in session or application mode? Could you
> provide the stacktrace. I'm wondering whether that would be helpful to pin
> a code location for further investigation.
> So far, I couldn't come up with a definite answer about placing the jar in
> the lib directory. Initially, I would have thought that it's fine
> considering that all dependencies are included and the job jar itself ends
> up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer
> to that one.
>
> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
>> Hey everyone,
>>
>> I’m migrating a Flink Kubernetes standalone job to the Flink operator
>> (with Kubernetes native mode).
>>
>> I have a lot of classloading issues when trying to run with the operator
>> in native mode. For example, I have a Postgres driver as a dependency (I
>> can confirm the files are included in the uber jar), but I still get
>> "java.sql.SQLException: No suitable driver found for jdbc:postgresql:..."
>> exception.
>>
>> In the Kubernetes standalone setup my uber jar is placed in the
>> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator
>> config. Is this supported? Should I only be using /opt/flink/usrlib?
>>
>> Thanks for any suggestions.
>>
>

Re: Classloading issues with Flink Operator / Kubernetes Native

[Matthias Pohl via user <us...@flink.apache.org>]

Are you deploying the job in session or application mode? Could you provide
the stacktrace. I'm wondering whether that would be helpful to pin a code
location for further investigation.
So far, I couldn't come up with a definite answer about placing the jar in
the lib directory. Initially, I would have thought that it's fine
considering that all dependencies are included and the job jar itself ends
up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer
to that one.

On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:

> Hey everyone,
>
> I’m migrating a Flink Kubernetes standalone job to the Flink operator
> (with Kubernetes native mode).
>
> I have a lot of classloading issues when trying to run with the operator
> in native mode. For example, I have a Postgres driver as a dependency (I
> can confirm the files are included in the uber jar), but I still get
> "java.sql.SQLException: No suitable driver found for jdbc:postgresql:..."
> exception.
>
> In the Kubernetes standalone setup my uber jar is placed in the
> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator
> config. Is this supported? Should I only be using /opt/flink/usrlib?
>
> Thanks for any suggestions.
>