You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alexander Fedulov <al...@deltastream.io> on 2022/11/04 10:15:04 UTC
UDFs classloading changes in 1.16
Hi everyone,
1.16 introduced quite a lot of changes with respect to classloading in the
Table API. The way UDFs could previously be loaded from JARs in 1.15 does
not work in 1.16 anymore - it fails with the ClassNotFound exception when
UDFs are used at runtime.
Here is a repository with a reproducible example:
https://github.com/afedulov/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java
It works as is (Flink 1.15.2) and fails when switching the dependencies to
1.16.0.
Here are some of the PRs that, I believe, might be related to the issue:
https://github.com/apache/flink/pull/20001
https://github.com/apache/flink/pull/19845
https://github.com/apache/flink/pull/20211 (fixes a similar issue
introduced after classloading changes in 1.16)
How can UDF JARs be loaded in 1.16?
Best,
Alexander Fedulov
Fwd: UDFs classloading changes in 1.16
Posted by Martijn Visser <ma...@apache.org>.
I'm forwarding this issue to the Dev mailing list since it should be
discussed there, moving User ML to BCC. Also adding Dong since he was
involved in the discussion
Thanks for reporting this Charles. I've re-opened
https://issues.apache.org/jira/browse/FLINK-28897 but it could be that this
ultimately ends up as a new ticket. Due to Chinese New Year, replies might
take a little longer than normally.
Best regards,
Martijn
---------- Forwarded message ---------
Van: Charles Tan <ct...@gmail.com>
Date: vr 20 jan. 2023 om 16:21
Subject: Re: UDFs classloading changes in 1.16
To: Alexander Fedulov <al...@deltastream.io>
Cc: Leonard Xu <xb...@gmail.com>, user <us...@flink.apache.org>, Shengkai
Fang <fs...@gmail.com>
Hi everyone,
I wanted to bring some attention to this issue again. It seems that
the suggested approach to creating UDFs in
https://issues.apache.org/jira/browse/FLINK-29890 breaks when
checkpointing is enabled.
https://issues.apache.org/jira/browse/FLINK-28897 reports a very
similar issue. I've updated both tickets with comments and new code
snippets. Is there another way to load UDFs in Flink 1.16?
Code examples for reference:
https://github.com/charles-tan/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java
Thanks,
Charles
On Fri, Nov 4, 2022 at 10:51 AM Alexander Fedulov <al...@deltastream.io>
wrote:
>
> Hi Leonard,
>
> Sure, here is the new ticket:
https://issues.apache.org/jira/browse/FLINK-29890
>
> Best,
> Alexander Fedulov
>
> On Fri, Nov 4, 2022 at 2:12 PM Leonard Xu <xb...@gmail.com> wrote:
>>
>> Thanks Alexander for reporting this issue, Could you open a jira ticket
as well?
>>
>> CC: Shengkai, please take a look this ticket, looks like an
incompatibility change.
>>
>> Best,
>> Leonard
>>
>>
>>
>> 2022年11月4日 下午6:15，Alexander Fedulov <al...@deltastream.io> 写道：
>>
>> Hi everyone,
>>
>> 1.16 introduced quite a lot of changes with respect to classloading in
the Table API. The way UDFs could previously be loaded from JARs in 1.15
does not work in 1.16 anymore - it fails with the ClassNotFound exception
when UDFs are used at runtime.
>>
>> Here is a repository with a reproducible example:
>>
https://github.com/afedulov/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java
>>
>> It works as is (Flink 1.15.2) and fails when switching the dependencies
to 1.16.0.
>>
>> Here are some of the PRs that, I believe, might be related to the issue:
>> https://github.com/apache/flink/pull/20001
>> https://github.com/apache/flink/pull/19845
>> https://github.com/apache/flink/pull/20211 (fixes a similar issue
introduced after classloading changes in 1.16)
>>
>> How can UDF JARs be loaded in 1.16?
>>
>> Best,
>> Alexander Fedulov
>>
>>
Fwd: UDFs classloading changes in 1.16
Posted by Martijn Visser <ma...@apache.org>.
I'm forwarding this issue to the Dev mailing list since it should be
discussed there, moving User ML to BCC. Also adding Dong since he was
involved in the discussion
Thanks for reporting this Charles. I've re-opened
https://issues.apache.org/jira/browse/FLINK-28897 but it could be that this
ultimately ends up as a new ticket. Due to Chinese New Year, replies might
take a little longer than normally.
Best regards,
Martijn
---------- Forwarded message ---------
Van: Charles Tan <ct...@gmail.com>
Date: vr 20 jan. 2023 om 16:21
Subject: Re: UDFs classloading changes in 1.16
To: Alexander Fedulov <al...@deltastream.io>
Cc: Leonard Xu <xb...@gmail.com>, user <us...@flink.apache.org>, Shengkai
Fang <fs...@gmail.com>
Hi everyone,
I wanted to bring some attention to this issue again. It seems that
the suggested approach to creating UDFs in
https://issues.apache.org/jira/browse/FLINK-29890 breaks when
checkpointing is enabled.
https://issues.apache.org/jira/browse/FLINK-28897 reports a very
similar issue. I've updated both tickets with comments and new code
snippets. Is there another way to load UDFs in Flink 1.16?
Code examples for reference:
https://github.com/charles-tan/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java
Thanks,
Charles
On Fri, Nov 4, 2022 at 10:51 AM Alexander Fedulov <al...@deltastream.io>
wrote:
>
> Hi Leonard,
>
> Sure, here is the new ticket:
https://issues.apache.org/jira/browse/FLINK-29890
>
> Best,
> Alexander Fedulov
>
> On Fri, Nov 4, 2022 at 2:12 PM Leonard Xu <xb...@gmail.com> wrote:
>>
>> Thanks Alexander for reporting this issue, Could you open a jira ticket
as well?
>>
>> CC: Shengkai, please take a look this ticket, looks like an
incompatibility change.
>>
>> Best,
>> Leonard
>>
>>
>>
>> 2022年11月4日 下午6:15，Alexander Fedulov <al...@deltastream.io> 写道：
>>
>> Hi everyone,
>>
>> 1.16 introduced quite a lot of changes with respect to classloading in
the Table API. The way UDFs could previously be loaded from JARs in 1.15
does not work in 1.16 anymore - it fails with the ClassNotFound exception
when UDFs are used at runtime.
>>
>> Here is a repository with a reproducible example:
>>
https://github.com/afedulov/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java
>>
>> It works as is (Flink 1.15.2) and fails when switching the dependencies
to 1.16.0.
>>
>> Here are some of the PRs that, I believe, might be related to the issue:
>> https://github.com/apache/flink/pull/20001
>> https://github.com/apache/flink/pull/19845
>> https://github.com/apache/flink/pull/20211 (fixes a similar issue
introduced after classloading changes in 1.16)
>>
>> How can UDF JARs be loaded in 1.16?
>>
>> Best,
>> Alexander Fedulov
>>
>>
Re: UDFs classloading changes in 1.16
Posted by Charles Tan <ct...@gmail.com>.
Hi everyone,
I wanted to bring some attention to this issue again. It seems that
the suggested approach to creating UDFs in
https://issues.apache.org/jira/browse/FLINK-29890 breaks when
checkpointing is enabled.
https://issues.apache.org/jira/browse/FLINK-28897 reports a very
similar issue. I've updated both tickets with comments and new code
snippets. Is there another way to load UDFs in Flink 1.16?
Code examples for reference:
https://github.com/charles-tan/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java
Thanks,
Charles
On Fri, Nov 4, 2022 at 10:51 AM Alexander Fedulov <al...@deltastream.io> wrote:
>
> Hi Leonard,
>
> Sure, here is the new ticket: https://issues.apache.org/jira/browse/FLINK-29890
>
> Best,
> Alexander Fedulov
>
> On Fri, Nov 4, 2022 at 2:12 PM Leonard Xu <xb...@gmail.com> wrote:
>>
>> Thanks Alexander for reporting this issue, Could you open a jira ticket as well?
>>
>> CC: Shengkai, please take a look this ticket, looks like an incompatibility change.
>>
>> Best,
>> Leonard
>>
>>
>>
>> 2022年11月4日 下午6:15，Alexander Fedulov <al...@deltastream.io> 写道：
>>
>> Hi everyone,
>>
>> 1.16 introduced quite a lot of changes with respect to classloading in the Table API. The way UDFs could previously be loaded from JARs in 1.15 does not work in 1.16 anymore - it fails with the ClassNotFound exception when UDFs are used at runtime.
>>
>> Here is a repository with a reproducible example:
>> https://github.com/afedulov/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java
>>
>> It works as is (Flink 1.15.2) and fails when switching the dependencies to 1.16.0.
>>
>> Here are some of the PRs that, I believe, might be related to the issue:
>> https://github.com/apache/flink/pull/20001
>> https://github.com/apache/flink/pull/19845
>> https://github.com/apache/flink/pull/20211 (fixes a similar issue introduced after classloading changes in 1.16)
>>
>> How can UDF JARs be loaded in 1.16?
>>
>> Best,
>> Alexander Fedulov
>>
>>
Re: UDFs classloading changes in 1.16
Posted by Alexander Fedulov <al...@deltastream.io>.
Hi Leonard,
Sure, here is the new ticket:
https://issues.apache.org/jira/browse/FLINK-29890
Best,
Alexander Fedulov
On Fri, Nov 4, 2022 at 2:12 PM Leonard Xu <xb...@gmail.com> wrote:
> Thanks Alexander for reporting this issue, Could you open a jira ticket as
> well?
>
> CC: Shengkai, please take a look this ticket, looks like an
> incompatibility change.
>
> Best,
> Leonard
>
>
>
> 2022年11月4日 下午6:15，Alexander Fedulov <al...@deltastream.io> 写道：
>
> Hi everyone,
>
> 1.16 introduced quite a lot of changes with respect to classloading in the
> Table API. The way UDFs could previously be loaded from JARs in 1.15 does
> not work in 1.16 anymore - it fails with the ClassNotFound exception when
> UDFs are used at runtime.
>
> Here is a repository with a reproducible example:
>
> https://github.com/afedulov/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java
>
> It works as is (Flink 1.15.2) and fails when switching the dependencies to
> 1.16.0.
>
> Here are some of the PRs that, I believe, might be related to the issue:
> https://github.com/apache/flink/pull/20001
> https://github.com/apache/flink/pull/19845
> https://github.com/apache/flink/pull/20211 (fixes a similar issue
> introduced after classloading changes in 1.16)
>
> How can UDF JARs be loaded in 1.16?
>
> Best,
> Alexander Fedulov
>
>
>
Re: UDFs classloading changes in 1.16
Posted by Leonard Xu <xb...@gmail.com>.
Thanks Alexander for reporting this issue, Could you open a jira ticket as well?
CC: Shengkai, please take a look this ticket, looks like an incompatibility change.
Best,
Leonard
> 2022年11月4日 下午6:15，Alexander Fedulov <al...@deltastream.io> 写道：
>
> Hi everyone,
>
> 1.16 introduced quite a lot of changes with respect to classloading in the Table API. The way UDFs could previously be loaded from JARs in 1.15 does not work in 1.16 anymore - it fails with the ClassNotFound exception when UDFs are used at runtime.
>
> Here is a repository with a reproducible example:
> https://github.com/afedulov/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java <https://github.com/afedulov/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java>
>
> It works as is (Flink 1.15.2) and fails when switching the dependencies to 1.16.0.
>
> Here are some of the PRs that, I believe, might be related to the issue:
> https://github.com/apache/flink/pull/20001 <https://github.com/apache/flink/pull/20001>
> https://github.com/apache/flink/pull/19845 <https://github.com/apache/flink/pull/19845>
> https://github.com/apache/flink/pull/20211 <https://github.com/apache/flink/pull/20211> (fixes a similar issue introduced after classloading changes in 1.16)
>
> How can UDF JARs be loaded in 1.16?
>
> Best,
> Alexander Fedulov