You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Prasanna kumar <pr...@gmail.com> on 2021/04/20 13:07:27 UTC
Flink support for Kafka versions
Hi Flinksters,
We are researching about if we could use the latest version of kafka (2.6.1
or 2.7.0)
Since we are using Flink as a processor , we came across this
https://issues.apache.org/jira/browse/FLINK-19168.
It says that it does not support version 2.5.0 and beyond.
That was created 8 months back , just checking if there is any effort on
that front.
Thanks,
Prasanna
Re: Re: Flink support for Kafka versions
Posted by Arvid Heise <ar...@apache.org>.
Hi Prasanna,
we cannot backport these changes as they change dependency coordinates.
What I'd recommend you to do is to use the Flink 2_12 dependencies and then
bump Kafka client to 2.8.X in your application jar. That should work in any
case.
Best,
Arvid
On Wed, Oct 27, 2021 at 12:05 PM Yun Gao <yu...@aliyun.com> wrote:
> Hi Prasanna,
>
> I think the two issues would not be picked back to 1.12.x since they are
> all large modification and user-visiable and thus they should be not
> suitable
> to be picked back to the bugfix version. Which scala version are you going
> to
> use? Flink currently provided both version with scala 2.11 and scala 2.12
> [1].
>
> [1] https://flink.apache.org/downloads.html#apache-flink-1125
>
>
> ------------------Original Mail ------------------
> *Sender:*Prasanna kumar <pr...@gmail.com>
> *Send Date:*Wed Oct 27 00:01:45 2021
> *Recipients:*Arvid Heise <ar...@apache.org>
> *CC:*Austin Cawley-Edwards <au...@gmail.com>, user <
> user@flink.apache.org>, Chesnay Schepler <ch...@apache.org>
> *Subject:*Re: Flink support for Kafka versions
>
>> Hi ,
>>
>> We are using Kafka broker version 2.4.1.1.
>> Also kafka client 2.4.1.1 jar which is part of flink kafka connector
>> recently was marked with high security issue.
>> So we excluded the dependency and overriden it with kafka client 2.8.1
>> client jar and it works fine with the 2.4.1.1 broker. ( since its backward
>> compatible)
>>
>> 1) If we tried to connect to broker with 2.8.1 version (with kafka client
>> 2.8.1 jar override ) would it work or would it throw errors because of
>> Scala 2.11?
>> 2) https://issues.apache.org/jira/browse/FLINK-20845 is marked as done
>> but the fix version is marked as 1.15.0.. Wouldn't this change be available
>> for 1.12.x ?
>> 3) https://issues.apache.org/jira/browse/FLINK-14105 is marked as done
>> but the fix version is marked as 1.14.0.. Wouldn't this change be available
>> for 1.12.x ?
>>
>> Thanks,
>> Prasanna.
>>
>> On Wed, Apr 21, 2021 at 11:03 PM Arvid Heise <ar...@apache.org> wrote:
>>
>>> I'm wondering if we could shade scala 1.13 dependencies inside the Kafka
>>> connector? Then we would be independent of the rather big FLINK-20845.
>>>
>>> On Tue, Apr 20, 2021 at 5:54 PM Austin Cawley-Edwards <
>>> austin.cawley@gmail.com> wrote:
>>>
>>>> Hi Prasanna,
>>>>
>>>> It looks like the Kafka 2.5.0 connector upgrade is tied to dropping
>>>> support for Scala 2.11. The best place to track that would be the ticket
>>>> for Scala 2.13 support, FLINK-13414 [1], and its subtask FLINK-20845 [2].
>>>>
>>>> I have listed FLINK-20845 as a blocker for FLINK-19168 for better
>>>> visibility.
>>>>
>>>> Best,
>>>> Austin
>>>>
>>>> [1]: https://issues.apache.org/jira/browse/FLINK-13414
>>>> [2]: https://issues.apache.org/jira/browse/FLINK-20845
>>>>
>>>> On Tue, Apr 20, 2021 at 9:08 AM Prasanna kumar <
>>>> prasannakumarramani@gmail.com> wrote:
>>>>
>>>>> Hi Flinksters,
>>>>>
>>>>> We are researching about if we could use the latest version of kafka
>>>>> (2.6.1 or 2.7.0)
>>>>>
>>>>> Since we are using Flink as a processor , we came across this
>>>>> https://issues.apache.org/jira/browse/FLINK-19168.
>>>>>
>>>>> It says that it does not support version 2.5.0 and beyond.
>>>>>
>>>>> That was created 8 months back , just checking if there is any effort
>>>>> on that front.
>>>>>
>>>>> Thanks,
>>>>> Prasanna
>>>>>
>>>>
Re: Re: Flink support for Kafka versions
Posted by Yun Gao <yu...@aliyun.com>.
Hi Prasanna,
I think the two issues would not be picked back to 1.12.x since they are
all large modification and user-visiable and thus they should be not suitable
to be picked back to the bugfix version. Which scala version are you going to
use? Flink currently provided both version with scala 2.11 and scala 2.12 [1].
[1] https://flink.apache.org/downloads.html#apache-flink-1125
------------------Original Mail ------------------
Sender:Prasanna kumar <pr...@gmail.com>
Send Date:Wed Oct 27 00:01:45 2021
Recipients:Arvid Heise <ar...@apache.org>
CC:Austin Cawley-Edwards <au...@gmail.com>, user <us...@flink.apache.org>, Chesnay Schepler <ch...@apache.org>
Subject:Re: Flink support for Kafka versions
Hi ,
We are using Kafka broker version 2.4.1.1.
Also kafka client 2.4.1.1 jar which is part of flink kafka connector recently was marked with high security issue.
So we excluded the dependency and overriden it with kafka client 2.8.1 client jar and it works fine with the 2.4.1.1 broker. ( since its backward compatible)
1) If we tried to connect to broker with 2.8.1 version (with kafka client 2.8.1 jar override ) would it work or would it throw errors because of Scala 2.11?
2) https://issues.apache.org/jira/browse/FLINK-20845 is marked as done but the fix version is marked as 1.15.0.. Wouldn't this change be available for 1.12.x ?
3) https://issues.apache.org/jira/browse/FLINK-14105 is marked as done but the fix version is marked as 1.14.0.. Wouldn't this change be available for 1.12.x ?
Thanks,
Prasanna.
On Wed, Apr 21, 2021 at 11:03 PM Arvid Heise <ar...@apache.org> wrote:
I'm wondering if we could shade scala 1.13 dependencies inside the Kafka connector? Then we would be independent of the rather big FLINK-20845.
On Tue, Apr 20, 2021 at 5:54 PM Austin Cawley-Edwards <au...@gmail.com> wrote:
Hi Prasanna,
It looks like the Kafka 2.5.0 connector upgrade is tied to dropping support for Scala 2.11. The best place to track that would be the ticket for Scala 2.13 support, FLINK-13414 [1], and its subtask FLINK-20845 [2].
I have listed FLINK-20845 as a blocker for FLINK-19168 for better visibility.
Best,
Austin
[1]: https://issues.apache.org/jira/browse/FLINK-13414
[2]: https://issues.apache.org/jira/browse/FLINK-20845
On Tue, Apr 20, 2021 at 9:08 AM Prasanna kumar <pr...@gmail.com> wrote:
Hi Flinksters,
We are researching about if we could use the latest version of kafka (2.6.1 or 2.7.0)
Since we are using Flink as a processor , we came across this https://issues.apache.org/jira/browse/FLINK-19168.
It says that it does not support version 2.5.0 and beyond.
That was created 8 months back , just checking if there is any effort on that front.
Thanks,
Prasanna
Re: Flink support for Kafka versions
Posted by Prasanna kumar <pr...@gmail.com>.
Hi ,
We are using Kafka broker version 2.4.1.1.
Also kafka client 2.4.1.1 jar which is part of flink kafka connector
recently was marked with high security issue.
So we excluded the dependency and overriden it with kafka client 2.8.1
client jar and it works fine with the 2.4.1.1 broker. ( since its backward
compatible)
1) If we tried to connect to broker with 2.8.1 version (with kafka client
2.8.1 jar override ) would it work or would it throw errors because of
Scala 2.11?
2) https://issues.apache.org/jira/browse/FLINK-20845 is marked as done but
the fix version is marked as 1.15.0.. Wouldn't this change be available for
1.12.x ?
3) https://issues.apache.org/jira/browse/FLINK-14105 is marked as done but
the fix version is marked as 1.14.0.. Wouldn't this change be available for
1.12.x ?
Thanks,
Prasanna.
On Wed, Apr 21, 2021 at 11:03 PM Arvid Heise <ar...@apache.org> wrote:
> I'm wondering if we could shade scala 1.13 dependencies inside the Kafka
> connector? Then we would be independent of the rather big FLINK-20845.
>
> On Tue, Apr 20, 2021 at 5:54 PM Austin Cawley-Edwards <
> austin.cawley@gmail.com> wrote:
>
>> Hi Prasanna,
>>
>> It looks like the Kafka 2.5.0 connector upgrade is tied to dropping
>> support for Scala 2.11. The best place to track that would be the ticket
>> for Scala 2.13 support, FLINK-13414 [1], and its subtask FLINK-20845 [2].
>>
>> I have listed FLINK-20845 as a blocker for FLINK-19168 for better
>> visibility.
>>
>> Best,
>> Austin
>>
>> [1]: https://issues.apache.org/jira/browse/FLINK-13414
>> [2]: https://issues.apache.org/jira/browse/FLINK-20845
>>
>> On Tue, Apr 20, 2021 at 9:08 AM Prasanna kumar <
>> prasannakumarramani@gmail.com> wrote:
>>
>>> Hi Flinksters,
>>>
>>> We are researching about if we could use the latest version of kafka
>>> (2.6.1 or 2.7.0)
>>>
>>> Since we are using Flink as a processor , we came across this
>>> https://issues.apache.org/jira/browse/FLINK-19168.
>>>
>>> It says that it does not support version 2.5.0 and beyond.
>>>
>>> That was created 8 months back , just checking if there is any effort on
>>> that front.
>>>
>>> Thanks,
>>> Prasanna
>>>
>>
Re: Flink support for Kafka versions
Posted by Arvid Heise <ar...@apache.org>.
I'm wondering if we could shade scala 1.13 dependencies inside the Kafka
connector? Then we would be independent of the rather big FLINK-20845.
On Tue, Apr 20, 2021 at 5:54 PM Austin Cawley-Edwards <
austin.cawley@gmail.com> wrote:
> Hi Prasanna,
>
> It looks like the Kafka 2.5.0 connector upgrade is tied to dropping
> support for Scala 2.11. The best place to track that would be the ticket
> for Scala 2.13 support, FLINK-13414 [1], and its subtask FLINK-20845 [2].
>
> I have listed FLINK-20845 as a blocker for FLINK-19168 for better
> visibility.
>
> Best,
> Austin
>
> [1]: https://issues.apache.org/jira/browse/FLINK-13414
> [2]: https://issues.apache.org/jira/browse/FLINK-20845
>
> On Tue, Apr 20, 2021 at 9:08 AM Prasanna kumar <
> prasannakumarramani@gmail.com> wrote:
>
>> Hi Flinksters,
>>
>> We are researching about if we could use the latest version of kafka
>> (2.6.1 or 2.7.0)
>>
>> Since we are using Flink as a processor , we came across this
>> https://issues.apache.org/jira/browse/FLINK-19168.
>>
>> It says that it does not support version 2.5.0 and beyond.
>>
>> That was created 8 months back , just checking if there is any effort on
>> that front.
>>
>> Thanks,
>> Prasanna
>>
>
Re: Flink support for Kafka versions
Posted by Austin Cawley-Edwards <au...@gmail.com>.
Hi Prasanna,
It looks like the Kafka 2.5.0 connector upgrade is tied to dropping support
for Scala 2.11. The best place to track that would be the ticket for Scala
2.13 support, FLINK-13414 [1], and its subtask FLINK-20845 [2].
I have listed FLINK-20845 as a blocker for FLINK-19168 for better
visibility.
Best,
Austin
[1]: https://issues.apache.org/jira/browse/FLINK-13414
[2]: https://issues.apache.org/jira/browse/FLINK-20845
On Tue, Apr 20, 2021 at 9:08 AM Prasanna kumar <
prasannakumarramani@gmail.com> wrote:
> Hi Flinksters,
>
> We are researching about if we could use the latest version of kafka
> (2.6.1 or 2.7.0)
>
> Since we are using Flink as a processor , we came across this
> https://issues.apache.org/jira/browse/FLINK-19168.
>
> It says that it does not support version 2.5.0 and beyond.
>
> That was created 8 months back , just checking if there is any effort on
> that front.
>
> Thanks,
> Prasanna
>