You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Piotr Szuberski <pi...@polidea.com> on 2020/10/22 08:47:10 UTC
[DISCUSS] Update Kafka dependencies in Beam Java SDK
Should we update Kafka dependencies to the recent ones (Kafka clients to 2.6.0 and Kafka_2.11 to 2.4.1)?
What would have to be done to keep the backwards compatibility and which previous versions would we want to support?
Kafka's backward compatibility is quite good so maybe there wouldn't be anything to do?
Let's vote/discuss.
Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK
Posted by Alexey Romanenko <ar...@gmail.com>.
*raising this question, of course
> On 28 Oct 2020, at 18:06, Alexey Romanenko <ar...@gmail.com> wrote:
>
> tasing this question
Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK
Posted by Alexey Romanenko <ar...@gmail.com>.
> On 30 Oct 2020, at 19:33, Piotr Szuberski <pi...@polidea.com> wrote:
>
> Oh I see, I thought that our tests are testing different Kafka versions right now. There is a test suite for 2.1.0 version in addition to 1.0.0 if I understand that gradle.build file correctly.
Yes and iirc, it was added to test Kafka headers deserialisation. Perhaps, we could follow the similar approach with testing for other versions.
> I don't mean to drop the support to earlier Kafka versions. I just mean to set the recent versions as default and add tests verifying that earlier versions are also still supported.
Until we do test the old major versions, that should work, imo. Though, it would be great to hear other opinions on this because KafkaIO is a very popular IO connector and used by many users in different environments.
>
> On 2020/10/30 16:24:45, Alexey Romanenko <ar...@gmail.com> wrote:
>> We have a Jira task about testing KafkaIO against different Kafka versions [1]. Once it will be implemented, then I think it should solve a potential problem with new Kafka client versions.
>>
>> In the same time, imho, until there are Beam users, that use old Kafka client versions, we need to support them.
>>
>> [1] https://issues.apache.org/jira/browse/BEAM-7003
>>
>>> On 30 Oct 2020, at 10:05, Piotr Szuberski <pi...@polidea.com> wrote:
>>>
>>> In my opinion it would be good to keep Beam's dependencies as close to the recent stable versions as possible and, if needed, keep the support for earlier versions.
>>>
>>> For now we keep the old dependency as the base and test whether it works for some newer versions. That way we may always ignore the dependency check report that will get resolved
>>> at 1.0.0 and miss out that some important change has been released because every dependency check points at the Kafka dependency's deprecation. If we used newer version as the base then the Dependency check will do its job in this case.
>>>
>>> I've run Kafka both local and performance tests using kafka-clients:2.6.0 of Kafka clients and they passed - I'm not sure whether it answers your question about the implementation with the usage of ConsumerSpEL class.
>>>
>>>
>>> On 2020/10/28 17:06:56, Alexey Romanenko <ar...@gmail.com> wrote:
>>>> Piotr, thank you for tasing this question. Let me ask some questions before.
>>>>
>>>> What will give us this dependencies update? What are the pros and cons? Can users use recent versions of Kafka client with current implementation based on ConsumerSpEL class?
>>>>
>>>>
>>>>> On 22 Oct 2020, at 10:47, Piotr Szuberski <pi...@polidea.com> wrote:
>>>>>
>>>>> Should we update Kafka dependencies to the recent ones (Kafka clients to 2.6.0 and Kafka_2.11 to 2.4.1)?
>>>>>
>>>>> What would have to be done to keep the backwards compatibility and which previous versions would we want to support?
>>>>>
>>>>> Kafka's backward compatibility is quite good so maybe there wouldn't be anything to do?
>>>>>
>>>>> Let's vote/discuss.
>>>>
>>>>
>>
>>
Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK
Posted by Piotr Szuberski <pi...@polidea.com>.
Oh I see, I thought that our tests are testing different Kafka versions right now. There is a test suite for 2.1.0 version in addition to 1.0.0 if I understand that gradle.build file correctly.
I don't mean to drop the support to earlier Kafka versions. I just mean to set the recent versions as default and add tests verifying that earlier versions are also still supported.
On 2020/10/30 16:24:45, Alexey Romanenko <ar...@gmail.com> wrote:
> We have a Jira task about testing KafkaIO against different Kafka versions [1]. Once it will be implemented, then I think it should solve a potential problem with new Kafka client versions.
>
> In the same time, imho, until there are Beam users, that use old Kafka client versions, we need to support them.
>
> [1] https://issues.apache.org/jira/browse/BEAM-7003
>
> > On 30 Oct 2020, at 10:05, Piotr Szuberski <pi...@polidea.com> wrote:
> >
> > In my opinion it would be good to keep Beam's dependencies as close to the recent stable versions as possible and, if needed, keep the support for earlier versions.
> >
> > For now we keep the old dependency as the base and test whether it works for some newer versions. That way we may always ignore the dependency check report that will get resolved
> > at 1.0.0 and miss out that some important change has been released because every dependency check points at the Kafka dependency's deprecation. If we used newer version as the base then the Dependency check will do its job in this case.
> >
> > I've run Kafka both local and performance tests using kafka-clients:2.6.0 of Kafka clients and they passed - I'm not sure whether it answers your question about the implementation with the usage of ConsumerSpEL class.
> >
> >
> > On 2020/10/28 17:06:56, Alexey Romanenko <ar...@gmail.com> wrote:
> >> Piotr, thank you for tasing this question. Let me ask some questions before.
> >>
> >> What will give us this dependencies update? What are the pros and cons? Can users use recent versions of Kafka client with current implementation based on ConsumerSpEL class?
> >>
> >>
> >>> On 22 Oct 2020, at 10:47, Piotr Szuberski <pi...@polidea.com> wrote:
> >>>
> >>> Should we update Kafka dependencies to the recent ones (Kafka clients to 2.6.0 and Kafka_2.11 to 2.4.1)?
> >>>
> >>> What would have to be done to keep the backwards compatibility and which previous versions would we want to support?
> >>>
> >>> Kafka's backward compatibility is quite good so maybe there wouldn't be anything to do?
> >>>
> >>> Let's vote/discuss.
> >>
> >>
>
>
Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK
Posted by Alexey Romanenko <ar...@gmail.com>.
We have a Jira task about testing KafkaIO against different Kafka versions [1]. Once it will be implemented, then I think it should solve a potential problem with new Kafka client versions.
In the same time, imho, until there are Beam users, that use old Kafka client versions, we need to support them.
[1] https://issues.apache.org/jira/browse/BEAM-7003
> On 30 Oct 2020, at 10:05, Piotr Szuberski <pi...@polidea.com> wrote:
>
> In my opinion it would be good to keep Beam's dependencies as close to the recent stable versions as possible and, if needed, keep the support for earlier versions.
>
> For now we keep the old dependency as the base and test whether it works for some newer versions. That way we may always ignore the dependency check report that will get resolved
> at 1.0.0 and miss out that some important change has been released because every dependency check points at the Kafka dependency's deprecation. If we used newer version as the base then the Dependency check will do its job in this case.
>
> I've run Kafka both local and performance tests using kafka-clients:2.6.0 of Kafka clients and they passed - I'm not sure whether it answers your question about the implementation with the usage of ConsumerSpEL class.
>
>
> On 2020/10/28 17:06:56, Alexey Romanenko <ar...@gmail.com> wrote:
>> Piotr, thank you for tasing this question. Let me ask some questions before.
>>
>> What will give us this dependencies update? What are the pros and cons? Can users use recent versions of Kafka client with current implementation based on ConsumerSpEL class?
>>
>>
>>> On 22 Oct 2020, at 10:47, Piotr Szuberski <pi...@polidea.com> wrote:
>>>
>>> Should we update Kafka dependencies to the recent ones (Kafka clients to 2.6.0 and Kafka_2.11 to 2.4.1)?
>>>
>>> What would have to be done to keep the backwards compatibility and which previous versions would we want to support?
>>>
>>> Kafka's backward compatibility is quite good so maybe there wouldn't be anything to do?
>>>
>>> Let's vote/discuss.
>>
>>
Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK
Posted by Piotr Szuberski <pi...@polidea.com>.
In my opinion it would be good to keep Beam's dependencies as close to the recent stable versions as possible and, if needed, keep the support for earlier versions.
For now we keep the old dependency as the base and test whether it works for some newer versions. That way we may always ignore the dependency check report that will get resolved
at 1.0.0 and miss out that some important change has been released because every dependency check points at the Kafka dependency's deprecation. If we used newer version as the base then the Dependency check will do its job in this case.
I've run Kafka both local and performance tests using kafka-clients:2.6.0 of Kafka clients and they passed - I'm not sure whether it answers your question about the implementation with the usage of ConsumerSpEL class.
On 2020/10/28 17:06:56, Alexey Romanenko <ar...@gmail.com> wrote:
> Piotr, thank you for tasing this question. Let me ask some questions before.
>
> What will give us this dependencies update? What are the pros and cons? Can users use recent versions of Kafka client with current implementation based on ConsumerSpEL class?
>
>
> > On 22 Oct 2020, at 10:47, Piotr Szuberski <pi...@polidea.com> wrote:
> >
> > Should we update Kafka dependencies to the recent ones (Kafka clients to 2.6.0 and Kafka_2.11 to 2.4.1)?
> >
> > What would have to be done to keep the backwards compatibility and which previous versions would we want to support?
> >
> > Kafka's backward compatibility is quite good so maybe there wouldn't be anything to do?
> >
> > Let's vote/discuss.
>
>
Re: [DISCUSS] Update Kafka dependencies in Beam Java SDK
Posted by Alexey Romanenko <ar...@gmail.com>.
Piotr, thank you for tasing this question. Let me ask some questions before.
What will give us this dependencies update? What are the pros and cons? Can users use recent versions of Kafka client with current implementation based on ConsumerSpEL class?
> On 22 Oct 2020, at 10:47, Piotr Szuberski <pi...@polidea.com> wrote:
>
> Should we update Kafka dependencies to the recent ones (Kafka clients to 2.6.0 and Kafka_2.11 to 2.4.1)?
>
> What would have to be done to keep the backwards compatibility and which previous versions would we want to support?
>
> Kafka's backward compatibility is quite good so maybe there wouldn't be anything to do?
>
> Let's vote/discuss.