You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@edgent.apache.org by vino yang <ya...@gmail.com> on 2018/03/04 07:10:05 UTC
[discuss] What about splitting the kafka Connector into kafka 0.8 and 0.9?
The reason is that Kafka 0.9+ provided a new consumer API which has more
features and better performance.
Just like Flink's implementation :
https://github.com/apache/flink/tree/master/flink-connectors.
vinoyang
Thanks.
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi Chris,
the version upgrade and the main features which affect the produce/consume
API list below :
- 0.8.2.x -> 0.9+ : support save offset, position by Kafka server
itself, Kerberos, TLS authentication, need not specify zookeeper server
- 0.9.x -> 0.10.x : let consumer control max records, introduced
"max.poll.records" config item
- 0.10.x -> 0.11.x : support producer exactly-once (with
*beginTransaction*/*commitTransaction *API)
- 0.11.x -> 1.x : seems no specific feature which affect the API
Vino yang
Thanks.
2018-03-14 9:27 GMT+08:00 vino yang <ya...@gmail.com>:
> Hi Chris,
>
> No objections about this approach. Good division of the work. I will
> provide the mapping of Kafka version and the specified feature later.
>
> Vino yang
> Thanks.
>
> 2018-03-13 20:11 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
>
>> Well I have implemented something like the Version checking before, so I
>> would opt to take care of that.
>>
>> I would define an Annotation with an optional "from" and "to" version ...
>> you could use that
>> I would need something that provides the version of the server from your
>> side.
>>
>> With this I would then implement an Aspect that intercepts these calls,
>> does the check and eventually throws Exceptions with a message what the
>> minimum or maximum version for a feature would be.
>>
>> I would use a compile-time weaver as this does not add any more
>> dependencies or setup complexity to the construct.
>>
>> Any objections to this approach?
>>
>> Chris
>>
>>
>> Am 13.03.18, 03:06 schrieb "vino yang" <ya...@gmail.com>:
>>
>> Hi Chris,
>>
>> It looks like a good idea. I think to finish this job, we can split
>> it into
>> three sub tasks:
>>
>> - upgrade kafka version to 1.x and test it to match the 0.8.x
>> connector's function and behaivor;
>> - Carding and defining the annotation which contains different
>> kafka
>> version and features
>> - expose the new feature's API to user and check with annotation
>>
>> What's your opinion?
>>
>>
>> 2018-03-12 21:00 GMT+08:00 Christofer Dutz <christofer.dutz@c-ware.de
>> >:
>>
>> > Don't know if this would be an option:
>> >
>> > If we defined and used a Java annotation which defines what
>> Kafka-Version
>> > a feature is available from (or up to which version it is
>> supported) and
>> > then we could do quick checks that compare the current version with
>> the
>> > annotations on the methods we call. I think this type of check
>> should be
>> > quite easy to understand and we wouldn't have to build, maintain,
>> test,
>> > document etc. loads of separate modules.
>> >
>> > Chris
>> >
>> >
>> >
>> > Am 12.03.18, 13:30 schrieb "vino yang" <ya...@gmail.com>:
>> >
>> > Hi Chris,
>> >
>> > OK, Hope for listening someone's opinion.
>> >
>> > Vino yang.
>> >
>> > 2018-03-12 20:23 GMT+08:00 Christofer Dutz <
>> christofer.dutz@c-ware.de
>> > >:
>> >
>> > > Hi Vino,
>> > >
>> > > please don't interpret my opinion as some official project
>> decision.
>> > > For discussions like this I would definitely prefer to hear
>> the
>> > opinions
>> > > of others in the project.
>> > > Perhaps having a new client API and having compatibility
>> layers
>> > inside the
>> > > connector would be another option.
>> > > So per default the compatibility level of the Kafka client
>> lib would
>> > be
>> > > used but a developer could explicitly choose
>> > > older compatibility levels, where we have taken care of the
>> work to
>> > decide
>> > > what works and what doesn't.
>> > >
>> > > Chris
>> > >
>> > >
>> > >
>> > > Am 12.03.18, 13:07 schrieb "vino yang" <yanghua1127@gmail.com
>> >:
>> > >
>> > > Hi Chris,
>> > >
>> > > In some ways, I argee with you. Though kafka API has the
>> > > compatibility. But
>> > >
>> > >
>> > > - old API + higher server version : this mode would
>> miss some
>> > key
>> > > new
>> > > feature.
>> > > - new API + older server version : this mode, users
>> are in a
>> > puzzle
>> > > about which feature they could use and which could
>> not. Also,
>> > new
>> > > API will
>> > > do more logic judgement and something else (which cause
>> > performance
>> > > cost)
>> > > for backward compatibility.
>> > >
>> > > I think it's the main reason that other framework split
>> > different kafka
>> > > connector with versions.
>> > >
>> > > Anyway, I will respect your decision. Can I claim this
>> task about
>> > > upgrading
>> > > the kafka client's version to 1.x?
>> > >
>> > >
>> > > 2018-03-12 16:30 GMT+08:00 Christofer Dutz <
>> > christofer.dutz@c-ware.de
>> > > >:
>> > >
>> > > > Hi Vino,
>> > > >
>> > > > I would rather go a different path. I talked to some
>> Kafka
>> > pros and
>> > > they
>> > > > sort of confirmed my gut-feeling.
>> > > > The greatest changes to Kafka have been in the layers
>> behind
>> > the API
>> > > > itself. The API seems to have been designed with
>> backward
>> > > compatibility in
>> > > > mind.
>> > > > That means you can generally use a newer API with an
>> older
>> > broker as
>> > > well
>> > > > as use a new broker with an older API (This is probably
>> even
>> > the
>> > > safer way
>> > > > around). As soon as you try to do something with the
>> API which
>> > your
>> > > broker
>> > > > doesn't support, you get error messages.
>> > > >
>> > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > Compatibility+Matrix
>> > > >
>> > > > I would rather update the existing connector to a newer
>> Kafka
>> > > version ...
>> > > > 0.8.2.2 is quite old and we should update to a version
>> of at
>> > least
>> > > 0.10.0
>> > > > (I would prefer a 1.x) and stick with that. I doubt
>> many will
>> > be
>> > > using an
>> > > > ancient 0.8.2 version (09.09.2015). And everything
>> starting
>> > with
>> > > 0.10.x
>> > > > should be interchangeable.
>> > > >
>> > > > I wouldn't like to have yet another project maintaining
>> a Zoo
>> > of
>> > > adapters
>> > > > for Kafka.
>> > > >
>> > > > Eventually a Kafka-Streams client would make sense
>> though ...
>> > to
>> > > sort of
>> > > > extend the Edgent streams from the edge to the Kafka
>> cluster.
>> > > >
>> > > > Chris
>> > > >
>> > > >
>> > > >
>> > > > Am 12.03.18, 03:41 schrieb "vino yang" <
>> yanghua1127@gmail.com
>> > >:
>> > > >
>> > > > Hi guys,
>> > > >
>> > > > How about this idea, I think we should support
>> kafka's new
>> > > client API.
>> > > >
>> > > > 2018-03-04 15:10 GMT+08:00 vino yang <
>> > yanghua1127@gmail.com>:
>> > > >
>> > > > > The reason is that Kafka 0.9+ provided a new
>> consumer API
>> > > which has
>> > > > more
>> > > > > features and better performance.
>> > > > >
>> > > > > Just like Flink's implementation :
>> > https://github.com/apache/
>> > > > > flink/tree/master/flink-connectors.
>> > > > >
>> > > > > vinoyang
>> > > > > Thanks.
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>>
>>
>>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi Chris,
No objections about this approach. Good division of the work. I will
provide the mapping of Kafka version and the specified feature later.
Vino yang
Thanks.
2018-03-13 20:11 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Well I have implemented something like the Version checking before, so I
> would opt to take care of that.
>
> I would define an Annotation with an optional "from" and "to" version ...
> you could use that
> I would need something that provides the version of the server from your
> side.
>
> With this I would then implement an Aspect that intercepts these calls,
> does the check and eventually throws Exceptions with a message what the
> minimum or maximum version for a feature would be.
>
> I would use a compile-time weaver as this does not add any more
> dependencies or setup complexity to the construct.
>
> Any objections to this approach?
>
> Chris
>
>
> Am 13.03.18, 03:06 schrieb "vino yang" <ya...@gmail.com>:
>
> Hi Chris,
>
> It looks like a good idea. I think to finish this job, we can split it
> into
> three sub tasks:
>
> - upgrade kafka version to 1.x and test it to match the 0.8.x
> connector's function and behaivor;
> - Carding and defining the annotation which contains different kafka
> version and features
> - expose the new feature's API to user and check with annotation
>
> What's your opinion?
>
>
> 2018-03-12 21:00 GMT+08:00 Christofer Dutz <christofer.dutz@c-ware.de
> >:
>
> > Don't know if this would be an option:
> >
> > If we defined and used a Java annotation which defines what
> Kafka-Version
> > a feature is available from (or up to which version it is supported)
> and
> > then we could do quick checks that compare the current version with
> the
> > annotations on the methods we call. I think this type of check
> should be
> > quite easy to understand and we wouldn't have to build, maintain,
> test,
> > document etc. loads of separate modules.
> >
> > Chris
> >
> >
> >
> > Am 12.03.18, 13:30 schrieb "vino yang" <ya...@gmail.com>:
> >
> > Hi Chris,
> >
> > OK, Hope for listening someone's opinion.
> >
> > Vino yang.
> >
> > 2018-03-12 20:23 GMT+08:00 Christofer Dutz <
> christofer.dutz@c-ware.de
> > >:
> >
> > > Hi Vino,
> > >
> > > please don't interpret my opinion as some official project
> decision.
> > > For discussions like this I would definitely prefer to hear the
> > opinions
> > > of others in the project.
> > > Perhaps having a new client API and having compatibility layers
> > inside the
> > > connector would be another option.
> > > So per default the compatibility level of the Kafka client lib
> would
> > be
> > > used but a developer could explicitly choose
> > > older compatibility levels, where we have taken care of the
> work to
> > decide
> > > what works and what doesn't.
> > >
> > > Chris
> > >
> > >
> > >
> > > Am 12.03.18, 13:07 schrieb "vino yang" <yanghua1127@gmail.com
> >:
> > >
> > > Hi Chris,
> > >
> > > In some ways, I argee with you. Though kafka API has the
> > > compatibility. But
> > >
> > >
> > > - old API + higher server version : this mode would
> miss some
> > key
> > > new
> > > feature.
> > > - new API + older server version : this mode, users are
> in a
> > puzzle
> > > about which feature they could use and which could not.
> Also,
> > new
> > > API will
> > > do more logic judgement and something else (which cause
> > performance
> > > cost)
> > > for backward compatibility.
> > >
> > > I think it's the main reason that other framework split
> > different kafka
> > > connector with versions.
> > >
> > > Anyway, I will respect your decision. Can I claim this
> task about
> > > upgrading
> > > the kafka client's version to 1.x?
> > >
> > >
> > > 2018-03-12 16:30 GMT+08:00 Christofer Dutz <
> > christofer.dutz@c-ware.de
> > > >:
> > >
> > > > Hi Vino,
> > > >
> > > > I would rather go a different path. I talked to some
> Kafka
> > pros and
> > > they
> > > > sort of confirmed my gut-feeling.
> > > > The greatest changes to Kafka have been in the layers
> behind
> > the API
> > > > itself. The API seems to have been designed with backward
> > > compatibility in
> > > > mind.
> > > > That means you can generally use a newer API with an
> older
> > broker as
> > > well
> > > > as use a new broker with an older API (This is probably
> even
> > the
> > > safer way
> > > > around). As soon as you try to do something with the API
> which
> > your
> > > broker
> > > > doesn't support, you get error messages.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > Compatibility+Matrix
> > > >
> > > > I would rather update the existing connector to a newer
> Kafka
> > > version ...
> > > > 0.8.2.2 is quite old and we should update to a version
> of at
> > least
> > > 0.10.0
> > > > (I would prefer a 1.x) and stick with that. I doubt many
> will
> > be
> > > using an
> > > > ancient 0.8.2 version (09.09.2015). And everything
> starting
> > with
> > > 0.10.x
> > > > should be interchangeable.
> > > >
> > > > I wouldn't like to have yet another project maintaining
> a Zoo
> > of
> > > adapters
> > > > for Kafka.
> > > >
> > > > Eventually a Kafka-Streams client would make sense
> though ...
> > to
> > > sort of
> > > > extend the Edgent streams from the edge to the Kafka
> cluster.
> > > >
> > > > Chris
> > > >
> > > >
> > > >
> > > > Am 12.03.18, 03:41 schrieb "vino yang" <
> yanghua1127@gmail.com
> > >:
> > > >
> > > > Hi guys,
> > > >
> > > > How about this idea, I think we should support
> kafka's new
> > > client API.
> > > >
> > > > 2018-03-04 15:10 GMT+08:00 vino yang <
> > yanghua1127@gmail.com>:
> > > >
> > > > > The reason is that Kafka 0.9+ provided a new
> consumer API
> > > which has
> > > > more
> > > > > features and better performance.
> > > > >
> > > > > Just like Flink's implementation :
> > https://github.com/apache/
> > > > > flink/tree/master/flink-connectors.
> > > > >
> > > > > vinoyang
> > > > > Thanks.
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Well I have implemented something like the Version checking before, so I would opt to take care of that.
I would define an Annotation with an optional "from" and "to" version ... you could use that
I would need something that provides the version of the server from your side.
With this I would then implement an Aspect that intercepts these calls, does the check and eventually throws Exceptions with a message what the minimum or maximum version for a feature would be.
I would use a compile-time weaver as this does not add any more dependencies or setup complexity to the construct.
Any objections to this approach?
Chris
Am 13.03.18, 03:06 schrieb "vino yang" <ya...@gmail.com>:
Hi Chris,
It looks like a good idea. I think to finish this job, we can split it into
three sub tasks:
- upgrade kafka version to 1.x and test it to match the 0.8.x
connector's function and behaivor;
- Carding and defining the annotation which contains different kafka
version and features
- expose the new feature's API to user and check with annotation
What's your opinion?
2018-03-12 21:00 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Don't know if this would be an option:
>
> If we defined and used a Java annotation which defines what Kafka-Version
> a feature is available from (or up to which version it is supported) and
> then we could do quick checks that compare the current version with the
> annotations on the methods we call. I think this type of check should be
> quite easy to understand and we wouldn't have to build, maintain, test,
> document etc. loads of separate modules.
>
> Chris
>
>
>
> Am 12.03.18, 13:30 schrieb "vino yang" <ya...@gmail.com>:
>
> Hi Chris,
>
> OK, Hope for listening someone's opinion.
>
> Vino yang.
>
> 2018-03-12 20:23 GMT+08:00 Christofer Dutz <christofer.dutz@c-ware.de
> >:
>
> > Hi Vino,
> >
> > please don't interpret my opinion as some official project decision.
> > For discussions like this I would definitely prefer to hear the
> opinions
> > of others in the project.
> > Perhaps having a new client API and having compatibility layers
> inside the
> > connector would be another option.
> > So per default the compatibility level of the Kafka client lib would
> be
> > used but a developer could explicitly choose
> > older compatibility levels, where we have taken care of the work to
> decide
> > what works and what doesn't.
> >
> > Chris
> >
> >
> >
> > Am 12.03.18, 13:07 schrieb "vino yang" <ya...@gmail.com>:
> >
> > Hi Chris,
> >
> > In some ways, I argee with you. Though kafka API has the
> > compatibility. But
> >
> >
> > - old API + higher server version : this mode would miss some
> key
> > new
> > feature.
> > - new API + older server version : this mode, users are in a
> puzzle
> > about which feature they could use and which could not. Also,
> new
> > API will
> > do more logic judgement and something else (which cause
> performance
> > cost)
> > for backward compatibility.
> >
> > I think it's the main reason that other framework split
> different kafka
> > connector with versions.
> >
> > Anyway, I will respect your decision. Can I claim this task about
> > upgrading
> > the kafka client's version to 1.x?
> >
> >
> > 2018-03-12 16:30 GMT+08:00 Christofer Dutz <
> christofer.dutz@c-ware.de
> > >:
> >
> > > Hi Vino,
> > >
> > > I would rather go a different path. I talked to some Kafka
> pros and
> > they
> > > sort of confirmed my gut-feeling.
> > > The greatest changes to Kafka have been in the layers behind
> the API
> > > itself. The API seems to have been designed with backward
> > compatibility in
> > > mind.
> > > That means you can generally use a newer API with an older
> broker as
> > well
> > > as use a new broker with an older API (This is probably even
> the
> > safer way
> > > around). As soon as you try to do something with the API which
> your
> > broker
> > > doesn't support, you get error messages.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > Compatibility+Matrix
> > >
> > > I would rather update the existing connector to a newer Kafka
> > version ...
> > > 0.8.2.2 is quite old and we should update to a version of at
> least
> > 0.10.0
> > > (I would prefer a 1.x) and stick with that. I doubt many will
> be
> > using an
> > > ancient 0.8.2 version (09.09.2015). And everything starting
> with
> > 0.10.x
> > > should be interchangeable.
> > >
> > > I wouldn't like to have yet another project maintaining a Zoo
> of
> > adapters
> > > for Kafka.
> > >
> > > Eventually a Kafka-Streams client would make sense though ...
> to
> > sort of
> > > extend the Edgent streams from the edge to the Kafka cluster.
> > >
> > > Chris
> > >
> > >
> > >
> > > Am 12.03.18, 03:41 schrieb "vino yang" <yanghua1127@gmail.com
> >:
> > >
> > > Hi guys,
> > >
> > > How about this idea, I think we should support kafka's new
> > client API.
> > >
> > > 2018-03-04 15:10 GMT+08:00 vino yang <
> yanghua1127@gmail.com>:
> > >
> > > > The reason is that Kafka 0.9+ provided a new consumer API
> > which has
> > > more
> > > > features and better performance.
> > > >
> > > > Just like Flink's implementation :
> https://github.com/apache/
> > > > flink/tree/master/flink-connectors.
> > > >
> > > > vinoyang
> > > > Thanks.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi Chris,
It looks like a good idea. I think to finish this job, we can split it into
three sub tasks:
- upgrade kafka version to 1.x and test it to match the 0.8.x
connector's function and behaivor;
- Carding and defining the annotation which contains different kafka
version and features
- expose the new feature's API to user and check with annotation
What's your opinion?
2018-03-12 21:00 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Don't know if this would be an option:
>
> If we defined and used a Java annotation which defines what Kafka-Version
> a feature is available from (or up to which version it is supported) and
> then we could do quick checks that compare the current version with the
> annotations on the methods we call. I think this type of check should be
> quite easy to understand and we wouldn't have to build, maintain, test,
> document etc. loads of separate modules.
>
> Chris
>
>
>
> Am 12.03.18, 13:30 schrieb "vino yang" <ya...@gmail.com>:
>
> Hi Chris,
>
> OK, Hope for listening someone's opinion.
>
> Vino yang.
>
> 2018-03-12 20:23 GMT+08:00 Christofer Dutz <christofer.dutz@c-ware.de
> >:
>
> > Hi Vino,
> >
> > please don't interpret my opinion as some official project decision.
> > For discussions like this I would definitely prefer to hear the
> opinions
> > of others in the project.
> > Perhaps having a new client API and having compatibility layers
> inside the
> > connector would be another option.
> > So per default the compatibility level of the Kafka client lib would
> be
> > used but a developer could explicitly choose
> > older compatibility levels, where we have taken care of the work to
> decide
> > what works and what doesn't.
> >
> > Chris
> >
> >
> >
> > Am 12.03.18, 13:07 schrieb "vino yang" <ya...@gmail.com>:
> >
> > Hi Chris,
> >
> > In some ways, I argee with you. Though kafka API has the
> > compatibility. But
> >
> >
> > - old API + higher server version : this mode would miss some
> key
> > new
> > feature.
> > - new API + older server version : this mode, users are in a
> puzzle
> > about which feature they could use and which could not. Also,
> new
> > API will
> > do more logic judgement and something else (which cause
> performance
> > cost)
> > for backward compatibility.
> >
> > I think it's the main reason that other framework split
> different kafka
> > connector with versions.
> >
> > Anyway, I will respect your decision. Can I claim this task about
> > upgrading
> > the kafka client's version to 1.x?
> >
> >
> > 2018-03-12 16:30 GMT+08:00 Christofer Dutz <
> christofer.dutz@c-ware.de
> > >:
> >
> > > Hi Vino,
> > >
> > > I would rather go a different path. I talked to some Kafka
> pros and
> > they
> > > sort of confirmed my gut-feeling.
> > > The greatest changes to Kafka have been in the layers behind
> the API
> > > itself. The API seems to have been designed with backward
> > compatibility in
> > > mind.
> > > That means you can generally use a newer API with an older
> broker as
> > well
> > > as use a new broker with an older API (This is probably even
> the
> > safer way
> > > around). As soon as you try to do something with the API which
> your
> > broker
> > > doesn't support, you get error messages.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > Compatibility+Matrix
> > >
> > > I would rather update the existing connector to a newer Kafka
> > version ...
> > > 0.8.2.2 is quite old and we should update to a version of at
> least
> > 0.10.0
> > > (I would prefer a 1.x) and stick with that. I doubt many will
> be
> > using an
> > > ancient 0.8.2 version (09.09.2015). And everything starting
> with
> > 0.10.x
> > > should be interchangeable.
> > >
> > > I wouldn't like to have yet another project maintaining a Zoo
> of
> > adapters
> > > for Kafka.
> > >
> > > Eventually a Kafka-Streams client would make sense though ...
> to
> > sort of
> > > extend the Edgent streams from the edge to the Kafka cluster.
> > >
> > > Chris
> > >
> > >
> > >
> > > Am 12.03.18, 03:41 schrieb "vino yang" <yanghua1127@gmail.com
> >:
> > >
> > > Hi guys,
> > >
> > > How about this idea, I think we should support kafka's new
> > client API.
> > >
> > > 2018-03-04 15:10 GMT+08:00 vino yang <
> yanghua1127@gmail.com>:
> > >
> > > > The reason is that Kafka 0.9+ provided a new consumer API
> > which has
> > > more
> > > > features and better performance.
> > > >
> > > > Just like Flink's implementation :
> https://github.com/apache/
> > > > flink/tree/master/flink-connectors.
> > > >
> > > > vinoyang
> > > > Thanks.
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Don't know if this would be an option:
If we defined and used a Java annotation which defines what Kafka-Version a feature is available from (or up to which version it is supported) and then we could do quick checks that compare the current version with the annotations on the methods we call. I think this type of check should be quite easy to understand and we wouldn't have to build, maintain, test, document etc. loads of separate modules.
Chris
Am 12.03.18, 13:30 schrieb "vino yang" <ya...@gmail.com>:
Hi Chris,
OK, Hope for listening someone's opinion.
Vino yang.
2018-03-12 20:23 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Hi Vino,
>
> please don't interpret my opinion as some official project decision.
> For discussions like this I would definitely prefer to hear the opinions
> of others in the project.
> Perhaps having a new client API and having compatibility layers inside the
> connector would be another option.
> So per default the compatibility level of the Kafka client lib would be
> used but a developer could explicitly choose
> older compatibility levels, where we have taken care of the work to decide
> what works and what doesn't.
>
> Chris
>
>
>
> Am 12.03.18, 13:07 schrieb "vino yang" <ya...@gmail.com>:
>
> Hi Chris,
>
> In some ways, I argee with you. Though kafka API has the
> compatibility. But
>
>
> - old API + higher server version : this mode would miss some key
> new
> feature.
> - new API + older server version : this mode, users are in a puzzle
> about which feature they could use and which could not. Also, new
> API will
> do more logic judgement and something else (which cause performance
> cost)
> for backward compatibility.
>
> I think it's the main reason that other framework split different kafka
> connector with versions.
>
> Anyway, I will respect your decision. Can I claim this task about
> upgrading
> the kafka client's version to 1.x?
>
>
> 2018-03-12 16:30 GMT+08:00 Christofer Dutz <christofer.dutz@c-ware.de
> >:
>
> > Hi Vino,
> >
> > I would rather go a different path. I talked to some Kafka pros and
> they
> > sort of confirmed my gut-feeling.
> > The greatest changes to Kafka have been in the layers behind the API
> > itself. The API seems to have been designed with backward
> compatibility in
> > mind.
> > That means you can generally use a newer API with an older broker as
> well
> > as use a new broker with an older API (This is probably even the
> safer way
> > around). As soon as you try to do something with the API which your
> broker
> > doesn't support, you get error messages.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> Compatibility+Matrix
> >
> > I would rather update the existing connector to a newer Kafka
> version ...
> > 0.8.2.2 is quite old and we should update to a version of at least
> 0.10.0
> > (I would prefer a 1.x) and stick with that. I doubt many will be
> using an
> > ancient 0.8.2 version (09.09.2015). And everything starting with
> 0.10.x
> > should be interchangeable.
> >
> > I wouldn't like to have yet another project maintaining a Zoo of
> adapters
> > for Kafka.
> >
> > Eventually a Kafka-Streams client would make sense though ... to
> sort of
> > extend the Edgent streams from the edge to the Kafka cluster.
> >
> > Chris
> >
> >
> >
> > Am 12.03.18, 03:41 schrieb "vino yang" <ya...@gmail.com>:
> >
> > Hi guys,
> >
> > How about this idea, I think we should support kafka's new
> client API.
> >
> > 2018-03-04 15:10 GMT+08:00 vino yang <ya...@gmail.com>:
> >
> > > The reason is that Kafka 0.9+ provided a new consumer API
> which has
> > more
> > > features and better performance.
> > >
> > > Just like Flink's implementation : https://github.com/apache/
> > > flink/tree/master/flink-connectors.
> > >
> > > vinoyang
> > > Thanks.
> > >
> > >
> >
> >
> >
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi Chris,
OK, Hope for listening someone's opinion.
Vino yang.
2018-03-12 20:23 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Hi Vino,
>
> please don't interpret my opinion as some official project decision.
> For discussions like this I would definitely prefer to hear the opinions
> of others in the project.
> Perhaps having a new client API and having compatibility layers inside the
> connector would be another option.
> So per default the compatibility level of the Kafka client lib would be
> used but a developer could explicitly choose
> older compatibility levels, where we have taken care of the work to decide
> what works and what doesn't.
>
> Chris
>
>
>
> Am 12.03.18, 13:07 schrieb "vino yang" <ya...@gmail.com>:
>
> Hi Chris,
>
> In some ways, I argee with you. Though kafka API has the
> compatibility. But
>
>
> - old API + higher server version : this mode would miss some key
> new
> feature.
> - new API + older server version : this mode, users are in a puzzle
> about which feature they could use and which could not. Also, new
> API will
> do more logic judgement and something else (which cause performance
> cost)
> for backward compatibility.
>
> I think it's the main reason that other framework split different kafka
> connector with versions.
>
> Anyway, I will respect your decision. Can I claim this task about
> upgrading
> the kafka client's version to 1.x?
>
>
> 2018-03-12 16:30 GMT+08:00 Christofer Dutz <christofer.dutz@c-ware.de
> >:
>
> > Hi Vino,
> >
> > I would rather go a different path. I talked to some Kafka pros and
> they
> > sort of confirmed my gut-feeling.
> > The greatest changes to Kafka have been in the layers behind the API
> > itself. The API seems to have been designed with backward
> compatibility in
> > mind.
> > That means you can generally use a newer API with an older broker as
> well
> > as use a new broker with an older API (This is probably even the
> safer way
> > around). As soon as you try to do something with the API which your
> broker
> > doesn't support, you get error messages.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> Compatibility+Matrix
> >
> > I would rather update the existing connector to a newer Kafka
> version ...
> > 0.8.2.2 is quite old and we should update to a version of at least
> 0.10.0
> > (I would prefer a 1.x) and stick with that. I doubt many will be
> using an
> > ancient 0.8.2 version (09.09.2015). And everything starting with
> 0.10.x
> > should be interchangeable.
> >
> > I wouldn't like to have yet another project maintaining a Zoo of
> adapters
> > for Kafka.
> >
> > Eventually a Kafka-Streams client would make sense though ... to
> sort of
> > extend the Edgent streams from the edge to the Kafka cluster.
> >
> > Chris
> >
> >
> >
> > Am 12.03.18, 03:41 schrieb "vino yang" <ya...@gmail.com>:
> >
> > Hi guys,
> >
> > How about this idea, I think we should support kafka's new
> client API.
> >
> > 2018-03-04 15:10 GMT+08:00 vino yang <ya...@gmail.com>:
> >
> > > The reason is that Kafka 0.9+ provided a new consumer API
> which has
> > more
> > > features and better performance.
> > >
> > > Just like Flink's implementation : https://github.com/apache/
> > > flink/tree/master/flink-connectors.
> > >
> > > vinoyang
> > > Thanks.
> > >
> > >
> >
> >
> >
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Vino,
please don't interpret my opinion as some official project decision.
For discussions like this I would definitely prefer to hear the opinions of others in the project.
Perhaps having a new client API and having compatibility layers inside the connector would be another option.
So per default the compatibility level of the Kafka client lib would be used but a developer could explicitly choose
older compatibility levels, where we have taken care of the work to decide what works and what doesn't.
Chris
Am 12.03.18, 13:07 schrieb "vino yang" <ya...@gmail.com>:
Hi Chris,
In some ways, I argee with you. Though kafka API has the compatibility. But
- old API + higher server version : this mode would miss some key new
feature.
- new API + older server version : this mode, users are in a puzzle
about which feature they could use and which could not. Also, new API will
do more logic judgement and something else (which cause performance cost)
for backward compatibility.
I think it's the main reason that other framework split different kafka
connector with versions.
Anyway, I will respect your decision. Can I claim this task about upgrading
the kafka client's version to 1.x?
2018-03-12 16:30 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Hi Vino,
>
> I would rather go a different path. I talked to some Kafka pros and they
> sort of confirmed my gut-feeling.
> The greatest changes to Kafka have been in the layers behind the API
> itself. The API seems to have been designed with backward compatibility in
> mind.
> That means you can generally use a newer API with an older broker as well
> as use a new broker with an older API (This is probably even the safer way
> around). As soon as you try to do something with the API which your broker
> doesn't support, you get error messages.
>
> https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix
>
> I would rather update the existing connector to a newer Kafka version ...
> 0.8.2.2 is quite old and we should update to a version of at least 0.10.0
> (I would prefer a 1.x) and stick with that. I doubt many will be using an
> ancient 0.8.2 version (09.09.2015). And everything starting with 0.10.x
> should be interchangeable.
>
> I wouldn't like to have yet another project maintaining a Zoo of adapters
> for Kafka.
>
> Eventually a Kafka-Streams client would make sense though ... to sort of
> extend the Edgent streams from the edge to the Kafka cluster.
>
> Chris
>
>
>
> Am 12.03.18, 03:41 schrieb "vino yang" <ya...@gmail.com>:
>
> Hi guys,
>
> How about this idea, I think we should support kafka's new client API.
>
> 2018-03-04 15:10 GMT+08:00 vino yang <ya...@gmail.com>:
>
> > The reason is that Kafka 0.9+ provided a new consumer API which has
> more
> > features and better performance.
> >
> > Just like Flink's implementation : https://github.com/apache/
> > flink/tree/master/flink-connectors.
> >
> > vinoyang
> > Thanks.
> >
> >
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi Chris,
In some ways, I argee with you. Though kafka API has the compatibility. But
- old API + higher server version : this mode would miss some key new
feature.
- new API + older server version : this mode, users are in a puzzle
about which feature they could use and which could not. Also, new API will
do more logic judgement and something else (which cause performance cost)
for backward compatibility.
I think it's the main reason that other framework split different kafka
connector with versions.
Anyway, I will respect your decision. Can I claim this task about upgrading
the kafka client's version to 1.x?
2018-03-12 16:30 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Hi Vino,
>
> I would rather go a different path. I talked to some Kafka pros and they
> sort of confirmed my gut-feeling.
> The greatest changes to Kafka have been in the layers behind the API
> itself. The API seems to have been designed with backward compatibility in
> mind.
> That means you can generally use a newer API with an older broker as well
> as use a new broker with an older API (This is probably even the safer way
> around). As soon as you try to do something with the API which your broker
> doesn't support, you get error messages.
>
> https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix
>
> I would rather update the existing connector to a newer Kafka version ...
> 0.8.2.2 is quite old and we should update to a version of at least 0.10.0
> (I would prefer a 1.x) and stick with that. I doubt many will be using an
> ancient 0.8.2 version (09.09.2015). And everything starting with 0.10.x
> should be interchangeable.
>
> I wouldn't like to have yet another project maintaining a Zoo of adapters
> for Kafka.
>
> Eventually a Kafka-Streams client would make sense though ... to sort of
> extend the Edgent streams from the edge to the Kafka cluster.
>
> Chris
>
>
>
> Am 12.03.18, 03:41 schrieb "vino yang" <ya...@gmail.com>:
>
> Hi guys,
>
> How about this idea, I think we should support kafka's new client API.
>
> 2018-03-04 15:10 GMT+08:00 vino yang <ya...@gmail.com>:
>
> > The reason is that Kafka 0.9+ provided a new consumer API which has
> more
> > features and better performance.
> >
> > Just like Flink's implementation : https://github.com/apache/
> > flink/tree/master/flink-connectors.
> >
> > vinoyang
> > Thanks.
> >
> >
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Vino,
I would rather go a different path. I talked to some Kafka pros and they sort of confirmed my gut-feeling.
The greatest changes to Kafka have been in the layers behind the API itself. The API seems to have been designed with backward compatibility in mind.
That means you can generally use a newer API with an older broker as well as use a new broker with an older API (This is probably even the safer way around). As soon as you try to do something with the API which your broker doesn't support, you get error messages.
https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix
I would rather update the existing connector to a newer Kafka version ... 0.8.2.2 is quite old and we should update to a version of at least 0.10.0 (I would prefer a 1.x) and stick with that. I doubt many will be using an ancient 0.8.2 version (09.09.2015). And everything starting with 0.10.x should be interchangeable.
I wouldn't like to have yet another project maintaining a Zoo of adapters for Kafka.
Eventually a Kafka-Streams client would make sense though ... to sort of extend the Edgent streams from the edge to the Kafka cluster.
Chris
Am 12.03.18, 03:41 schrieb "vino yang" <ya...@gmail.com>:
Hi guys,
How about this idea, I think we should support kafka's new client API.
2018-03-04 15:10 GMT+08:00 vino yang <ya...@gmail.com>:
> The reason is that Kafka 0.9+ provided a new consumer API which has more
> features and better performance.
>
> Just like Flink's implementation : https://github.com/apache/
> flink/tree/master/flink-connectors.
>
> vinoyang
> Thanks.
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi guys,
How about this idea, I think we should support kafka's new client API.
2018-03-04 15:10 GMT+08:00 vino yang <ya...@gmail.com>:
> The reason is that Kafka 0.9+ provided a new consumer API which has more
> features and better performance.
>
> Just like Flink's implementation : https://github.com/apache/
> flink/tree/master/flink-connectors.
>
> vinoyang
> Thanks.
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi Vino,
Yeah ... but I did it without an ASF header ... that's why the build was failing for 23 days ( (I am really ashamed about that)
I tried updating the two Kafka dependencies to the 1.1.0 verison (and to Scala 2.12) and that worked without any noticeable problems.
Chris
Am 04.04.18, 13:38 schrieb "vino yang" <ya...@gmail.com>:
Hi Chris,
I rechecked the old mails between you and me. I misunderstand your message.
I thought you will create the annotation. In fact, you have created the
annotation.
I will do this work soon, hold on.
Vino yang.
Thanks.
2018-04-04 19:32 GMT+08:00 vino yang <ya...@gmail.com>:
> Hi Chris,
>
> I have not done this. And I would upgrade it soon.
>
> Vino yang
> Thanks!
>
> 2018-04-04 19:23 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
>
>> Hi,
>>
>> so I updated the libs locally, built and re-ran the example with this
>> version and it now worked without any problems.
>>
>> Chris
>>
>>
>>
>> Am 04.04.18, 12:58 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de
>> >:
>>
>> Hi all,
>>
>> reporting back from my easter holidays :-)
>>
>> Today I had to help a customer with getting a POC working that uses
>> PLC4X and Edgent. Unfortunately it seems that in order to use the kafka
>> connector I can only use 0.x versions of Kafka. When connecting to 1.x
>> versions I get stack-overflows and OutOfMemory errors. I did a quick test
>> with updating the kafaka libs from the ancient 0.8.2.2 to 1.1.0 seemed to
>> not break anything ... I'll do some local tests with an updated Kafka
>> client.
>>
>> @vino yang ... have you been working on adding the Annotations to the
>> client?
>>
>> @all others ... does anyone have objections to updating the kafka
>> client libs to 1.1.0? It shouldn't break anything as it should be backward
>> compatible. As we are currently not using anything above the API level of
>> 0.8.2 there should also not be any Exceptions (I don't know of any removed
>> things, which could be a problem).
>>
>> Chris
>>
>>
>>
>> Am 20.03.18, 10:33 schrieb "Christofer Dutz" <
>> christofer.dutz@c-ware.de>:
>>
>> Ok,
>>
>> So I just added a new Annotation type to the Kafka module.
>>
>> org.apache.edgent.connectors.kafka.annotations.KafkaVersion
>>
>> It has a fromVersion and a toVersion attribute. Both should be
>> optional so just adding the annotation would have no effect (besides a few
>> additional CPU operations). The annotation can be applied to methods or
>> classes (every method then inherits this). I hope that's ok, because
>> implementing this on a parameter Level would make things extremely
>> difficult.
>>
>> @vino yang With this you should be able to provide Kafka version
>> constraints to your code changes. Just tell me if something's missing or
>> needs to be done differently
>>
>> For now this annotation will have no effect as I haven't
>> implemented the Aspect for doing the checks, but I'll start working on that
>> as soon as you have annotated something.
>>
>> Chris
>>
>> Am 20.03.18, 10:11 schrieb "Christofer Dutz" <
>> christofer.dutz@c-ware.de>:
>>
>> Ok ... maybe I should add the Annotation prior to continuing
>> my work on the AWS connector ...
>>
>>
>> Chris
>>
>> Am 04.03.18, 08:10 schrieb "vino yang" <yanghua1127@gmail.com
>> >:
>>
>> The reason is that Kafka 0.9+ provided a new consumer API
>> which has more
>> features and better performance.
>>
>> Just like Flink's implementation :
>> https://github.com/apache/flin
>> k/tree/master/flink-connectors.
>>
>> vinoyang
>> Thanks.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi Chris,
I rechecked the old mails between you and me. I misunderstand your message.
I thought you will create the annotation. In fact, you have created the
annotation.
I will do this work soon, hold on.
Vino yang.
Thanks.
2018-04-04 19:32 GMT+08:00 vino yang <ya...@gmail.com>:
> Hi Chris,
>
> I have not done this. And I would upgrade it soon.
>
> Vino yang
> Thanks!
>
> 2018-04-04 19:23 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
>
>> Hi,
>>
>> so I updated the libs locally, built and re-ran the example with this
>> version and it now worked without any problems.
>>
>> Chris
>>
>>
>>
>> Am 04.04.18, 12:58 schrieb "Christofer Dutz" <christofer.dutz@c-ware.de
>> >:
>>
>> Hi all,
>>
>> reporting back from my easter holidays :-)
>>
>> Today I had to help a customer with getting a POC working that uses
>> PLC4X and Edgent. Unfortunately it seems that in order to use the kafka
>> connector I can only use 0.x versions of Kafka. When connecting to 1.x
>> versions I get stack-overflows and OutOfMemory errors. I did a quick test
>> with updating the kafaka libs from the ancient 0.8.2.2 to 1.1.0 seemed to
>> not break anything ... I'll do some local tests with an updated Kafka
>> client.
>>
>> @vino yang ... have you been working on adding the Annotations to the
>> client?
>>
>> @all others ... does anyone have objections to updating the kafka
>> client libs to 1.1.0? It shouldn't break anything as it should be backward
>> compatible. As we are currently not using anything above the API level of
>> 0.8.2 there should also not be any Exceptions (I don't know of any removed
>> things, which could be a problem).
>>
>> Chris
>>
>>
>>
>> Am 20.03.18, 10:33 schrieb "Christofer Dutz" <
>> christofer.dutz@c-ware.de>:
>>
>> Ok,
>>
>> So I just added a new Annotation type to the Kafka module.
>>
>> org.apache.edgent.connectors.kafka.annotations.KafkaVersion
>>
>> It has a fromVersion and a toVersion attribute. Both should be
>> optional so just adding the annotation would have no effect (besides a few
>> additional CPU operations). The annotation can be applied to methods or
>> classes (every method then inherits this). I hope that's ok, because
>> implementing this on a parameter Level would make things extremely
>> difficult.
>>
>> @vino yang With this you should be able to provide Kafka version
>> constraints to your code changes. Just tell me if something's missing or
>> needs to be done differently
>>
>> For now this annotation will have no effect as I haven't
>> implemented the Aspect for doing the checks, but I'll start working on that
>> as soon as you have annotated something.
>>
>> Chris
>>
>> Am 20.03.18, 10:11 schrieb "Christofer Dutz" <
>> christofer.dutz@c-ware.de>:
>>
>> Ok ... maybe I should add the Annotation prior to continuing
>> my work on the AWS connector ...
>>
>>
>> Chris
>>
>> Am 04.03.18, 08:10 schrieb "vino yang" <yanghua1127@gmail.com
>> >:
>>
>> The reason is that Kafka 0.9+ provided a new consumer API
>> which has more
>> features and better performance.
>>
>> Just like Flink's implementation :
>> https://github.com/apache/flin
>> k/tree/master/flink-connectors.
>>
>> vinoyang
>> Thanks.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi Chris,
I have not done this. And I would upgrade it soon.
Vino yang
Thanks!
2018-04-04 19:23 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Hi,
>
> so I updated the libs locally, built and re-ran the example with this
> version and it now worked without any problems.
>
> Chris
>
>
>
> Am 04.04.18, 12:58 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> Hi all,
>
> reporting back from my easter holidays :-)
>
> Today I had to help a customer with getting a POC working that uses
> PLC4X and Edgent. Unfortunately it seems that in order to use the kafka
> connector I can only use 0.x versions of Kafka. When connecting to 1.x
> versions I get stack-overflows and OutOfMemory errors. I did a quick test
> with updating the kafaka libs from the ancient 0.8.2.2 to 1.1.0 seemed to
> not break anything ... I'll do some local tests with an updated Kafka
> client.
>
> @vino yang ... have you been working on adding the Annotations to the
> client?
>
> @all others ... does anyone have objections to updating the kafka
> client libs to 1.1.0? It shouldn't break anything as it should be backward
> compatible. As we are currently not using anything above the API level of
> 0.8.2 there should also not be any Exceptions (I don't know of any removed
> things, which could be a problem).
>
> Chris
>
>
>
> Am 20.03.18, 10:33 schrieb "Christofer Dutz" <
> christofer.dutz@c-ware.de>:
>
> Ok,
>
> So I just added a new Annotation type to the Kafka module.
>
> org.apache.edgent.connectors.kafka.annotations.KafkaVersion
>
> It has a fromVersion and a toVersion attribute. Both should be
> optional so just adding the annotation would have no effect (besides a few
> additional CPU operations). The annotation can be applied to methods or
> classes (every method then inherits this). I hope that's ok, because
> implementing this on a parameter Level would make things extremely
> difficult.
>
> @vino yang With this you should be able to provide Kafka version
> constraints to your code changes. Just tell me if something's missing or
> needs to be done differently
>
> For now this annotation will have no effect as I haven't
> implemented the Aspect for doing the checks, but I'll start working on that
> as soon as you have annotated something.
>
> Chris
>
> Am 20.03.18, 10:11 schrieb "Christofer Dutz" <
> christofer.dutz@c-ware.de>:
>
> Ok ... maybe I should add the Annotation prior to continuing
> my work on the AWS connector ...
>
>
> Chris
>
> Am 04.03.18, 08:10 schrieb "vino yang" <yanghua1127@gmail.com
> >:
>
> The reason is that Kafka 0.9+ provided a new consumer API
> which has more
> features and better performance.
>
> Just like Flink's implementation :
> https://github.com/apache/flink/tree/master/flink-
> connectors.
>
> vinoyang
> Thanks.
>
>
>
>
>
>
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi,
so I updated the libs locally, built and re-ran the example with this version and it now worked without any problems.
Chris
Am 04.04.18, 12:58 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Hi all,
reporting back from my easter holidays :-)
Today I had to help a customer with getting a POC working that uses PLC4X and Edgent. Unfortunately it seems that in order to use the kafka connector I can only use 0.x versions of Kafka. When connecting to 1.x versions I get stack-overflows and OutOfMemory errors. I did a quick test with updating the kafaka libs from the ancient 0.8.2.2 to 1.1.0 seemed to not break anything ... I'll do some local tests with an updated Kafka client.
@vino yang ... have you been working on adding the Annotations to the client?
@all others ... does anyone have objections to updating the kafka client libs to 1.1.0? It shouldn't break anything as it should be backward compatible. As we are currently not using anything above the API level of 0.8.2 there should also not be any Exceptions (I don't know of any removed things, which could be a problem).
Chris
Am 20.03.18, 10:33 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ok,
So I just added a new Annotation type to the Kafka module.
org.apache.edgent.connectors.kafka.annotations.KafkaVersion
It has a fromVersion and a toVersion attribute. Both should be optional so just adding the annotation would have no effect (besides a few additional CPU operations). The annotation can be applied to methods or classes (every method then inherits this). I hope that's ok, because implementing this on a parameter Level would make things extremely difficult.
@vino yang With this you should be able to provide Kafka version constraints to your code changes. Just tell me if something's missing or needs to be done differently
For now this annotation will have no effect as I haven't implemented the Aspect for doing the checks, but I'll start working on that as soon as you have annotated something.
Chris
Am 20.03.18, 10:11 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ok ... maybe I should add the Annotation prior to continuing my work on the AWS connector ...
Chris
Am 04.03.18, 08:10 schrieb "vino yang" <ya...@gmail.com>:
The reason is that Kafka 0.9+ provided a new consumer API which has more
features and better performance.
Just like Flink's implementation :
https://github.com/apache/flink/tree/master/flink-connectors.
vinoyang
Thanks.
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Hi all,
reporting back from my easter holidays :-)
Today I had to help a customer with getting a POC working that uses PLC4X and Edgent. Unfortunately it seems that in order to use the kafka connector I can only use 0.x versions of Kafka. When connecting to 1.x versions I get stack-overflows and OutOfMemory errors. I did a quick test with updating the kafaka libs from the ancient 0.8.2.2 to 1.1.0 seemed to not break anything ... I'll do some local tests with an updated Kafka client.
@vino yang ... have you been working on adding the Annotations to the client?
@all others ... does anyone have objections to updating the kafka client libs to 1.1.0? It shouldn't break anything as it should be backward compatible. As we are currently not using anything above the API level of 0.8.2 there should also not be any Exceptions (I don't know of any removed things, which could be a problem).
Chris
Am 20.03.18, 10:33 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ok,
So I just added a new Annotation type to the Kafka module.
org.apache.edgent.connectors.kafka.annotations.KafkaVersion
It has a fromVersion and a toVersion attribute. Both should be optional so just adding the annotation would have no effect (besides a few additional CPU operations). The annotation can be applied to methods or classes (every method then inherits this). I hope that's ok, because implementing this on a parameter Level would make things extremely difficult.
@vino yang With this you should be able to provide Kafka version constraints to your code changes. Just tell me if something's missing or needs to be done differently
For now this annotation will have no effect as I haven't implemented the Aspect for doing the checks, but I'll start working on that as soon as you have annotated something.
Chris
Am 20.03.18, 10:11 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ok ... maybe I should add the Annotation prior to continuing my work on the AWS connector ...
Chris
Am 04.03.18, 08:10 schrieb "vino yang" <ya...@gmail.com>:
The reason is that Kafka 0.9+ provided a new consumer API which has more
features and better performance.
Just like Flink's implementation :
https://github.com/apache/flink/tree/master/flink-connectors.
vinoyang
Thanks.
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by vino yang <ya...@gmail.com>.
Hi Chris,
I will try to do this, if I have any question, keep in touch!
Vino yang
Thanks.
2018-03-20 17:32 GMT+08:00 Christofer Dutz <ch...@c-ware.de>:
> Ok,
>
> So I just added a new Annotation type to the Kafka module.
>
> org.apache.edgent.connectors.kafka.annotations.KafkaVersion
>
> It has a fromVersion and a toVersion attribute. Both should be optional so
> just adding the annotation would have no effect (besides a few additional
> CPU operations). The annotation can be applied to methods or classes (every
> method then inherits this). I hope that's ok, because implementing this on
> a parameter Level would make things extremely difficult.
>
> @vino yang With this you should be able to provide Kafka version
> constraints to your code changes. Just tell me if something's missing or
> needs to be done differently
>
> For now this annotation will have no effect as I haven't implemented the
> Aspect for doing the checks, but I'll start working on that as soon as you
> have annotated something.
>
> Chris
>
> Am 20.03.18, 10:11 schrieb "Christofer Dutz" <ch...@c-ware.de>:
>
> Ok ... maybe I should add the Annotation prior to continuing my work
> on the AWS connector ...
>
>
> Chris
>
> Am 04.03.18, 08:10 schrieb "vino yang" <ya...@gmail.com>:
>
> The reason is that Kafka 0.9+ provided a new consumer API which
> has more
> features and better performance.
>
> Just like Flink's implementation :
> https://github.com/apache/flink/tree/master/flink-connectors.
>
> vinoyang
> Thanks.
>
>
>
>
>
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Geeee ...
I just noticed that my last commit broke the build and no one complained for 23 days (
I Just fixed that (hopefully) ...
Chris
Am 20.03.18, 10:33 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ok,
So I just added a new Annotation type to the Kafka module.
org.apache.edgent.connectors.kafka.annotations.KafkaVersion
It has a fromVersion and a toVersion attribute. Both should be optional so just adding the annotation would have no effect (besides a few additional CPU operations). The annotation can be applied to methods or classes (every method then inherits this). I hope that's ok, because implementing this on a parameter Level would make things extremely difficult.
@vino yang With this you should be able to provide Kafka version constraints to your code changes. Just tell me if something's missing or needs to be done differently
For now this annotation will have no effect as I haven't implemented the Aspect for doing the checks, but I'll start working on that as soon as you have annotated something.
Chris
Am 20.03.18, 10:11 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ok ... maybe I should add the Annotation prior to continuing my work on the AWS connector ...
Chris
Am 04.03.18, 08:10 schrieb "vino yang" <ya...@gmail.com>:
The reason is that Kafka 0.9+ provided a new consumer API which has more
features and better performance.
Just like Flink's implementation :
https://github.com/apache/flink/tree/master/flink-connectors.
vinoyang
Thanks.
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Ok,
So I just added a new Annotation type to the Kafka module.
org.apache.edgent.connectors.kafka.annotations.KafkaVersion
It has a fromVersion and a toVersion attribute. Both should be optional so just adding the annotation would have no effect (besides a few additional CPU operations). The annotation can be applied to methods or classes (every method then inherits this). I hope that's ok, because implementing this on a parameter Level would make things extremely difficult.
@vino yang With this you should be able to provide Kafka version constraints to your code changes. Just tell me if something's missing or needs to be done differently
For now this annotation will have no effect as I haven't implemented the Aspect for doing the checks, but I'll start working on that as soon as you have annotated something.
Chris
Am 20.03.18, 10:11 schrieb "Christofer Dutz" <ch...@c-ware.de>:
Ok ... maybe I should add the Annotation prior to continuing my work on the AWS connector ...
Chris
Am 04.03.18, 08:10 schrieb "vino yang" <ya...@gmail.com>:
The reason is that Kafka 0.9+ provided a new consumer API which has more
features and better performance.
Just like Flink's implementation :
https://github.com/apache/flink/tree/master/flink-connectors.
vinoyang
Thanks.
Re: [discuss] What about splitting the kafka Connector into kafka 0.8
and 0.9?
Posted by Christofer Dutz <ch...@c-ware.de>.
Ok ... maybe I should add the Annotation prior to continuing my work on the AWS connector ...
Chris
Am 04.03.18, 08:10 schrieb "vino yang" <ya...@gmail.com>:
The reason is that Kafka 0.9+ provided a new consumer API which has more
features and better performance.
Just like Flink's implementation :
https://github.com/apache/flink/tree/master/flink-connectors.
vinoyang
Thanks.