You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Yi Pan <ni...@gmail.com> on 2017/09/01 17:18:02 UTC
Re: Kafka 0.10.2 compatibility with Samza 0.13.1
Hi, XiaoChuan,
Just confirmed with Jiangjie Qin (our in-house Kafka PMC), Kafka client
0.10.1.1 does not work with broker 0.10.0 due to new protocol version
introduced in 0.10.1.1. Hence, if you try to run Samza 0.13.1 against Kafka
broker 0.10.0 version, it won't work. The best option for that seems to be
swapping out the runtime Kafka client 0.10.1.1 version in Samza 0.13.1 to
Kafka client 0.10.0.1 (which was used in Samza 0.12). Since there is no
incompatible API calls to Kafka client lib between Samza 0.12.0 and Samza
0.13.1, that should work and is worth trying out.
Let us know if we can be further assistance here.
Thanks!
-Yi
On Thu, Aug 31, 2017 at 4:14 PM, Yi Pan <ni...@gmail.com> wrote:
> Hi, Xiaochuan,
>
> I am relaying your question to our in-house Kafka expert to confirm. But
> as far as I can tell from the description, the main breaking changes are in
> the KStreams API. Samza only uses the standard Kafka consumer/producer APIs
> now and should not be affected.
>
> I would recommend to give it a try in your question 1.
>
> As for your question 2, are you referring to a) binding a Kafka 0.10.0.0
> client library with Samza 0.13.1? Or b) running Samza 0.13.1 against Kafka
> broker version 0.10.0.0?
>
> If you are referring to a), from the commit history, there is no source
> code level changes when we upgrade the dependency from Kafka 0.10.0.1 to
> 0.10.1.1 in Samza 0.13.1. If you want to make sure that works, you can try
> to downgrade the Kafka version in your build environment to 0.10.0.0 and
> build your app to see whether it breaks the build. If the build passes, the
> client should work w/ Samza 0.13.1, unless Kafka client has an API that
> changes runtime behavior between 0.10.0.1 and 0.10.1.1.
>
> If you are referring to b), it is not guaranteed that the higher version
> of Kafka client library will work with an older version of Kafka broker.
> However, Samza has been very conservative in the usage of new features in
> Kafka client library. Hence, you would have a higher chance of success to
> run Samza 0.13.1 against Kafka 0.10.0 brokers.
>
> Unfortunately, we have not test all the above combinations internally. I
> would wait for the confirmation from our Kafka expert before I can give you
> a definite answer.
>
> Thanks!
>
> -Yi
>
Re: Kafka 0.10.2 compatibility with Samza 0.13.1
Posted by Yi Pan <ni...@gmail.com>.
Hi, XiaoChuan,
Great! So I assume that you were able to upgrade to Samza 0.13.1
successfully? If you need any further help, feel free to send us emails!
Thanks!
-Yi
On Thu, Sep 14, 2017 at 8:11 AM, XiaoChuan Yu <xi...@kik.com> wrote:
> Hi Yi,
>
> Thanks for all the suggestions.
> Our team eventually decided to move off CDH Kafka(and YARN) due to some
> issues not relevant to Samza.
> For what its worth, I did test the a couple basic Hello Samza (Samza
> 0.13.1) jobs against a single node CDH Kafka version
> 0.10.2.0+kafka2.2.0+110 with success.
> The testing was done on AWS EC2 instances running RedHat 7.
> We also found out that running against CDH YARN required various dependency
> overrides such as the ones in Hello Samza's "cdh5" build profile
> <https://github.com/apache/samza-hello-samza/blob/master/pom.xml#L282-L297
> >.
>
> Thanks,
> Xiaochuan Yu
>
> On Fri, Sep 1, 2017 at 1:18 PM Yi Pan <ni...@gmail.com> wrote:
>
> > Hi, XiaoChuan,
> >
> > Just confirmed with Jiangjie Qin (our in-house Kafka PMC), Kafka client
> > 0.10.1.1 does not work with broker 0.10.0 due to new protocol version
> > introduced in 0.10.1.1. Hence, if you try to run Samza 0.13.1 against
> Kafka
> > broker 0.10.0 version, it won't work. The best option for that seems to
> be
> > swapping out the runtime Kafka client 0.10.1.1 version in Samza 0.13.1 to
> > Kafka client 0.10.0.1 (which was used in Samza 0.12). Since there is no
> > incompatible API calls to Kafka client lib between Samza 0.12.0 and Samza
> > 0.13.1, that should work and is worth trying out.
> >
> > Let us know if we can be further assistance here.
> >
> > Thanks!
> >
> > -Yi
> >
> > On Thu, Aug 31, 2017 at 4:14 PM, Yi Pan <ni...@gmail.com> wrote:
> >
> > > Hi, Xiaochuan,
> > >
> > > I am relaying your question to our in-house Kafka expert to confirm.
> But
> > > as far as I can tell from the description, the main breaking changes
> are
> > in
> > > the KStreams API. Samza only uses the standard Kafka consumer/producer
> > APIs
> > > now and should not be affected.
> > >
> > > I would recommend to give it a try in your question 1.
> > >
> > > As for your question 2, are you referring to a) binding a Kafka
> 0.10.0.0
> > > client library with Samza 0.13.1? Or b) running Samza 0.13.1 against
> > Kafka
> > > broker version 0.10.0.0?
> > >
> > > If you are referring to a), from the commit history, there is no source
> > > code level changes when we upgrade the dependency from Kafka 0.10.0.1
> to
> > > 0.10.1.1 in Samza 0.13.1. If you want to make sure that works, you can
> > try
> > > to downgrade the Kafka version in your build environment to 0.10.0.0
> and
> > > build your app to see whether it breaks the build. If the build passes,
> > the
> > > client should work w/ Samza 0.13.1, unless Kafka client has an API that
> > > changes runtime behavior between 0.10.0.1 and 0.10.1.1.
> > >
> > > If you are referring to b), it is not guaranteed that the higher
> version
> > > of Kafka client library will work with an older version of Kafka
> broker.
> > > However, Samza has been very conservative in the usage of new features
> in
> > > Kafka client library. Hence, you would have a higher chance of success
> to
> > > run Samza 0.13.1 against Kafka 0.10.0 brokers.
> > >
> > > Unfortunately, we have not test all the above combinations internally.
> I
> > > would wait for the confirmation from our Kafka expert before I can give
> > you
> > > a definite answer.
> > >
> > > Thanks!
> > >
> > > -Yi
> > >
> >
>
Re: Kafka 0.10.2 compatibility with Samza 0.13.1
Posted by XiaoChuan Yu <xi...@kik.com>.
Hi Yi,
Thanks for all the suggestions.
Our team eventually decided to move off CDH Kafka(and YARN) due to some
issues not relevant to Samza.
For what its worth, I did test the a couple basic Hello Samza (Samza
0.13.1) jobs against a single node CDH Kafka version
0.10.2.0+kafka2.2.0+110 with success.
The testing was done on AWS EC2 instances running RedHat 7.
We also found out that running against CDH YARN required various dependency
overrides such as the ones in Hello Samza's "cdh5" build profile
<https://github.com/apache/samza-hello-samza/blob/master/pom.xml#L282-L297>.
Thanks,
Xiaochuan Yu
On Fri, Sep 1, 2017 at 1:18 PM Yi Pan <ni...@gmail.com> wrote:
> Hi, XiaoChuan,
>
> Just confirmed with Jiangjie Qin (our in-house Kafka PMC), Kafka client
> 0.10.1.1 does not work with broker 0.10.0 due to new protocol version
> introduced in 0.10.1.1. Hence, if you try to run Samza 0.13.1 against Kafka
> broker 0.10.0 version, it won't work. The best option for that seems to be
> swapping out the runtime Kafka client 0.10.1.1 version in Samza 0.13.1 to
> Kafka client 0.10.0.1 (which was used in Samza 0.12). Since there is no
> incompatible API calls to Kafka client lib between Samza 0.12.0 and Samza
> 0.13.1, that should work and is worth trying out.
>
> Let us know if we can be further assistance here.
>
> Thanks!
>
> -Yi
>
> On Thu, Aug 31, 2017 at 4:14 PM, Yi Pan <ni...@gmail.com> wrote:
>
> > Hi, Xiaochuan,
> >
> > I am relaying your question to our in-house Kafka expert to confirm. But
> > as far as I can tell from the description, the main breaking changes are
> in
> > the KStreams API. Samza only uses the standard Kafka consumer/producer
> APIs
> > now and should not be affected.
> >
> > I would recommend to give it a try in your question 1.
> >
> > As for your question 2, are you referring to a) binding a Kafka 0.10.0.0
> > client library with Samza 0.13.1? Or b) running Samza 0.13.1 against
> Kafka
> > broker version 0.10.0.0?
> >
> > If you are referring to a), from the commit history, there is no source
> > code level changes when we upgrade the dependency from Kafka 0.10.0.1 to
> > 0.10.1.1 in Samza 0.13.1. If you want to make sure that works, you can
> try
> > to downgrade the Kafka version in your build environment to 0.10.0.0 and
> > build your app to see whether it breaks the build. If the build passes,
> the
> > client should work w/ Samza 0.13.1, unless Kafka client has an API that
> > changes runtime behavior between 0.10.0.1 and 0.10.1.1.
> >
> > If you are referring to b), it is not guaranteed that the higher version
> > of Kafka client library will work with an older version of Kafka broker.
> > However, Samza has been very conservative in the usage of new features in
> > Kafka client library. Hence, you would have a higher chance of success to
> > run Samza 0.13.1 against Kafka 0.10.0 brokers.
> >
> > Unfortunately, we have not test all the above combinations internally. I
> > would wait for the confirmation from our Kafka expert before I can give
> you
> > a definite answer.
> >
> > Thanks!
> >
> > -Yi
> >
>