You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Dana Powers <da...@gmail.com> on 2015/12/24 01:11:41 UTC
The JIRA Awakens [KAFKA-1841]
Hi all,
I've been helping debug an issue filed against kafka-python related to
compatibility w/ Hortonworks 2.3.0.0 kafka release. As I understand it, HDP
is currently based on snapshots of apache/kafka trunk, merged with some
custom patches from HDP itself.
In this case, HDP's 2.3.0.0 kafka release missed a compatibility patch that
I believe is critical for third-party library support. Unfortunately the
patch -- KAFKA-1841 -- was initially only applied to the 0.8.2 branch (it
was merged to trunk several months later in KAFKA-2068). Because it wasn't
on trunk, it didn't get included in the HDP kafka releases.
If you recall, KAFKA-1841 was needed to maintain backwards and forwards
compatibility wrt the change from zookeeper to kafka-backed offset storage.
Not having this patch is fine if you only ever use the clients / libraries
distributed in the that release -- and I imagine that is probably most
folks that are using it. But if you remember the thread on this issue back
in the 0.8.2-beta review, the API incompatibility made third-party clients
hard to develop and maintain if the goal is to support multiple broker
versions w/ the same client code [this is the goal of kafka-python].
Anyways, I'm really glad that the fix made it into the apache release, but
now I'm sad that it didn't make it into HDP's release.
Anyways, I think there's a couple takeaways here:
(1) I'd recommend anyone using HDP who intends to use third-party kafka
consumers should upgrade to 2.3.4.0 or later. That version appears to
include the compatibility patch (KAFKA-2068). Of course if anyone is on
list from HDP, they may be able to provide better help on this.
(2) I think more care should probably be taken to help vendors or anyone
tracking changes on trunk wrt released versions. Is there a list of all
KAFKA-XXXX patches that are released but not merged into trunk ? KAFKA-1841
is obviously near and dear to my heart, but I wonder if there are other
patches like it?
Happy holidays to all, and may the force be with you
-Dana
Re: [kafka-clients] The JIRA Awakens [KAFKA-1841]
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Dana,
Not sure about the old merge script, but with the new one used for GitHub
PRs it tracks the branches you choose to commit to and can update the JIRA
automatically, tagging the appropriate fix versions. So from now on, the
appropriate JIRA queries for issues with, e.g., fix version 0.9.0.1 but not
0.9.1.0, should give you the results you're looking for (although I would
think that committing changes only to a release branch and not to trunk
should be very unusual...).
-Ewen
On Wed, Dec 23, 2015 at 4:11 PM, Dana Powers <da...@gmail.com> wrote:
> Hi all,
>
> I've been helping debug an issue filed against kafka-python related to
> compatibility w/ Hortonworks 2.3.0.0 kafka release. As I understand it, HDP
> is currently based on snapshots of apache/kafka trunk, merged with some
> custom patches from HDP itself.
>
> In this case, HDP's 2.3.0.0 kafka release missed a compatibility patch
> that I believe is critical for third-party library support. Unfortunately
> the patch -- KAFKA-1841 -- was initially only applied to the 0.8.2 branch
> (it was merged to trunk several months later in KAFKA-2068). Because it
> wasn't on trunk, it didn't get included in the HDP kafka releases.
>
> If you recall, KAFKA-1841 was needed to maintain backwards and forwards
> compatibility wrt the change from zookeeper to kafka-backed offset storage.
> Not having this patch is fine if you only ever use the clients / libraries
> distributed in the that release -- and I imagine that is probably most
> folks that are using it. But if you remember the thread on this issue back
> in the 0.8.2-beta review, the API incompatibility made third-party clients
> hard to develop and maintain if the goal is to support multiple broker
> versions w/ the same client code [this is the goal of kafka-python].
> Anyways, I'm really glad that the fix made it into the apache release, but
> now I'm sad that it didn't make it into HDP's release.
>
> Anyways, I think there's a couple takeaways here:
>
> (1) I'd recommend anyone using HDP who intends to use third-party kafka
> consumers should upgrade to 2.3.4.0 or later. That version appears to
> include the compatibility patch (KAFKA-2068). Of course if anyone is on
> list from HDP, they may be able to provide better help on this.
>
> (2) I think more care should probably be taken to help vendors or anyone
> tracking changes on trunk wrt released versions. Is there a list of all
> KAFKA-XXXX patches that are released but not merged into trunk ? KAFKA-1841
> is obviously near and dear to my heart, but I wonder if there are other
> patches like it?
>
> Happy holidays to all, and may the force be with you
>
> -Dana
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscribe@googlegroups.com.
> To post to this group, send email to kafka-clients@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAAndJ_yarA%2BpTRLC1%3DdY3KtYY0esa6z9U8%3DM7k1WPwKB%2BtBNMQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/kafka-clients/CAAndJ_yarA%2BpTRLC1%3DdY3KtYY0esa6z9U8%3DM7k1WPwKB%2BtBNMQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
Thanks,
Ewen
Re: The JIRA Awakens [KAFKA-1841]
Posted by Dana Powers <da...@gmail.com>.
Awesome ! Looks like a great distribution. I'd love to add HDP releases to
the kafka-python integration test suite.
-Dana
On Dec 24, 2015 9:33 AM, "Harsha" <ka...@harsha.io> wrote:
> HI Dana,
> I worked on that release. Yes HDP-2.3.0 has lot of
> additional patches on top of 0.8.2.1 mainly the kerberos
> patches.
> We did missed KAFKA-1841 which was fixed in later maint release. We
> already notified everyone to upgrade HDP-2.3.4 this is the apache kafka
> 0.9.0 + additional patches . We are making sure on our side not to miss
> any compatibility patches like these with 3rd party developers and have
> tests to ensure that.
>
> Thanks,
> Harsha
> On Wed, Dec 23, 2015, at 04:11 PM, Dana Powers wrote:
> > Hi all,
> >
> > I've been helping debug an issue filed against kafka-python related to
> > compatibility w/ Hortonworks 2.3.0.0 kafka release. As I understand it,
> > HDP
> > is currently based on snapshots of apache/kafka trunk, merged with some
> > custom patches from HDP itself.
> >
> > In this case, HDP's 2.3.0.0 kafka release missed a compatibility patch
> > that
> > I believe is critical for third-party library support. Unfortunately the
> > patch -- KAFKA-1841 -- was initially only applied to the 0.8.2 branch (it
> > was merged to trunk several months later in KAFKA-2068). Because it
> > wasn't
> > on trunk, it didn't get included in the HDP kafka releases.
> >
> > If you recall, KAFKA-1841 was needed to maintain backwards and forwards
> > compatibility wrt the change from zookeeper to kafka-backed offset
> > storage.
> > Not having this patch is fine if you only ever use the clients /
> > libraries
> > distributed in the that release -- and I imagine that is probably most
> > folks that are using it. But if you remember the thread on this issue
> > back
> > in the 0.8.2-beta review, the API incompatibility made third-party
> > clients
> > hard to develop and maintain if the goal is to support multiple broker
> > versions w/ the same client code [this is the goal of kafka-python].
> > Anyways, I'm really glad that the fix made it into the apache release,
> > but
> > now I'm sad that it didn't make it into HDP's release.
> >
> > Anyways, I think there's a couple takeaways here:
> >
> > (1) I'd recommend anyone using HDP who intends to use third-party kafka
> > consumers should upgrade to 2.3.4.0 or later. That version appears to
> > include the compatibility patch (KAFKA-2068). Of course if anyone is on
> > list from HDP, they may be able to provide better help on this.
> >
> > (2) I think more care should probably be taken to help vendors or anyone
> > tracking changes on trunk wrt released versions. Is there a list of all
> > KAFKA-XXXX patches that are released but not merged into trunk ?
> > KAFKA-1841
> > is obviously near and dear to my heart, but I wonder if there are other
> > patches like it?
> >
> > Happy holidays to all, and may the force be with you
> >
> > -Dana
>
Re: The JIRA Awakens [KAFKA-1841]
Posted by Harsha <ka...@harsha.io>.
HI Dana,
I worked on that release. Yes HDP-2.3.0 has lot of
additional patches on top of 0.8.2.1 mainly the kerberos
patches.
We did missed KAFKA-1841 which was fixed in later maint release. We
already notified everyone to upgrade HDP-2.3.4 this is the apache kafka
0.9.0 + additional patches . We are making sure on our side not to miss
any compatibility patches like these with 3rd party developers and have
tests to ensure that.
Thanks,
Harsha
On Wed, Dec 23, 2015, at 04:11 PM, Dana Powers wrote:
> Hi all,
>
> I've been helping debug an issue filed against kafka-python related to
> compatibility w/ Hortonworks 2.3.0.0 kafka release. As I understand it,
> HDP
> is currently based on snapshots of apache/kafka trunk, merged with some
> custom patches from HDP itself.
>
> In this case, HDP's 2.3.0.0 kafka release missed a compatibility patch
> that
> I believe is critical for third-party library support. Unfortunately the
> patch -- KAFKA-1841 -- was initially only applied to the 0.8.2 branch (it
> was merged to trunk several months later in KAFKA-2068). Because it
> wasn't
> on trunk, it didn't get included in the HDP kafka releases.
>
> If you recall, KAFKA-1841 was needed to maintain backwards and forwards
> compatibility wrt the change from zookeeper to kafka-backed offset
> storage.
> Not having this patch is fine if you only ever use the clients /
> libraries
> distributed in the that release -- and I imagine that is probably most
> folks that are using it. But if you remember the thread on this issue
> back
> in the 0.8.2-beta review, the API incompatibility made third-party
> clients
> hard to develop and maintain if the goal is to support multiple broker
> versions w/ the same client code [this is the goal of kafka-python].
> Anyways, I'm really glad that the fix made it into the apache release,
> but
> now I'm sad that it didn't make it into HDP's release.
>
> Anyways, I think there's a couple takeaways here:
>
> (1) I'd recommend anyone using HDP who intends to use third-party kafka
> consumers should upgrade to 2.3.4.0 or later. That version appears to
> include the compatibility patch (KAFKA-2068). Of course if anyone is on
> list from HDP, they may be able to provide better help on this.
>
> (2) I think more care should probably be taken to help vendors or anyone
> tracking changes on trunk wrt released versions. Is there a list of all
> KAFKA-XXXX patches that are released but not merged into trunk ?
> KAFKA-1841
> is obviously near and dear to my heart, but I wonder if there are other
> patches like it?
>
> Happy holidays to all, and may the force be with you
>
> -Dana
Re: [kafka-clients] The JIRA Awakens [KAFKA-1841]
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Dana,
Not sure about the old merge script, but with the new one used for GitHub
PRs it tracks the branches you choose to commit to and can update the JIRA
automatically, tagging the appropriate fix versions. So from now on, the
appropriate JIRA queries for issues with, e.g., fix version 0.9.0.1 but not
0.9.1.0, should give you the results you're looking for (although I would
think that committing changes only to a release branch and not to trunk
should be very unusual...).
-Ewen
On Wed, Dec 23, 2015 at 4:11 PM, Dana Powers <da...@gmail.com> wrote:
> Hi all,
>
> I've been helping debug an issue filed against kafka-python related to
> compatibility w/ Hortonworks 2.3.0.0 kafka release. As I understand it, HDP
> is currently based on snapshots of apache/kafka trunk, merged with some
> custom patches from HDP itself.
>
> In this case, HDP's 2.3.0.0 kafka release missed a compatibility patch
> that I believe is critical for third-party library support. Unfortunately
> the patch -- KAFKA-1841 -- was initially only applied to the 0.8.2 branch
> (it was merged to trunk several months later in KAFKA-2068). Because it
> wasn't on trunk, it didn't get included in the HDP kafka releases.
>
> If you recall, KAFKA-1841 was needed to maintain backwards and forwards
> compatibility wrt the change from zookeeper to kafka-backed offset storage.
> Not having this patch is fine if you only ever use the clients / libraries
> distributed in the that release -- and I imagine that is probably most
> folks that are using it. But if you remember the thread on this issue back
> in the 0.8.2-beta review, the API incompatibility made third-party clients
> hard to develop and maintain if the goal is to support multiple broker
> versions w/ the same client code [this is the goal of kafka-python].
> Anyways, I'm really glad that the fix made it into the apache release, but
> now I'm sad that it didn't make it into HDP's release.
>
> Anyways, I think there's a couple takeaways here:
>
> (1) I'd recommend anyone using HDP who intends to use third-party kafka
> consumers should upgrade to 2.3.4.0 or later. That version appears to
> include the compatibility patch (KAFKA-2068). Of course if anyone is on
> list from HDP, they may be able to provide better help on this.
>
> (2) I think more care should probably be taken to help vendors or anyone
> tracking changes on trunk wrt released versions. Is there a list of all
> KAFKA-XXXX patches that are released but not merged into trunk ? KAFKA-1841
> is obviously near and dear to my heart, but I wonder if there are other
> patches like it?
>
> Happy holidays to all, and may the force be with you
>
> -Dana
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscribe@googlegroups.com.
> To post to this group, send email to kafka-clients@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAAndJ_yarA%2BpTRLC1%3DdY3KtYY0esa6z9U8%3DM7k1WPwKB%2BtBNMQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/kafka-clients/CAAndJ_yarA%2BpTRLC1%3DdY3KtYY0esa6z9U8%3DM7k1WPwKB%2BtBNMQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
Thanks,
Ewen