You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Stevo Slavić <ss...@gmail.com> on 2015/07/22 13:26:51 UTC
Re: Should 0.8.3 consumers correctly function with 0.8.2 brokers?
I'm getting "Unknown api code 11" even when both client and server are
0.8.3/trunk, when "KafkaConsumer.subscribe(String... topics)" is used.
Bug?
Kind regards,
Stevo Slavic.
On Fri, Apr 24, 2015 at 7:13 PM, Neha Narkhede <ne...@confluent.io> wrote:
> Yes, I was clearly confused :-)
>
> On Fri, Apr 24, 2015 at 9:37 AM, Sean Lydon <ly...@gmail.com> wrote:
>
> > Thanks for the responses. Ewen is correct that I am referring to the
> > *new* consumer (org.apache.kafka.clients.consumer.KafkaConsumer).
> >
> > I am extending the consumer to allow my applications more control over
> > committed offsets. I really want to get away from zookeeper (so using
> > the offset storage), and re-balancing is something I haven't really
> > needed to tackle in an automated/seamless way. Either way, I'll hold
> > off going further down this road until there is more interest.
> >
> > @Gwen
> > I set up a single consumer without partition.assignment.strategy or
> > rebalance.callback.class. I was unable to subscribe to just a topic
> > ("Unknown api code 11" on broker), but I could subscribe to a
> > topicpartition. This makes sense as I would need to handle re-balance
> > outside the consumer. Things functioned as expected (well I have an
> > additional minor fix to code from KAFKA-2121), and the only exceptions
> > on broker were due to closing consumers (which I have become
> > accustomed to). My tests are specific to my extended version of the
> > consumer, but they basically do a little writing and reading with
> > different serde classes with application controlled commits (similar
> > to onSuccess and onFailure after each record, but with tolerance for
> > out of order acknowledgements).
> >
> > If you are interested, here is the patch of the hack against trunk.
> >
> > On Thu, Apr 23, 2015 at 10:27 PM, Ewen Cheslack-Postava
> > <ew...@confluent.io> wrote:
> > > @Neha I think you're mixing up the 0.8.1/0.8.2 updates and the
> > 0.8.2/0.8.3
> > > that's being discussed here?
> > >
> > > I think the original question was about using the *new* consumer
> > ("clients
> > > consumer") with 0.8.2. Gwen's right, it will use features not even
> > > implemented in the broker in trunk yet, let alone the 0.8.2.
> > >
> > > I don't think the "enable.commit.downgrade" type option, or supporting
> > the
> > > old protocol with the new consumer at all, makes much sense. You'd end
> up
> > > with some weird hybrid of simple and high-level consumers -- you could
> > use
> > > offset storage, but you'd have to manage rebalancing yourself since
> none
> > of
> > > the coordinator support would be there.
> > >
> > >
> > > On Thu, Apr 23, 2015 at 9:22 PM, Neha Narkhede <ne...@confluent.io>
> > wrote:
> > >
> > >> My understanding is that ideally the 0.8.3 consumer should work with
> an
> > >> 0.8.2 broker if the offset commit config was set to "zookeeper".
> > >>
> > >> The only thing that might not work is offset commit to Kafka, which
> > makes
> > >> sense since the 0.8.2 broker does not support Kafka based offset
> > >> management.
> > >>
> > >> If we broke all kinds of offset commits, then it seems like a
> > regression,
> > >> no?
> > >>
> > >> On Thu, Apr 23, 2015 at 7:26 PM, Gwen Shapira <gs...@cloudera.com>
> > >> wrote:
> > >>
> > >> > I didn't think 0.8.3 consumer will ever be able to talk to 0.8.2
> > >> > broker... there are some essential pieces that are missing in 0.8.2
> > >> > (Coordinator, Heartbeat, etc).
> > >> > Maybe I'm missing something. It will be nice if this will work :)
> > >> >
> > >> > Mind sharing what / how you tested? Were there no errors in broker
> > >> > logs after your fix?
> > >> >
> > >> > On Thu, Apr 23, 2015 at 5:37 PM, Sean Lydon <ly...@gmail.com>
> > >> wrote:
> > >> > > Currently the clients consumer (trunk) sends offset commit
> requests
> > of
> > >> > > version 2. The 0.8.2 brokers fail to handle this particular
> request
> > >> > > with a:
> > >> > >
> > >> > > java.lang.AssertionError: assertion failed: Version 2 is invalid
> for
> > >> > > OffsetCommitRequest. Valid versions are 0 or 1.
> > >> > >
> > >> > > I was able to make this work via a forceful downgrade of this
> > >> > > particular request, but I would like some feedback on whether a
> > >> > > "enable.commit.downgrade" configuration would be a tolerable
> method
> > to
> > >> > > allow 0.8.3 consumers to interact with 0.8.2 brokers. I'm also
> > >> > > interested in this even being a goal worth pursuing.
> > >> > >
> > >> > > Thanks,
> > >> > > Sean
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Neha
> > >>
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> >
>
>
>
> --
> Thanks,
> Neha
>
Re: Should 0.8.3 consumers correctly function with 0.8.2 brokers?
Posted by Stevo Slavić <ss...@gmail.com>.
Jiangjie,
Seems I was misunderstood; KafakConsumer.poll after subscribing to topic
via "KafkaConsumer.subscribe(String... topics)" does not work, "Unknown api
code 11" error, even with both client and broker being latest 0.8.3/trunk.
Will try to create a failing test and open bug report.
Kind regards,
Stevo Slavic.
On Wed, Jul 22, 2015 at 8:36 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> I don't think we have consumer coordinator in 0.8.2 brokers. So
> KafkaConsumer in 0.8.3 will only be able to subscribe to partitions
> explicitly. Subscribing to a topic won't work with 0.8.2 brokers.
>
> Jiangjie (Becket) Qin
>
> On Wed, Jul 22, 2015 at 4:26 AM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > I'm getting "Unknown api code 11" even when both client and server are
> > 0.8.3/trunk, when "KafkaConsumer.subscribe(String... topics)" is used.
> >
> > Bug?
> >
> > Kind regards,
> > Stevo Slavic.
> >
> > On Fri, Apr 24, 2015 at 7:13 PM, Neha Narkhede <ne...@confluent.io>
> wrote:
> >
> > > Yes, I was clearly confused :-)
> > >
> > > On Fri, Apr 24, 2015 at 9:37 AM, Sean Lydon <ly...@gmail.com>
> > wrote:
> > >
> > > > Thanks for the responses. Ewen is correct that I am referring to the
> > > > *new* consumer (org.apache.kafka.clients.consumer.KafkaConsumer).
> > > >
> > > > I am extending the consumer to allow my applications more control
> over
> > > > committed offsets. I really want to get away from zookeeper (so
> using
> > > > the offset storage), and re-balancing is something I haven't really
> > > > needed to tackle in an automated/seamless way. Either way, I'll hold
> > > > off going further down this road until there is more interest.
> > > >
> > > > @Gwen
> > > > I set up a single consumer without partition.assignment.strategy or
> > > > rebalance.callback.class. I was unable to subscribe to just a topic
> > > > ("Unknown api code 11" on broker), but I could subscribe to a
> > > > topicpartition. This makes sense as I would need to handle
> re-balance
> > > > outside the consumer. Things functioned as expected (well I have an
> > > > additional minor fix to code from KAFKA-2121), and the only
> exceptions
> > > > on broker were due to closing consumers (which I have become
> > > > accustomed to). My tests are specific to my extended version of the
> > > > consumer, but they basically do a little writing and reading with
> > > > different serde classes with application controlled commits (similar
> > > > to onSuccess and onFailure after each record, but with tolerance for
> > > > out of order acknowledgements).
> > > >
> > > > If you are interested, here is the patch of the hack against trunk.
> > > >
> > > > On Thu, Apr 23, 2015 at 10:27 PM, Ewen Cheslack-Postava
> > > > <ew...@confluent.io> wrote:
> > > > > @Neha I think you're mixing up the 0.8.1/0.8.2 updates and the
> > > > 0.8.2/0.8.3
> > > > > that's being discussed here?
> > > > >
> > > > > I think the original question was about using the *new* consumer
> > > > ("clients
> > > > > consumer") with 0.8.2. Gwen's right, it will use features not even
> > > > > implemented in the broker in trunk yet, let alone the 0.8.2.
> > > > >
> > > > > I don't think the "enable.commit.downgrade" type option, or
> > supporting
> > > > the
> > > > > old protocol with the new consumer at all, makes much sense. You'd
> > end
> > > up
> > > > > with some weird hybrid of simple and high-level consumers -- you
> > could
> > > > use
> > > > > offset storage, but you'd have to manage rebalancing yourself since
> > > none
> > > > of
> > > > > the coordinator support would be there.
> > > > >
> > > > >
> > > > > On Thu, Apr 23, 2015 at 9:22 PM, Neha Narkhede <ne...@confluent.io>
> > > > wrote:
> > > > >
> > > > >> My understanding is that ideally the 0.8.3 consumer should work
> with
> > > an
> > > > >> 0.8.2 broker if the offset commit config was set to "zookeeper".
> > > > >>
> > > > >> The only thing that might not work is offset commit to Kafka,
> which
> > > > makes
> > > > >> sense since the 0.8.2 broker does not support Kafka based offset
> > > > >> management.
> > > > >>
> > > > >> If we broke all kinds of offset commits, then it seems like a
> > > > regression,
> > > > >> no?
> > > > >>
> > > > >> On Thu, Apr 23, 2015 at 7:26 PM, Gwen Shapira <
> > gshapira@cloudera.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I didn't think 0.8.3 consumer will ever be able to talk to 0.8.2
> > > > >> > broker... there are some essential pieces that are missing in
> > 0.8.2
> > > > >> > (Coordinator, Heartbeat, etc).
> > > > >> > Maybe I'm missing something. It will be nice if this will work
> :)
> > > > >> >
> > > > >> > Mind sharing what / how you tested? Were there no errors in
> broker
> > > > >> > logs after your fix?
> > > > >> >
> > > > >> > On Thu, Apr 23, 2015 at 5:37 PM, Sean Lydon <
> lydon.sean@gmail.com
> > >
> > > > >> wrote:
> > > > >> > > Currently the clients consumer (trunk) sends offset commit
> > > requests
> > > > of
> > > > >> > > version 2. The 0.8.2 brokers fail to handle this particular
> > > request
> > > > >> > > with a:
> > > > >> > >
> > > > >> > > java.lang.AssertionError: assertion failed: Version 2 is
> invalid
> > > for
> > > > >> > > OffsetCommitRequest. Valid versions are 0 or 1.
> > > > >> > >
> > > > >> > > I was able to make this work via a forceful downgrade of this
> > > > >> > > particular request, but I would like some feedback on whether
> a
> > > > >> > > "enable.commit.downgrade" configuration would be a tolerable
> > > method
> > > > to
> > > > >> > > allow 0.8.3 consumers to interact with 0.8.2 brokers. I'm
> also
> > > > >> > > interested in this even being a goal worth pursuing.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > Sean
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Thanks,
> > > > >> Neha
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Ewen
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Neha
> > >
> >
>
Re: Should 0.8.3 consumers correctly function with 0.8.2 brokers?
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
I don't think we have consumer coordinator in 0.8.2 brokers. So
KafkaConsumer in 0.8.3 will only be able to subscribe to partitions
explicitly. Subscribing to a topic won't work with 0.8.2 brokers.
Jiangjie (Becket) Qin
On Wed, Jul 22, 2015 at 4:26 AM, Stevo Slavić <ss...@gmail.com> wrote:
> I'm getting "Unknown api code 11" even when both client and server are
> 0.8.3/trunk, when "KafkaConsumer.subscribe(String... topics)" is used.
>
> Bug?
>
> Kind regards,
> Stevo Slavic.
>
> On Fri, Apr 24, 2015 at 7:13 PM, Neha Narkhede <ne...@confluent.io> wrote:
>
> > Yes, I was clearly confused :-)
> >
> > On Fri, Apr 24, 2015 at 9:37 AM, Sean Lydon <ly...@gmail.com>
> wrote:
> >
> > > Thanks for the responses. Ewen is correct that I am referring to the
> > > *new* consumer (org.apache.kafka.clients.consumer.KafkaConsumer).
> > >
> > > I am extending the consumer to allow my applications more control over
> > > committed offsets. I really want to get away from zookeeper (so using
> > > the offset storage), and re-balancing is something I haven't really
> > > needed to tackle in an automated/seamless way. Either way, I'll hold
> > > off going further down this road until there is more interest.
> > >
> > > @Gwen
> > > I set up a single consumer without partition.assignment.strategy or
> > > rebalance.callback.class. I was unable to subscribe to just a topic
> > > ("Unknown api code 11" on broker), but I could subscribe to a
> > > topicpartition. This makes sense as I would need to handle re-balance
> > > outside the consumer. Things functioned as expected (well I have an
> > > additional minor fix to code from KAFKA-2121), and the only exceptions
> > > on broker were due to closing consumers (which I have become
> > > accustomed to). My tests are specific to my extended version of the
> > > consumer, but they basically do a little writing and reading with
> > > different serde classes with application controlled commits (similar
> > > to onSuccess and onFailure after each record, but with tolerance for
> > > out of order acknowledgements).
> > >
> > > If you are interested, here is the patch of the hack against trunk.
> > >
> > > On Thu, Apr 23, 2015 at 10:27 PM, Ewen Cheslack-Postava
> > > <ew...@confluent.io> wrote:
> > > > @Neha I think you're mixing up the 0.8.1/0.8.2 updates and the
> > > 0.8.2/0.8.3
> > > > that's being discussed here?
> > > >
> > > > I think the original question was about using the *new* consumer
> > > ("clients
> > > > consumer") with 0.8.2. Gwen's right, it will use features not even
> > > > implemented in the broker in trunk yet, let alone the 0.8.2.
> > > >
> > > > I don't think the "enable.commit.downgrade" type option, or
> supporting
> > > the
> > > > old protocol with the new consumer at all, makes much sense. You'd
> end
> > up
> > > > with some weird hybrid of simple and high-level consumers -- you
> could
> > > use
> > > > offset storage, but you'd have to manage rebalancing yourself since
> > none
> > > of
> > > > the coordinator support would be there.
> > > >
> > > >
> > > > On Thu, Apr 23, 2015 at 9:22 PM, Neha Narkhede <ne...@confluent.io>
> > > wrote:
> > > >
> > > >> My understanding is that ideally the 0.8.3 consumer should work with
> > an
> > > >> 0.8.2 broker if the offset commit config was set to "zookeeper".
> > > >>
> > > >> The only thing that might not work is offset commit to Kafka, which
> > > makes
> > > >> sense since the 0.8.2 broker does not support Kafka based offset
> > > >> management.
> > > >>
> > > >> If we broke all kinds of offset commits, then it seems like a
> > > regression,
> > > >> no?
> > > >>
> > > >> On Thu, Apr 23, 2015 at 7:26 PM, Gwen Shapira <
> gshapira@cloudera.com>
> > > >> wrote:
> > > >>
> > > >> > I didn't think 0.8.3 consumer will ever be able to talk to 0.8.2
> > > >> > broker... there are some essential pieces that are missing in
> 0.8.2
> > > >> > (Coordinator, Heartbeat, etc).
> > > >> > Maybe I'm missing something. It will be nice if this will work :)
> > > >> >
> > > >> > Mind sharing what / how you tested? Were there no errors in broker
> > > >> > logs after your fix?
> > > >> >
> > > >> > On Thu, Apr 23, 2015 at 5:37 PM, Sean Lydon <lydon.sean@gmail.com
> >
> > > >> wrote:
> > > >> > > Currently the clients consumer (trunk) sends offset commit
> > requests
> > > of
> > > >> > > version 2. The 0.8.2 brokers fail to handle this particular
> > request
> > > >> > > with a:
> > > >> > >
> > > >> > > java.lang.AssertionError: assertion failed: Version 2 is invalid
> > for
> > > >> > > OffsetCommitRequest. Valid versions are 0 or 1.
> > > >> > >
> > > >> > > I was able to make this work via a forceful downgrade of this
> > > >> > > particular request, but I would like some feedback on whether a
> > > >> > > "enable.commit.downgrade" configuration would be a tolerable
> > method
> > > to
> > > >> > > allow 0.8.3 consumers to interact with 0.8.2 brokers. I'm also
> > > >> > > interested in this even being a goal worth pursuing.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Sean
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Thanks,
> > > >> Neha
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ewen
> > >
> >
> >
> >
> > --
> > Thanks,
> > Neha
> >
>