You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Cees de Groot <ce...@pagerduty.com> on 2016/03/26 22:39:15 UTC
Unexpected response on SyncGroup call
I'm helping out on adding 0.9 stuff to the Elixir Kafka driver (
https://github.com/kafkaex/kafka_ex), and I'm getting an unexpected
response on an integration test that makes a simple SyncGroup call.
In Elixir terms, I'm getting <<0, 0, 0, 3, 0, 0, 0, 0, 0, 0>> back. My
interpretation:
32 bits correlation id = 0,0,0,3
16 bits error code = 0,0
There's now 32 bits left. However, I'm expecting (following
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-SyncGroupResponse)
a MemberAssignment structure here, which is
16 bits of version (0)
32 bits of length (it's a simple integration test, so for now i'm happy to
accept the answer "0 assignments for you, dude!" ;-)).
Which clearly is more data than left in the response.
Am I misinterpreting the documentation here?
Re: Unexpected response on SyncGroup call
Posted by Cees de Groot <ce...@pagerduty.com>.
Makes sense. Thanks!
(and now how to find out how to turn off that signature for mailing lists
;-))
On Sun, Mar 27, 2016 at 10:54 AM, Dana Powers <da...@gmail.com> wrote:
> Agree, it's a bit confusing based on wiki alone right now. I think of
> MemberAssignment as a double-serialized field: first serialize to bytes
> using the MemberAssignment definition (version, topics, partitions, etc);
> then serialize the bytes into a request. It's the second serialization step
> that adds the 32bit length. Repeat backwards for deserializing.
>
> -Dana
>
>
Re: Unexpected response on SyncGroup call
Posted by Dana Powers <da...@gmail.com>.
Agree, it's a bit confusing based on wiki alone right now. I think of
MemberAssignment as a double-serialized field: first serialize to bytes
using the MemberAssignment definition (version, topics, partitions, etc);
then serialize the bytes into a request. It's the second serialization step
that adds the 32bit length. Repeat backwards for deserializing.
-Dana
On Mar 27, 2016 6:01 AM, "Cees de Groot" <ce...@pagerduty.com> wrote:
> Ah. That makes sense. That also means that on the outgoing request, this
> part should be coded as <length><MemberAssignment bytes>? These parts of
> the documentation aren't entirely clear on what is part of the
> client/server protocol and what is part of the (client-side only) consumer
> group implementation.
>
> I'll put my request and response next to each other, I do send some
> assignments in the SyncGroup request but it may be that because of me
> misinterpreting the docs I accidently drop four zeroes in a bit that the
> server expects as a length marker.
>
> On Sat, Mar 26, 2016 at 6:05 PM, Dana Powers <da...@gmail.com>
> wrote:
>
> > The MemberAssignment bytes returned in SyncResponse should be the bytes
> > that your leader sent in its SyncRequest. <<0, 0, 0, 0>> is simply an
> empty
> > Bytes array (32 bit length of 0). The broker does not alter those bytes
> as
> > far as I know, so despite the protocol doc describing what a
> > MemberAssignment struct *should* look like, it really is up to your
> client
> > to encode and decode that struct. Can you check what bytes your leader
> code
> > is sending?
> >
> > I assume you have this covered, but for completeness: each consumer
> should
> > check its JoinResponse to determine whether it has been selected as the
> > group leader. If the consumer is the leader, it needs to do the
> assignments
> > and send those back in its SyncRequest. All other consumers just send an
> > empty SyncRequest. SyncResponse for all consumers then includes the
> > assignments sent by the leader.
> >
> > -Dana
> >
> > On Sat, Mar 26, 2016 at 2:39 PM, Cees de Groot <ce...@pagerduty.com>
> wrote:
> >
> > > I'm helping out on adding 0.9 stuff to the Elixir Kafka driver (
> > > https://github.com/kafkaex/kafka_ex), and I'm getting an unexpected
> > > response on an integration test that makes a simple SyncGroup call.
> > >
> > > In Elixir terms, I'm getting <<0, 0, 0, 3, 0, 0, 0, 0, 0, 0>> back. My
> > > interpretation:
> > >
> > > 32 bits correlation id = 0,0,0,3
> > > 16 bits error code = 0,0
> > >
> > > There's now 32 bits left. However, I'm expecting (following
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-SyncGroupResponse
> > > )
> > > a MemberAssignment structure here, which is
> > >
> > > 16 bits of version (0)
> > > 32 bits of length (it's a simple integration test, so for now i'm happy
> > to
> > > accept the answer "0 assignments for you, dude!" ;-)).
> > >
> > > Which clearly is more data than left in the response.
> > >
> > > Am I misinterpreting the documentation here?
> > >
> >
>
>
>
> --
>
> *Cees de Groot*
> PRINCIPAL SOFTWARE ENGINEER
> [image: PagerDuty logo] <http://pagerduty.com/>
> pagerduty.com
> cees@pagerduty.com <ma...@pagerduty.com>
> +1(416)435-4085
>
> [image: Twitter] <http://twitter.com/pagerduty>[image: FaceBook]
> <https://www.facebook.com/PagerDuty>[image: Google+]
> <https://plus.google.com/114339089137644062989>[image: LinkedIn]
> <https://www.linkedin.com/company/pagerduty>[image: Blog]
> <https://blog.pagerduty.com/>
>
Re: Unexpected response on SyncGroup call
Posted by Cees de Groot <ce...@pagerduty.com>.
Ah. That makes sense. That also means that on the outgoing request, this
part should be coded as <length><MemberAssignment bytes>? These parts of
the documentation aren't entirely clear on what is part of the
client/server protocol and what is part of the (client-side only) consumer
group implementation.
I'll put my request and response next to each other, I do send some
assignments in the SyncGroup request but it may be that because of me
misinterpreting the docs I accidently drop four zeroes in a bit that the
server expects as a length marker.
On Sat, Mar 26, 2016 at 6:05 PM, Dana Powers <da...@gmail.com> wrote:
> The MemberAssignment bytes returned in SyncResponse should be the bytes
> that your leader sent in its SyncRequest. <<0, 0, 0, 0>> is simply an empty
> Bytes array (32 bit length of 0). The broker does not alter those bytes as
> far as I know, so despite the protocol doc describing what a
> MemberAssignment struct *should* look like, it really is up to your client
> to encode and decode that struct. Can you check what bytes your leader code
> is sending?
>
> I assume you have this covered, but for completeness: each consumer should
> check its JoinResponse to determine whether it has been selected as the
> group leader. If the consumer is the leader, it needs to do the assignments
> and send those back in its SyncRequest. All other consumers just send an
> empty SyncRequest. SyncResponse for all consumers then includes the
> assignments sent by the leader.
>
> -Dana
>
> On Sat, Mar 26, 2016 at 2:39 PM, Cees de Groot <ce...@pagerduty.com> wrote:
>
> > I'm helping out on adding 0.9 stuff to the Elixir Kafka driver (
> > https://github.com/kafkaex/kafka_ex), and I'm getting an unexpected
> > response on an integration test that makes a simple SyncGroup call.
> >
> > In Elixir terms, I'm getting <<0, 0, 0, 3, 0, 0, 0, 0, 0, 0>> back. My
> > interpretation:
> >
> > 32 bits correlation id = 0,0,0,3
> > 16 bits error code = 0,0
> >
> > There's now 32 bits left. However, I'm expecting (following
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-SyncGroupResponse
> > )
> > a MemberAssignment structure here, which is
> >
> > 16 bits of version (0)
> > 32 bits of length (it's a simple integration test, so for now i'm happy
> to
> > accept the answer "0 assignments for you, dude!" ;-)).
> >
> > Which clearly is more data than left in the response.
> >
> > Am I misinterpreting the documentation here?
> >
>
--
*Cees de Groot*
PRINCIPAL SOFTWARE ENGINEER
[image: PagerDuty logo] <http://pagerduty.com/>
pagerduty.com
cees@pagerduty.com <ma...@pagerduty.com>
+1(416)435-4085
[image: Twitter] <http://twitter.com/pagerduty>[image: FaceBook]
<https://www.facebook.com/PagerDuty>[image: Google+]
<https://plus.google.com/114339089137644062989>[image: LinkedIn]
<https://www.linkedin.com/company/pagerduty>[image: Blog]
<https://blog.pagerduty.com/>
Re: Unexpected response on SyncGroup call
Posted by Dana Powers <da...@gmail.com>.
The MemberAssignment bytes returned in SyncResponse should be the bytes
that your leader sent in its SyncRequest. <<0, 0, 0, 0>> is simply an empty
Bytes array (32 bit length of 0). The broker does not alter those bytes as
far as I know, so despite the protocol doc describing what a
MemberAssignment struct *should* look like, it really is up to your client
to encode and decode that struct. Can you check what bytes your leader code
is sending?
I assume you have this covered, but for completeness: each consumer should
check its JoinResponse to determine whether it has been selected as the
group leader. If the consumer is the leader, it needs to do the assignments
and send those back in its SyncRequest. All other consumers just send an
empty SyncRequest. SyncResponse for all consumers then includes the
assignments sent by the leader.
-Dana
On Sat, Mar 26, 2016 at 2:39 PM, Cees de Groot <ce...@pagerduty.com> wrote:
> I'm helping out on adding 0.9 stuff to the Elixir Kafka driver (
> https://github.com/kafkaex/kafka_ex), and I'm getting an unexpected
> response on an integration test that makes a simple SyncGroup call.
>
> In Elixir terms, I'm getting <<0, 0, 0, 3, 0, 0, 0, 0, 0, 0>> back. My
> interpretation:
>
> 32 bits correlation id = 0,0,0,3
> 16 bits error code = 0,0
>
> There's now 32 bits left. However, I'm expecting (following
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-SyncGroupResponse
> )
> a MemberAssignment structure here, which is
>
> 16 bits of version (0)
> 32 bits of length (it's a simple integration test, so for now i'm happy to
> accept the answer "0 assignments for you, dude!" ;-)).
>
> Which clearly is more data than left in the response.
>
> Am I misinterpreting the documentation here?
>