You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Petar Maymounkov <pe...@tumblr.com> on 2012/05/15 16:43:47 UTC
multi-fetch response doc
Hi all,
The kafka doc "Writing a Kafka driver" does not seem to
have the format of the multi-fetch response.
(Surely it is not the same as the fetch response).
Can anyone point me to a place where I can read about it?
Thanks
Petar
Re: multi-fetch response doc
Posted by Jun Rao <ju...@gmail.com>.
The 100 bytes includes per message overhead and the payload.
Jun
On Tue, May 15, 2012 at 12:28 PM, Petar Maymounkov <pe...@tumblr.com> wrote:
> Ok, got it.
>
> I am observing another strange behavior now.
>
> When I do a simple Fetch request, I specify a MaxSize of 100.
>
> The setting is that the requested topic has much more messages than 100
> bytes, so
> I expect the broker will respond with however many messages fit in 100
> bytes.
>
> What happens:
>
> The FetchResponse's size parameter minus the size of the header (2 bytes
> for the error code) equals exactly 100 (what I specified as a MaxSize).
>
> So I have 100 bytes of messages in this response.
>
> I start reading the messages 1 by 1, until I consumed 96 bytes.
> There's 4 bytes left. This forms an incomplete message.
> The 4 bytes (the LENGTH parameter) give the correct length for the
> next message, but then the rest of the fields (MAGIC, COMPRESSION and BODY)
> are missing.
>
> Is this behavior y design?
>
> Thanks
> P
>
>
> On Tue, May 15, 2012 at 2:12 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > The response relies on the position of each fetch response matching the
> > position of the corresponding fetch request.
> >
> > Jun
> >
> > On Tue, May 15, 2012 at 10:38 AM, Petar Maymounkov <pe...@tumblr.com>
> > wrote:
> >
> > > So this is the same format as regular fetch then.
> > >
> > > What confuses me is how you differentiate which messages
> > > are in response to which part of the multifetch request.
> > >
> > > Thanks
> > > Petar
> > >
> > >
> > > On Tue, May 15, 2012 at 12:07 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > It's a 4-byte fetch count, followed by a 2-byte error code, followed
> > by a
> > > > list of ByteBufferMessageSet. The details can be in the
> > > MultiFetchResponse
> > > > class.
> > > >
> > > > Jun
> > > >
> > > > On Tue, May 15, 2012 at 7:43 AM, Petar Maymounkov <pe...@tumblr.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > The kafka doc "Writing a Kafka driver" does not seem to
> > > > > have the format of the multi-fetch response.
> > > > >
> > > > > (Surely it is not the same as the fetch response).
> > > > >
> > > > > Can anyone point me to a place where I can read about it?
> > > > >
> > > > > Thanks
> > > > > Petar
> > > > >
> > > >
> > >
> >
>
Re: multi-fetch response doc
Posted by Petar Maymounkov <pe...@tumblr.com>.
Ok, got it.
I am observing another strange behavior now.
When I do a simple Fetch request, I specify a MaxSize of 100.
The setting is that the requested topic has much more messages than 100
bytes, so
I expect the broker will respond with however many messages fit in 100
bytes.
What happens:
The FetchResponse's size parameter minus the size of the header (2 bytes
for the error code) equals exactly 100 (what I specified as a MaxSize).
So I have 100 bytes of messages in this response.
I start reading the messages 1 by 1, until I consumed 96 bytes.
There's 4 bytes left. This forms an incomplete message.
The 4 bytes (the LENGTH parameter) give the correct length for the
next message, but then the rest of the fields (MAGIC, COMPRESSION and BODY)
are missing.
Is this behavior y design?
Thanks
P
On Tue, May 15, 2012 at 2:12 PM, Jun Rao <ju...@gmail.com> wrote:
> The response relies on the position of each fetch response matching the
> position of the corresponding fetch request.
>
> Jun
>
> On Tue, May 15, 2012 at 10:38 AM, Petar Maymounkov <pe...@tumblr.com>
> wrote:
>
> > So this is the same format as regular fetch then.
> >
> > What confuses me is how you differentiate which messages
> > are in response to which part of the multifetch request.
> >
> > Thanks
> > Petar
> >
> >
> > On Tue, May 15, 2012 at 12:07 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > It's a 4-byte fetch count, followed by a 2-byte error code, followed
> by a
> > > list of ByteBufferMessageSet. The details can be in the
> > MultiFetchResponse
> > > class.
> > >
> > > Jun
> > >
> > > On Tue, May 15, 2012 at 7:43 AM, Petar Maymounkov <pe...@tumblr.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > The kafka doc "Writing a Kafka driver" does not seem to
> > > > have the format of the multi-fetch response.
> > > >
> > > > (Surely it is not the same as the fetch response).
> > > >
> > > > Can anyone point me to a place where I can read about it?
> > > >
> > > > Thanks
> > > > Petar
> > > >
> > >
> >
>
Re: multi-fetch response doc
Posted by Jun Rao <ju...@gmail.com>.
The response relies on the position of each fetch response matching the
position of the corresponding fetch request.
Jun
On Tue, May 15, 2012 at 10:38 AM, Petar Maymounkov <pe...@tumblr.com> wrote:
> So this is the same format as regular fetch then.
>
> What confuses me is how you differentiate which messages
> are in response to which part of the multifetch request.
>
> Thanks
> Petar
>
>
> On Tue, May 15, 2012 at 12:07 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > It's a 4-byte fetch count, followed by a 2-byte error code, followed by a
> > list of ByteBufferMessageSet. The details can be in the
> MultiFetchResponse
> > class.
> >
> > Jun
> >
> > On Tue, May 15, 2012 at 7:43 AM, Petar Maymounkov <pe...@tumblr.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > The kafka doc "Writing a Kafka driver" does not seem to
> > > have the format of the multi-fetch response.
> > >
> > > (Surely it is not the same as the fetch response).
> > >
> > > Can anyone point me to a place where I can read about it?
> > >
> > > Thanks
> > > Petar
> > >
> >
>
Re: multi-fetch response doc
Posted by Petar Maymounkov <pe...@tumblr.com>.
So this is the same format as regular fetch then.
What confuses me is how you differentiate which messages
are in response to which part of the multifetch request.
Thanks
Petar
On Tue, May 15, 2012 at 12:07 PM, Jun Rao <ju...@gmail.com> wrote:
> It's a 4-byte fetch count, followed by a 2-byte error code, followed by a
> list of ByteBufferMessageSet. The details can be in the MultiFetchResponse
> class.
>
> Jun
>
> On Tue, May 15, 2012 at 7:43 AM, Petar Maymounkov <pe...@tumblr.com>
> wrote:
>
> > Hi all,
> >
> > The kafka doc "Writing a Kafka driver" does not seem to
> > have the format of the multi-fetch response.
> >
> > (Surely it is not the same as the fetch response).
> >
> > Can anyone point me to a place where I can read about it?
> >
> > Thanks
> > Petar
> >
>
Re: multi-fetch response doc
Posted by Jun Rao <ju...@gmail.com>.
It's a 4-byte fetch count, followed by a 2-byte error code, followed by a
list of ByteBufferMessageSet. The details can be in the MultiFetchResponse
class.
Jun
On Tue, May 15, 2012 at 7:43 AM, Petar Maymounkov <pe...@tumblr.com> wrote:
> Hi all,
>
> The kafka doc "Writing a Kafka driver" does not seem to
> have the format of the multi-fetch response.
>
> (Surely it is not the same as the fetch response).
>
> Can anyone point me to a place where I can read about it?
>
> Thanks
> Petar
>