You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Oleksiy Krivoshey <ol...@gmail.com> on 2016/02/16 12:33:08 UTC
Compression - MessageSet size
Hi!
The Kafka 0.9 protocol guide
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression
explicitly
states that there should be only single compressed message in a MessageSet,
but I'm definitely receiving two compressed messages in a single
MessageSet.
Can someone please confirm if this should be expected behaviour?
Thanks!
Re: Compression - MessageSet size
Posted by Alexis Midon <al...@airbnb.com.INVALID>.
- it would be interesting to see the actual ProduceRequests/Responses and
FetchReq/Resp.
- at this point I would dive into the broker source code and follow the
fetch request handling.
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L418
On Tue, Feb 16, 2016 at 10:24 AM Oleksiy Krivoshey <ol...@gmail.com>
wrote:
> I'm not using Java client, I'm developing my own:
> https://github.com/oleksiyk/kafka
> And I'm talking about MessageSet on the wire, not the one passed to library
> user.
>
> Example:
> Consumer is not started.
> Producer sends a batch #1 of 5 messages, compressed in a single message.
> delay 50 ms
> Producer sends a batch #2 of 5 messages, compressed in a single message.
>
> Start consumer and it will receive a single MessageSet with two messages in
> it, both messages are corresponding compressed batches. Both have codec=2
> (using Snappy). In order to receive all 10 original messages I need to
> decompress both messages in this single MessageSet. Yes, I will then
> concatenate these messages and send a single MessageSet with 10 messages to
> library user, but I just want to clarify, the Protocol Guide says there
> should be single compressed message but I'm able to receive 2, 3 and more,
> all in single MessageSet.
>
> Kafka 0.9
>
> I can provide the actual buffers received.
>
> Thanks!
>
>
> On Tue, 16 Feb 2016 at 20:01 Alexis Midon <alexis.midon@airbnb.com
> .invalid>
> wrote:
>
> > What makes you think there are 2? would you have data or code to share?
> >
> > When compression is enabled, multiple messages will be packed and
> > compressed in a MessageSet. That MessageSet will then have a single
> > message.
> > The interface however will let you iterate over the unpacked messages.
> See
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L166
> >
> >
> >
> >
> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala
> >
> >
> >
> > On Tue, Feb 16, 2016 at 3:33 AM Oleksiy Krivoshey <ol...@gmail.com>
> > wrote:
> >
> > > Hi!
> > >
> > > The Kafka 0.9 protocol guide
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression
> > > explicitly
> > > states that there should be only single compressed message in a
> > MessageSet,
> > > but I'm definitely receiving two compressed messages in a single
> > > MessageSet.
> > >
> > > Can someone please confirm if this should be expected behaviour?
> > >
> > > Thanks!
> > >
> >
>
Re: Compression - MessageSet size
Posted by Oleksiy Krivoshey <ol...@gmail.com>.
I'm not using Java client, I'm developing my own:
https://github.com/oleksiyk/kafka
And I'm talking about MessageSet on the wire, not the one passed to library
user.
Example:
Consumer is not started.
Producer sends a batch #1 of 5 messages, compressed in a single message.
delay 50 ms
Producer sends a batch #2 of 5 messages, compressed in a single message.
Start consumer and it will receive a single MessageSet with two messages in
it, both messages are corresponding compressed batches. Both have codec=2
(using Snappy). In order to receive all 10 original messages I need to
decompress both messages in this single MessageSet. Yes, I will then
concatenate these messages and send a single MessageSet with 10 messages to
library user, but I just want to clarify, the Protocol Guide says there
should be single compressed message but I'm able to receive 2, 3 and more,
all in single MessageSet.
Kafka 0.9
I can provide the actual buffers received.
Thanks!
On Tue, 16 Feb 2016 at 20:01 Alexis Midon <al...@airbnb.com.invalid>
wrote:
> What makes you think there are 2? would you have data or code to share?
>
> When compression is enabled, multiple messages will be packed and
> compressed in a MessageSet. That MessageSet will then have a single
> message.
> The interface however will let you iterate over the unpacked messages. See
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L166
>
>
>
> https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala
>
>
>
> On Tue, Feb 16, 2016 at 3:33 AM Oleksiy Krivoshey <ol...@gmail.com>
> wrote:
>
> > Hi!
> >
> > The Kafka 0.9 protocol guide
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression
> > explicitly
> > states that there should be only single compressed message in a
> MessageSet,
> > but I'm definitely receiving two compressed messages in a single
> > MessageSet.
> >
> > Can someone please confirm if this should be expected behaviour?
> >
> > Thanks!
> >
>
Re: Compression - MessageSet size
Posted by Alexis Midon <al...@airbnb.com.INVALID>.
What makes you think there are 2? would you have data or code to share?
When compression is enabled, multiple messages will be packed and
compressed in a MessageSet. That MessageSet will then have a single message.
The interface however will let you iterate over the unpacked messages. See
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L166
https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala
On Tue, Feb 16, 2016 at 3:33 AM Oleksiy Krivoshey <ol...@gmail.com>
wrote:
> Hi!
>
> The Kafka 0.9 protocol guide
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression
> explicitly
> states that there should be only single compressed message in a MessageSet,
> but I'm definitely receiving two compressed messages in a single
> MessageSet.
>
> Can someone please confirm if this should be expected behaviour?
>
> Thanks!
>