You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kane Kane <ka...@gmail.com> on 2014/06/06 23:15:42 UTC
message.max.bytes and mirrormaker tool
I've tried to run mirrormaker tools in async mode and I get
WARN Produce request with correlation id 263 failed due to
[benchmark1,30]: kafka.common.MessageSizeTooLargeException
(kafka.producer.async.DefaultEventHandler)
I don't get error in sync mode. My message.max.bytes is default
(1000000). As I understand the difference between sync and async mode
is that async collects batches before sending. Does it mean that batch
should be less than message.max.bytes, or any single message within
batch should be less than that.
Thanks.
Re: message.max.bytes and mirrormaker tool
Posted by Guozhang Wang <wa...@gmail.com>.
You can try to reduce batch.num.messages first and see if the throughput
gets affected.
For your second question, we do not have a good solution to that since the
offsets are not consistent across data centers like you said. One way we
did is to have the consumer consuming both data centers, but keep the slave
data center's consumer do no-op; once the master data center is totally
gone, turn the nob to let the slave data center's consumers to both consume
and process. Doing so still do not give you consistency as you may consume
duplicates or data loss, but gives you a relatively close point of resuming
compared with the previous consumer.
Guozhang
On Mon, Jun 9, 2014 at 2:08 PM, Kane Kane <ka...@gmail.com> wrote:
> What would you recommend in this case? Bump up the max.message.bytes,
> use sync producer or lower batch.num.messages? Also does it matter on
> which side mirrormaker is located, source or target one?
>
> And another related question: as I understand, potentially offsets
> might be different for source and target, and if the whole datacenter
> is lost, you probably loose offsets in zookeeper as well. What is your
> strategy to switch over producers and consumers to the new cluster,
> given consumer should pickup from the same point where it was dropped
> in another datacenter?
>
> Thanks.
>
> On Sat, Jun 7, 2014 at 3:49 PM, Guozhang Wang <wa...@gmail.com> wrote:
> > In the old producer yes, in the new producer (available in 0.8.1.1) the
> > batch size is by bytes instead of #. messages, which gives you a better
> > control.
> >
> > Guozhang
> >
> >
> > On Sat, Jun 7, 2014 at 2:48 PM, Kane Kane <ka...@gmail.com> wrote:
> >
> >> Ah, that makes sense then. As I understand right now there is no
> >> something like queue.buffering.max.size to control batch size?
> >>
> >> On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >> > With compression, the batch of messages are compressed in to a
> "single"
> >> > wrapper message and sent to the broker, and the broker will reject the
> >> > request if this single message's size is larger than 1MB. So you need
> to
> >> > either change your max request size on broker or reduce your producer
> >> batch
> >> > size.
> >> >
> >> > Guozhang
> >> >
> >> >
> >> > On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <ka...@gmail.com>
> wrote:
> >> >
> >> >> Yes, messages were compressed with gzip and I've enabled the same
> >> >> compression in mirrormaker producer.
> >> >>
> >> >> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wa...@gmail.com>
> >> wrote:
> >> >> > Kane,
> >> >> >
> >> >> > Did you use any compression method?
> >> >> >
> >> >> > Guozhang
> >> >> >
> >> >> >
> >> >> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <ka...@gmail.com>
> >> wrote:
> >> >> >
> >> >> >> I've tried to run mirrormaker tools in async mode and I get
> >> >> >> WARN Produce request with correlation id 263 failed due to
> >> >> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
> >> >> >> (kafka.producer.async.DefaultEventHandler)
> >> >> >>
> >> >> >> I don't get error in sync mode. My message.max.bytes is default
> >> >> >> (1000000). As I understand the difference between sync and async
> mode
> >> >> >> is that async collects batches before sending. Does it mean that
> >> batch
> >> >> >> should be less than message.max.bytes, or any single message
> within
> >> >> >> batch should be less than that.
> >> >> >>
> >> >> >> Thanks.
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > -- Guozhang
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > -- Guozhang
> >>
> >
> >
> >
> > --
> > -- Guozhang
>
--
-- Guozhang
Re: message.max.bytes and mirrormaker tool
Posted by Kane Kane <ka...@gmail.com>.
What would you recommend in this case? Bump up the max.message.bytes,
use sync producer or lower batch.num.messages? Also does it matter on
which side mirrormaker is located, source or target one?
And another related question: as I understand, potentially offsets
might be different for source and target, and if the whole datacenter
is lost, you probably loose offsets in zookeeper as well. What is your
strategy to switch over producers and consumers to the new cluster,
given consumer should pickup from the same point where it was dropped
in another datacenter?
Thanks.
On Sat, Jun 7, 2014 at 3:49 PM, Guozhang Wang <wa...@gmail.com> wrote:
> In the old producer yes, in the new producer (available in 0.8.1.1) the
> batch size is by bytes instead of #. messages, which gives you a better
> control.
>
> Guozhang
>
>
> On Sat, Jun 7, 2014 at 2:48 PM, Kane Kane <ka...@gmail.com> wrote:
>
>> Ah, that makes sense then. As I understand right now there is no
>> something like queue.buffering.max.size to control batch size?
>>
>> On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang <wa...@gmail.com> wrote:
>> > With compression, the batch of messages are compressed in to a "single"
>> > wrapper message and sent to the broker, and the broker will reject the
>> > request if this single message's size is larger than 1MB. So you need to
>> > either change your max request size on broker or reduce your producer
>> batch
>> > size.
>> >
>> > Guozhang
>> >
>> >
>> > On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <ka...@gmail.com> wrote:
>> >
>> >> Yes, messages were compressed with gzip and I've enabled the same
>> >> compression in mirrormaker producer.
>> >>
>> >> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wa...@gmail.com>
>> wrote:
>> >> > Kane,
>> >> >
>> >> > Did you use any compression method?
>> >> >
>> >> > Guozhang
>> >> >
>> >> >
>> >> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <ka...@gmail.com>
>> wrote:
>> >> >
>> >> >> I've tried to run mirrormaker tools in async mode and I get
>> >> >> WARN Produce request with correlation id 263 failed due to
>> >> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
>> >> >> (kafka.producer.async.DefaultEventHandler)
>> >> >>
>> >> >> I don't get error in sync mode. My message.max.bytes is default
>> >> >> (1000000). As I understand the difference between sync and async mode
>> >> >> is that async collects batches before sending. Does it mean that
>> batch
>> >> >> should be less than message.max.bytes, or any single message within
>> >> >> batch should be less than that.
>> >> >>
>> >> >> Thanks.
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > -- Guozhang
>> >>
>> >
>> >
>> >
>> > --
>> > -- Guozhang
>>
>
>
>
> --
> -- Guozhang
Re: message.max.bytes and mirrormaker tool
Posted by Guozhang Wang <wa...@gmail.com>.
In the old producer yes, in the new producer (available in 0.8.1.1) the
batch size is by bytes instead of #. messages, which gives you a better
control.
Guozhang
On Sat, Jun 7, 2014 at 2:48 PM, Kane Kane <ka...@gmail.com> wrote:
> Ah, that makes sense then. As I understand right now there is no
> something like queue.buffering.max.size to control batch size?
>
> On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang <wa...@gmail.com> wrote:
> > With compression, the batch of messages are compressed in to a "single"
> > wrapper message and sent to the broker, and the broker will reject the
> > request if this single message's size is larger than 1MB. So you need to
> > either change your max request size on broker or reduce your producer
> batch
> > size.
> >
> > Guozhang
> >
> >
> > On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <ka...@gmail.com> wrote:
> >
> >> Yes, messages were compressed with gzip and I've enabled the same
> >> compression in mirrormaker producer.
> >>
> >> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >> > Kane,
> >> >
> >> > Did you use any compression method?
> >> >
> >> > Guozhang
> >> >
> >> >
> >> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <ka...@gmail.com>
> wrote:
> >> >
> >> >> I've tried to run mirrormaker tools in async mode and I get
> >> >> WARN Produce request with correlation id 263 failed due to
> >> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
> >> >> (kafka.producer.async.DefaultEventHandler)
> >> >>
> >> >> I don't get error in sync mode. My message.max.bytes is default
> >> >> (1000000). As I understand the difference between sync and async mode
> >> >> is that async collects batches before sending. Does it mean that
> batch
> >> >> should be less than message.max.bytes, or any single message within
> >> >> batch should be less than that.
> >> >>
> >> >> Thanks.
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > -- Guozhang
> >>
> >
> >
> >
> > --
> > -- Guozhang
>
--
-- Guozhang
Re: message.max.bytes and mirrormaker tool
Posted by Kane Kane <ka...@gmail.com>.
Ah, that makes sense then. As I understand right now there is no
something like queue.buffering.max.size to control batch size?
On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang <wa...@gmail.com> wrote:
> With compression, the batch of messages are compressed in to a "single"
> wrapper message and sent to the broker, and the broker will reject the
> request if this single message's size is larger than 1MB. So you need to
> either change your max request size on broker or reduce your producer batch
> size.
>
> Guozhang
>
>
> On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <ka...@gmail.com> wrote:
>
>> Yes, messages were compressed with gzip and I've enabled the same
>> compression in mirrormaker producer.
>>
>> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wa...@gmail.com> wrote:
>> > Kane,
>> >
>> > Did you use any compression method?
>> >
>> > Guozhang
>> >
>> >
>> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <ka...@gmail.com> wrote:
>> >
>> >> I've tried to run mirrormaker tools in async mode and I get
>> >> WARN Produce request with correlation id 263 failed due to
>> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
>> >> (kafka.producer.async.DefaultEventHandler)
>> >>
>> >> I don't get error in sync mode. My message.max.bytes is default
>> >> (1000000). As I understand the difference between sync and async mode
>> >> is that async collects batches before sending. Does it mean that batch
>> >> should be less than message.max.bytes, or any single message within
>> >> batch should be less than that.
>> >>
>> >> Thanks.
>> >>
>> >
>> >
>> >
>> > --
>> > -- Guozhang
>>
>
>
>
> --
> -- Guozhang
Re: message.max.bytes and mirrormaker tool
Posted by Guozhang Wang <wa...@gmail.com>.
With compression, the batch of messages are compressed in to a "single"
wrapper message and sent to the broker, and the broker will reject the
request if this single message's size is larger than 1MB. So you need to
either change your max request size on broker or reduce your producer batch
size.
Guozhang
On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <ka...@gmail.com> wrote:
> Yes, messages were compressed with gzip and I've enabled the same
> compression in mirrormaker producer.
>
> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wa...@gmail.com> wrote:
> > Kane,
> >
> > Did you use any compression method?
> >
> > Guozhang
> >
> >
> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <ka...@gmail.com> wrote:
> >
> >> I've tried to run mirrormaker tools in async mode and I get
> >> WARN Produce request with correlation id 263 failed due to
> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
> >> (kafka.producer.async.DefaultEventHandler)
> >>
> >> I don't get error in sync mode. My message.max.bytes is default
> >> (1000000). As I understand the difference between sync and async mode
> >> is that async collects batches before sending. Does it mean that batch
> >> should be less than message.max.bytes, or any single message within
> >> batch should be less than that.
> >>
> >> Thanks.
> >>
> >
> >
> >
> > --
> > -- Guozhang
>
--
-- Guozhang
Re: message.max.bytes and mirrormaker tool
Posted by Kane Kane <ka...@gmail.com>.
Yes, messages were compressed with gzip and I've enabled the same
compression in mirrormaker producer.
On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Kane,
>
> Did you use any compression method?
>
> Guozhang
>
>
> On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <ka...@gmail.com> wrote:
>
>> I've tried to run mirrormaker tools in async mode and I get
>> WARN Produce request with correlation id 263 failed due to
>> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
>> (kafka.producer.async.DefaultEventHandler)
>>
>> I don't get error in sync mode. My message.max.bytes is default
>> (1000000). As I understand the difference between sync and async mode
>> is that async collects batches before sending. Does it mean that batch
>> should be less than message.max.bytes, or any single message within
>> batch should be less than that.
>>
>> Thanks.
>>
>
>
>
> --
> -- Guozhang
Re: message.max.bytes and mirrormaker tool
Posted by Guozhang Wang <wa...@gmail.com>.
Kane,
Did you use any compression method?
Guozhang
On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <ka...@gmail.com> wrote:
> I've tried to run mirrormaker tools in async mode and I get
> WARN Produce request with correlation id 263 failed due to
> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
> (kafka.producer.async.DefaultEventHandler)
>
> I don't get error in sync mode. My message.max.bytes is default
> (1000000). As I understand the difference between sync and async mode
> is that async collects batches before sending. Does it mean that batch
> should be less than message.max.bytes, or any single message within
> batch should be less than that.
>
> Thanks.
>
--
-- Guozhang