You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kyle Banker <ky...@gmail.com> on 2014/09/23 00:41:19 UTC
Producer errors (failed to send producer request, failed to send
requests for topics)
I have a test data set of 1500 messages (~2.5 MB each) that I'm using to
test Kafka throughput. I'm pushing this data using 40 Kafka producers, and
I'm losing about 10% of the message on each trial.
I'm seeing errors of the following form:
Failed to send producer request with correlation id 80 to broker 6 with
data for partitions [kafka_end_test_3r,339]
Produce request with correlation id 88 failed due to
[kafka_end_test_3r,317]: kafka.common.RequestTimedOutException
Failed to send requests for topics kafka_end_test_3r with correlation ids
in [15,77]
My topic has 400 partitions and a replication factor of 3. I have a 9-node
broker cluster.
In the producer, I've set request.required.acks to -1, retry.backoff.ms to
500, message.send.max.retries to 20.
What am I doing wrong? How can I make Kafka to accept 100% of messages? Any
help would be greatly appreciated.
Re: Producer errors (failed to send producer request, failed to send
requests for topics)
Posted by Jun Rao <ju...@gmail.com>.
Kyle,
We have developed a new (pure java) producer in trunk. It should have
better error messaging. Could you give it a try and see if it points out
the problem clearer to you?
Thanks,
Jun
On Tue, Sep 23, 2014 at 8:30 PM, Kyle Banker <ky...@gmail.com> wrote:
> Thanks so much, Jun. That seems to have fixed the problem. I increased both
> message.max.bytes and replica.fetch.max.bytes on the broker.
>
> For the benefit of future Kafka users, how hard would it be to build out
> some clearer error messaging for this case?
>
> On Mon, Sep 22, 2014 at 10:38 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Also, don't forget to increase replica.fetch.max.bytes to be larger than
> > the max message size.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Sep 22, 2014 at 9:35 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > What version of Kafka are you using? Have you increased the max message
> > > size on the broker (default to 1MB)?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <ky...@gmail.com>
> > wrote:
> > >
> > >> I have a test data set of 1500 messages (~2.5 MB each) that I'm using
> to
> > >> test Kafka throughput. I'm pushing this data using 40 Kafka producers,
> > and
> > >> I'm losing about 10% of the message on each trial.
> > >>
> > >> I'm seeing errors of the following form:
> > >>
> > >> Failed to send producer request with correlation id 80 to broker 6
> with
> > >> data for partitions [kafka_end_test_3r,339]
> > >> Produce request with correlation id 88 failed due to
> > >> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException
> > >> Failed to send requests for topics kafka_end_test_3r with correlation
> > ids
> > >> in [15,77]
> > >>
> > >> My topic has 400 partitions and a replication factor of 3. I have a
> > 9-node
> > >> broker cluster.
> > >>
> > >> In the producer, I've set request.required.acks to -1,
> retry.backoff.ms
> > >> to
> > >> 500, message.send.max.retries to 20.
> > >>
> > >> What am I doing wrong? How can I make Kafka to accept 100% of
> messages?
> > >> Any
> > >> help would be greatly appreciated.
> > >>
> > >
> > >
> >
>
Re: Producer errors (failed to send producer request, failed to send
requests for topics)
Posted by Guozhang Wang <wa...@gmail.com>.
Kyle, the new producer will handle this exception with the specific
exception type, and the callback handling function can treat it
accordingly. Could you give the new producer client a try and see if that
is better handled now?
On Tue, Sep 23, 2014 at 8:30 PM, Kyle Banker <ky...@gmail.com> wrote:
> Thanks so much, Jun. That seems to have fixed the problem. I increased both
> message.max.bytes and replica.fetch.max.bytes on the broker.
>
> For the benefit of future Kafka users, how hard would it be to build out
> some clearer error messaging for this case?
>
> On Mon, Sep 22, 2014 at 10:38 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Also, don't forget to increase replica.fetch.max.bytes to be larger than
> > the max message size.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Sep 22, 2014 at 9:35 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > What version of Kafka are you using? Have you increased the max message
> > > size on the broker (default to 1MB)?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <ky...@gmail.com>
> > wrote:
> > >
> > >> I have a test data set of 1500 messages (~2.5 MB each) that I'm using
> to
> > >> test Kafka throughput. I'm pushing this data using 40 Kafka producers,
> > and
> > >> I'm losing about 10% of the message on each trial.
> > >>
> > >> I'm seeing errors of the following form:
> > >>
> > >> Failed to send producer request with correlation id 80 to broker 6
> with
> > >> data for partitions [kafka_end_test_3r,339]
> > >> Produce request with correlation id 88 failed due to
> > >> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException
> > >> Failed to send requests for topics kafka_end_test_3r with correlation
> > ids
> > >> in [15,77]
> > >>
> > >> My topic has 400 partitions and a replication factor of 3. I have a
> > 9-node
> > >> broker cluster.
> > >>
> > >> In the producer, I've set request.required.acks to -1,
> retry.backoff.ms
> > >> to
> > >> 500, message.send.max.retries to 20.
> > >>
> > >> What am I doing wrong? How can I make Kafka to accept 100% of
> messages?
> > >> Any
> > >> help would be greatly appreciated.
> > >>
> > >
> > >
> >
>
--
-- Guozhang
Re: Producer errors (failed to send producer request, failed to send
requests for topics)
Posted by Kyle Banker <ky...@gmail.com>.
Thanks so much, Jun. That seems to have fixed the problem. I increased both
message.max.bytes and replica.fetch.max.bytes on the broker.
For the benefit of future Kafka users, how hard would it be to build out
some clearer error messaging for this case?
On Mon, Sep 22, 2014 at 10:38 PM, Jun Rao <ju...@gmail.com> wrote:
> Also, don't forget to increase replica.fetch.max.bytes to be larger than
> the max message size.
>
> Thanks,
>
> Jun
>
> On Mon, Sep 22, 2014 at 9:35 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > What version of Kafka are you using? Have you increased the max message
> > size on the broker (default to 1MB)?
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <ky...@gmail.com>
> wrote:
> >
> >> I have a test data set of 1500 messages (~2.5 MB each) that I'm using to
> >> test Kafka throughput. I'm pushing this data using 40 Kafka producers,
> and
> >> I'm losing about 10% of the message on each trial.
> >>
> >> I'm seeing errors of the following form:
> >>
> >> Failed to send producer request with correlation id 80 to broker 6 with
> >> data for partitions [kafka_end_test_3r,339]
> >> Produce request with correlation id 88 failed due to
> >> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException
> >> Failed to send requests for topics kafka_end_test_3r with correlation
> ids
> >> in [15,77]
> >>
> >> My topic has 400 partitions and a replication factor of 3. I have a
> 9-node
> >> broker cluster.
> >>
> >> In the producer, I've set request.required.acks to -1, retry.backoff.ms
> >> to
> >> 500, message.send.max.retries to 20.
> >>
> >> What am I doing wrong? How can I make Kafka to accept 100% of messages?
> >> Any
> >> help would be greatly appreciated.
> >>
> >
> >
>
Re: Producer errors (failed to send producer request, failed to send
requests for topics)
Posted by Jun Rao <ju...@gmail.com>.
Also, don't forget to increase replica.fetch.max.bytes to be larger than
the max message size.
Thanks,
Jun
On Mon, Sep 22, 2014 at 9:35 PM, Jun Rao <ju...@gmail.com> wrote:
> What version of Kafka are you using? Have you increased the max message
> size on the broker (default to 1MB)?
>
> Thanks,
>
> Jun
>
> On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <ky...@gmail.com> wrote:
>
>> I have a test data set of 1500 messages (~2.5 MB each) that I'm using to
>> test Kafka throughput. I'm pushing this data using 40 Kafka producers, and
>> I'm losing about 10% of the message on each trial.
>>
>> I'm seeing errors of the following form:
>>
>> Failed to send producer request with correlation id 80 to broker 6 with
>> data for partitions [kafka_end_test_3r,339]
>> Produce request with correlation id 88 failed due to
>> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException
>> Failed to send requests for topics kafka_end_test_3r with correlation ids
>> in [15,77]
>>
>> My topic has 400 partitions and a replication factor of 3. I have a 9-node
>> broker cluster.
>>
>> In the producer, I've set request.required.acks to -1, retry.backoff.ms
>> to
>> 500, message.send.max.retries to 20.
>>
>> What am I doing wrong? How can I make Kafka to accept 100% of messages?
>> Any
>> help would be greatly appreciated.
>>
>
>
Re: Producer errors (failed to send producer request, failed to send
requests for topics)
Posted by Jun Rao <ju...@gmail.com>.
What version of Kafka are you using? Have you increased the max message
size on the broker (default to 1MB)?
Thanks,
Jun
On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <ky...@gmail.com> wrote:
> I have a test data set of 1500 messages (~2.5 MB each) that I'm using to
> test Kafka throughput. I'm pushing this data using 40 Kafka producers, and
> I'm losing about 10% of the message on each trial.
>
> I'm seeing errors of the following form:
>
> Failed to send producer request with correlation id 80 to broker 6 with
> data for partitions [kafka_end_test_3r,339]
> Produce request with correlation id 88 failed due to
> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException
> Failed to send requests for topics kafka_end_test_3r with correlation ids
> in [15,77]
>
> My topic has 400 partitions and a replication factor of 3. I have a 9-node
> broker cluster.
>
> In the producer, I've set request.required.acks to -1, retry.backoff.ms to
> 500, message.send.max.retries to 20.
>
> What am I doing wrong? How can I make Kafka to accept 100% of messages? Any
> help would be greatly appreciated.
>