You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Eduardo Costa Alfaia <e....@unibs.it> on 2015/10/26 16:49:20 UTC
Queue Full
Hi Guys,
How could I solving this problem?
% Failed to produce message: Local: Queue full
% Failed to produce message: Local: Queue full
Thanks
--
Informativa sulla Privacy: http://www.unibs.it/node/8155
Re: Queue Full
Posted by 马哲超 <ma...@gmail.com>.
You can try to set "queue.buffering.max.messages" larger, for example:
confParam=["queue.buffering.max.messages=2000000",
"batch.num.messages=1000"]
And reference for you:
https://github.com/edenhill/librdkafka/issues/210
2015-10-27 1:19 GMT+08:00 Prabhjot Bharaj <pr...@gmail.com>:
> Hi,
>
> This is a type of problem where you operating more than network capacity.
> This can be handled at two places (you decide whichever is useful/practical
> for use case) :-
>
> 1. In case the bottleneck is because of the broker slowness, increase the
> number of partitions of your topic, balance them out to newer nodes (if
> required) and retry. Make sure your producer can write to the new
> partitions as well
> 2. In case this is because your producer is creating too many messages,
> there are 2 ways you can commit them (select the most appropriate according
> to your use case. I'm considering that you dont want to lose your
> messages):-
> ------> a. increase the number of producer machines (i.e. balance your
> producer load among more machines)
> -------> b. produce at a slower rate (in case your producers are not
> scalable, you are in a problem. you can solve this problem by sacrificing
> latency). Write to your queue in batches and wait for it to get cleared
> before you commit the next batch to your local queue.
>
> Regards,
> Prabcs
>
>
> On Mon, Oct 26, 2015 at 10:41 PM, Eduardo Costa Alfaia <
> e.costaalfaia@unibs.it> wrote:
>
> > Hi Magnus
> > I think this answer
> > c) producing messages at a higher rate than the network or broker can
> > handle
> > How could I manager this?
> >
> >
> > > On 26 Oct 2015, at 17:45, Magnus Edenhill <ma...@edenhill.se> wrote:
> > >
> > > c) producing messages at a higher rate than the network or broker can
> > > handle
> >
> >
> > --
> > Informativa sulla Privacy: http://www.unibs.it/node/8155
> >
>
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>
Re: Queue Full
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi,
This is a type of problem where you operating more than network capacity.
This can be handled at two places (you decide whichever is useful/practical
for use case) :-
1. In case the bottleneck is because of the broker slowness, increase the
number of partitions of your topic, balance them out to newer nodes (if
required) and retry. Make sure your producer can write to the new
partitions as well
2. In case this is because your producer is creating too many messages,
there are 2 ways you can commit them (select the most appropriate according
to your use case. I'm considering that you dont want to lose your
messages):-
------> a. increase the number of producer machines (i.e. balance your
producer load among more machines)
-------> b. produce at a slower rate (in case your producers are not
scalable, you are in a problem. you can solve this problem by sacrificing
latency). Write to your queue in batches and wait for it to get cleared
before you commit the next batch to your local queue.
Regards,
Prabcs
On Mon, Oct 26, 2015 at 10:41 PM, Eduardo Costa Alfaia <
e.costaalfaia@unibs.it> wrote:
> Hi Magnus
> I think this answer
> c) producing messages at a higher rate than the network or broker can
> handle
> How could I manager this?
>
>
> > On 26 Oct 2015, at 17:45, Magnus Edenhill <ma...@edenhill.se> wrote:
> >
> > c) producing messages at a higher rate than the network or broker can
> > handle
>
>
> --
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Queue Full
Posted by Eduardo Costa Alfaia <e....@unibs.it>.
Hi Magnus
I think this answer
c) producing messages at a higher rate than the network or broker can
handle
How could I manager this?
> On 26 Oct 2015, at 17:45, Magnus Edenhill <ma...@edenhill.se> wrote:
>
> c) producing messages at a higher rate than the network or broker can
> handle
--
Informativa sulla Privacy: http://www.unibs.it/node/8155
Re: Queue Full
Posted by Magnus Edenhill <ma...@edenhill.se>.
That's librdkafka's producer queue that has run full because:
a) client is not connected to broker
b) no leader broker for relevant partitions
c) producing messages at a higher rate than the network or broker can
handle
/Magnus
2015-10-26 16:49 GMT+01:00 Eduardo Costa Alfaia <e....@unibs.it>:
> Hi Guys,
>
> How could I solving this problem?
>
> % Failed to produce message: Local: Queue full
> % Failed to produce message: Local: Queue full
>
> Thanks
>
> --
> Informativa sulla Privacy: http://www.unibs.it/node/8155
>