You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by sunil kalva <sa...@gmail.com> on 2015/03/18 17:21:15 UTC
producer queue size
Hi
How do i get the size of the inmemory queue which are holding messages and
ready to send in async producer, i am using new KafkaProducer class in
0.8.2.
Basically instead of dropping the messages silently, i want to avoid
sending messages if the queue is already full. I am using async
KafkaProdcuer class.
Or is there anyother better way to handle this, since i am using async
client i can not catch the exception i think.
--
SunilKalva
Re: producer queue size
Posted by Mayuresh Gharat <gh...@gmail.com>.
I suppose sending is controlled by the linger time and max batch size of
the queue. The messages are sent to kafka when either of these meet.
The new kafkaProducer returns a Future. So its the responsibility of the
application to do a .get() on it to see the success or failure.
Thanks,
Mayuresh
On Wed, Mar 18, 2015 at 2:32 PM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> The setting you want is buffer.memory, but I don't think there's a way to
> get the amount of remaining space.
>
> The setting block.on.buffer.full controls the behavior when you run out of
> space. Neither setting silently drops messages. It will either block until
> there is space to add the message or throw an exception, which your
> application can catch and handle however it wants.
>
> -Ewen
>
> On Wed, Mar 18, 2015 at 9:24 AM, sunil kalva <sa...@gmail.com> wrote:
>
> > essentially i want to use this property "queue.buffering.max.messages"
> with
> > new KafkaProducer class, and also want to access the current value of the
> > queue
> >
> > SunilKalva
> >
> > On Wed, Mar 18, 2015 at 9:51 PM, sunil kalva <sa...@gmail.com>
> wrote:
> >
> > >
> > > Hi
> > > How do i get the size of the inmemory queue which are holding messages
> > and
> > > ready to send in async producer, i am using new KafkaProducer class in
> > > 0.8.2.
> > >
> > > Basically instead of dropping the messages silently, i want to avoid
> > > sending messages if the queue is already full. I am using async
> > > KafkaProdcuer class.
> > >
> > > Or is there anyother better way to handle this, since i am using async
> > > client i can not catch the exception i think.
> > >
> > > --
> > > SunilKalva
> > >
> >
> >
> >
> > --
> > SunilKalva
> >
>
>
>
> --
> Thanks,
> Ewen
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: producer queue size
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
The setting you want is buffer.memory, but I don't think there's a way to
get the amount of remaining space.
The setting block.on.buffer.full controls the behavior when you run out of
space. Neither setting silently drops messages. It will either block until
there is space to add the message or throw an exception, which your
application can catch and handle however it wants.
-Ewen
On Wed, Mar 18, 2015 at 9:24 AM, sunil kalva <sa...@gmail.com> wrote:
> essentially i want to use this property "queue.buffering.max.messages" with
> new KafkaProducer class, and also want to access the current value of the
> queue
>
> SunilKalva
>
> On Wed, Mar 18, 2015 at 9:51 PM, sunil kalva <sa...@gmail.com> wrote:
>
> >
> > Hi
> > How do i get the size of the inmemory queue which are holding messages
> and
> > ready to send in async producer, i am using new KafkaProducer class in
> > 0.8.2.
> >
> > Basically instead of dropping the messages silently, i want to avoid
> > sending messages if the queue is already full. I am using async
> > KafkaProdcuer class.
> >
> > Or is there anyother better way to handle this, since i am using async
> > client i can not catch the exception i think.
> >
> > --
> > SunilKalva
> >
>
>
>
> --
> SunilKalva
>
--
Thanks,
Ewen
Re: producer queue size
Posted by sunil kalva <sa...@gmail.com>.
essentially i want to use this property "queue.buffering.max.messages" with
new KafkaProducer class, and also want to access the current value of the
queue
SunilKalva
On Wed, Mar 18, 2015 at 9:51 PM, sunil kalva <sa...@gmail.com> wrote:
>
> Hi
> How do i get the size of the inmemory queue which are holding messages and
> ready to send in async producer, i am using new KafkaProducer class in
> 0.8.2.
>
> Basically instead of dropping the messages silently, i want to avoid
> sending messages if the queue is already full. I am using async
> KafkaProdcuer class.
>
> Or is there anyother better way to handle this, since i am using async
> client i can not catch the exception i think.
>
> --
> SunilKalva
>
--
SunilKalva