You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by pradeep s <sr...@gmail.com> on 2018/10/16 21:19:08 UTC
Kafka commit interval
Hi,
I have a usecase to stream messages from Kafka and buffer it in memory till
a message count is reached and then write these to output file . I am using
manual commit . I have a question on whats the maximum time i can wait
after consuming the message and till we commit back to Kafka . Is there a
upper limit for this ? Is consumer rebalance the only issue while waiting
long periods, then can i write the contents using a rebalance listener and
commit at that point .
Thanks
Pradeep
Re: Kafka commit interval
Posted by pradeep s <sr...@gmail.com>.
Thanks Matthias
On Fri, Oct 19, 2018 at 11:25 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> There is not upper limit.
>
> And yes, you are right about rebalancing. This would be an issue and
> yes, you can use the rebalance listener to address it (it's the purpose
> of the rebalance listener to be used for cases like this).
>
> -Matthias
>
>
> On 10/16/18 2:19 PM, pradeep s wrote:
> > Hi,
> > I have a usecase to stream messages from Kafka and buffer it in memory
> till
> > a message count is reached and then write these to output file . I am
> using
> > manual commit . I have a question on whats the maximum time i can wait
> > after consuming the message and till we commit back to Kafka . Is there a
> > upper limit for this ? Is consumer rebalance the only issue while waiting
> > long periods, then can i write the contents using a rebalance listener
> and
> > commit at that point .
> > Thanks
> > Pradeep
> >
>
>
Re: Kafka commit interval
Posted by "Matthias J. Sax" <ma...@confluent.io>.
There is not upper limit.
And yes, you are right about rebalancing. This would be an issue and
yes, you can use the rebalance listener to address it (it's the purpose
of the rebalance listener to be used for cases like this).
-Matthias
On 10/16/18 2:19 PM, pradeep s wrote:
> Hi,
> I have a usecase to stream messages from Kafka and buffer it in memory till
> a message count is reached and then write these to output file . I am using
> manual commit . I have a question on whats the maximum time i can wait
> after consuming the message and till we commit back to Kafka . Is there a
> upper limit for this ? Is consumer rebalance the only issue while waiting
> long periods, then can i write the contents using a rebalance listener and
> commit at that point .
> Thanks
> Pradeep
>