You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Pierre Coquentin <pi...@gmail.com> on 2019/04/04 08:52:40 UTC

How to be notified by Kafka stream during partitions rebalancing

Hi,

We have a cache in a processor based on assigned partitions and when Kafka
revokes those partitions we would like to flush the cache. The method
punctuate is neet to do that except that as a client of Kafka stream, I am
not notified during a revoke.
I found the same question on StackOverflow
https://stackoverflow.com/questions/51626089/kafka-streams-consumerrebalancelistener-implementation
but
nothing in Jira. Is it plan to develop this kind of feature? Or perhaps I
missed something?
Regards,

Pierre

Re: How to be notified by Kafka stream during partitions rebalancing

Posted by John Roesler <jo...@confluent.io>.
Hi Pierre,

If you're using a Processor (or Transformer), you might be able to use the
`close` method for this purpose. Streams invokes `close` on the Processor
when it suspends the task at the start of the rebalance, when the
partitions are revoked. (It invokes `init` once the rebalance is complete
and the partition is assigned again.)

Does this help?

Thanks,
-John

On Thu, Apr 4, 2019 at 6:39 AM Pierre Coquentin <pi...@gmail.com>
wrote:

> thanks :)
>
> On Thu, Apr 4, 2019 at 11:11 AM Dimitry Lvovsky <dl...@gmail.com>
> wrote:
>
> > You can detect state changes in your streaming app by implementing
> > KafkaStreams.StateListener,
> > and then registering that with your KafkaStreeams Object e.g new
> > KafkaStreams(...).setStateListener();
> >
> > Hope this helps.
> >
> > On Thu, Apr 4, 2019 at 10:52 AM Pierre Coquentin <
> > pierre.coquentin@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > We have a cache in a processor based on assigned partitions and when
> > Kafka
> > > revokes those partitions we would like to flush the cache. The method
> > > punctuate is neet to do that except that as a client of Kafka stream, I
> > am
> > > not notified during a revoke.
> > > I found the same question on StackOverflow
> > >
> > >
> >
> https://stackoverflow.com/questions/51626089/kafka-streams-consumerrebalancelistener-implementation
> > > but
> > > nothing in Jira. Is it plan to develop this kind of feature? Or
> perhaps I
> > > missed something?
> > > Regards,
> > >
> > > Pierre
> > >
> >
>

Re: How to be notified by Kafka stream during partitions rebalancing

Posted by Pierre Coquentin <pi...@gmail.com>.
thanks :)

On Thu, Apr 4, 2019 at 11:11 AM Dimitry Lvovsky <dl...@gmail.com> wrote:

> You can detect state changes in your streaming app by implementing
> KafkaStreams.StateListener,
> and then registering that with your KafkaStreeams Object e.g new
> KafkaStreams(...).setStateListener();
>
> Hope this helps.
>
> On Thu, Apr 4, 2019 at 10:52 AM Pierre Coquentin <
> pierre.coquentin@gmail.com>
> wrote:
>
> > Hi,
> >
> > We have a cache in a processor based on assigned partitions and when
> Kafka
> > revokes those partitions we would like to flush the cache. The method
> > punctuate is neet to do that except that as a client of Kafka stream, I
> am
> > not notified during a revoke.
> > I found the same question on StackOverflow
> >
> >
> https://stackoverflow.com/questions/51626089/kafka-streams-consumerrebalancelistener-implementation
> > but
> > nothing in Jira. Is it plan to develop this kind of feature? Or perhaps I
> > missed something?
> > Regards,
> >
> > Pierre
> >
>

Re: How to be notified by Kafka stream during partitions rebalancing

Posted by Dimitry Lvovsky <dl...@gmail.com>.
You can detect state changes in your streaming app by implementing
KafkaStreams.StateListener,
and then registering that with your KafkaStreeams Object e.g new
KafkaStreams(...).setStateListener();

Hope this helps.

On Thu, Apr 4, 2019 at 10:52 AM Pierre Coquentin <pi...@gmail.com>
wrote:

> Hi,
>
> We have a cache in a processor based on assigned partitions and when Kafka
> revokes those partitions we would like to flush the cache. The method
> punctuate is neet to do that except that as a client of Kafka stream, I am
> not notified during a revoke.
> I found the same question on StackOverflow
>
> https://stackoverflow.com/questions/51626089/kafka-streams-consumerrebalancelistener-implementation
> but
> nothing in Jira. Is it plan to develop this kind of feature? Or perhaps I
> missed something?
> Regards,
>
> Pierre
>