You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yifan Ying <na...@gmail.com> on 2016/09/28 04:21:58 UTC
How to keep consumers alive without polling new messages
Hi all,
0.10 consumers use poll() method to heartbeat Kafka brokers. Is there any
way that I can make the consumer heartbeat but not poll any messages? The
javadoc says, the recommended way is to move message processing to another
thread. But when message processing keeps failing(because a third party
service goes down for a while), the thread that actually processes messages
could have too many messages accumulated. Maybe re-sending failed messages
to another queue(IMQ) and re-processing them later is a good option?
Thanks!
--
Yifan
Re: How to keep consumers alive without polling new messages
Posted by Yifan Ying <na...@gmail.com>.
Thanks Vincent and Kamal. That solves my problem. :)
Yifan
On Tue, Sep 27, 2016 at 11:36 PM, Kamal C <ka...@gmail.com> wrote:
> You can refer this example[1]
>
> [1]:
> https://github.com/omkreddy/kafka-examples/blob/master/
> consumer/src/main/java/kafka/examples/consumer/advanced/
> AdvancedConsumer.java
>
> - Kamal
>
> On Wed, Sep 28, 2016 at 11:33 AM, Vincent Dautremont <
> vincent.dautremont@olamobile.com> wrote:
>
> > I had the same problem :
> > Call pause() on all partitions.
> > Then continue your loop that calls consume(), it will then poll without
> > consuming messages.
> >
> > When you want to consume again, call resume() on all partition
> >
> > It's not obvious at all, the doc should explain that in the documentation
> > of method consume() and put it in red and bold.
> > https://kafka.apache.org/090/javadoc/index.html?org/apache/
> > kafka/clients/consumer/KafkaConsumer.html
> >
> > > Le 28 sept. 2016 à 06:21, Yifan Ying <na...@gmail.com> a écrit :
> > >
> > > Hi all,
> > >
> > > 0.10 consumers use poll() method to heartbeat Kafka brokers. Is there
> any
> > > way that I can make the consumer heartbeat but not poll any messages?
> The
> > > javadoc says, the recommended way is to move message processing to
> > another
> > > thread. But when message processing keeps failing(because a third party
> > > service goes down for a while), the thread that actually processes
> > messages
> > > could have too many messages accumulated. Maybe re-sending failed
> > messages
> > > to another queue(IMQ) and re-processing them later is a good option?
> > >
> > > Thanks!
> > > --
> > > Yifan
> >
>
--
Yifan
Re: How to keep consumers alive without polling new messages
Posted by Kamal C <ka...@gmail.com>.
You can refer this example[1]
[1]:
https://github.com/omkreddy/kafka-examples/blob/master/consumer/src/main/java/kafka/examples/consumer/advanced/AdvancedConsumer.java
- Kamal
On Wed, Sep 28, 2016 at 11:33 AM, Vincent Dautremont <
vincent.dautremont@olamobile.com> wrote:
> I had the same problem :
> Call pause() on all partitions.
> Then continue your loop that calls consume(), it will then poll without
> consuming messages.
>
> When you want to consume again, call resume() on all partition
>
> It's not obvious at all, the doc should explain that in the documentation
> of method consume() and put it in red and bold.
> https://kafka.apache.org/090/javadoc/index.html?org/apache/
> kafka/clients/consumer/KafkaConsumer.html
>
> > Le 28 sept. 2016 à 06:21, Yifan Ying <na...@gmail.com> a écrit :
> >
> > Hi all,
> >
> > 0.10 consumers use poll() method to heartbeat Kafka brokers. Is there any
> > way that I can make the consumer heartbeat but not poll any messages? The
> > javadoc says, the recommended way is to move message processing to
> another
> > thread. But when message processing keeps failing(because a third party
> > service goes down for a while), the thread that actually processes
> messages
> > could have too many messages accumulated. Maybe re-sending failed
> messages
> > to another queue(IMQ) and re-processing them later is a good option?
> >
> > Thanks!
> > --
> > Yifan
>
Re: How to keep consumers alive without polling new messages
Posted by Vincent Dautremont <vi...@olamobile.com>.
I had the same problem :
Call pause() on all partitions.
Then continue your loop that calls consume(), it will then poll without consuming messages.
When you want to consume again, call resume() on all partition
It's not obvious at all, the doc should explain that in the documentation of method consume() and put it in red and bold.
https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
> Le 28 sept. 2016 à 06:21, Yifan Ying <na...@gmail.com> a écrit :
>
> Hi all,
>
> 0.10 consumers use poll() method to heartbeat Kafka brokers. Is there any
> way that I can make the consumer heartbeat but not poll any messages? The
> javadoc says, the recommended way is to move message processing to another
> thread. But when message processing keeps failing(because a third party
> service goes down for a while), the thread that actually processes messages
> could have too many messages accumulated. Maybe re-sending failed messages
> to another queue(IMQ) and re-processing them later is a good option?
>
> Thanks!
> --
> Yifan