You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by David Espinosa <es...@gmail.com> on 2018/09/14 07:46:19 UTC

Best way for reading all messages and close

Hi all,

Although the usage of Kafka is stream oriented, for a concrete use case I
need to read all the messages existing in a topic and once all them has
been read then closing the consumer.

What's the best way or framework for doing this?

Thanks in advance,
David,

Re: Best way for reading all messages and close

Posted by John Roesler <jo...@confluent.io>.
Yep, that should also work!
-John

On Mon, Sep 17, 2018 at 8:36 AM David Espinosa <es...@gmail.com> wrote:

> Thank you all for your responses!
> I also asked this on the confluent slack channel (
> https://confluentcommunity.slack.com) and I got this approach:
>
>    1. Query the partitions' high watermark offset
>    2. Set the consumer to consume from beginning
>    3. Break out when you've reached the high offset
>
> Still have some doubts regarding the implementation, but it seems a good
> approach (I'm using a single partition so a single loop would be enough per
> topic).
> What do you think?
>
> El sáb., 15 sept. 2018 a las 0:30, John Roesler (<jo...@confluent.io>)
> escribió:
>
> > Specifically, you can monitor the "records-lag-max" (
> > https://docs.confluent.io/current/kafka/monitoring.html#fetch-metrics)
> > metric. (or the more granular one per partition).
> >
> > Once this metric goes to 0, you know that you've caught up with the tail
> of
> > the log.
> >
> > Hope this helps,
> > -John
> >
> > On Fri, Sep 14, 2018 at 2:02 PM Matthias J. Sax <ma...@confluent.io>
> > wrote:
> >
> > > Using Kafka Streams this is a little tricky.
> > >
> > > The API itself has no built-in mechanism to do this. You would need to
> > > monitor the lag of the application, and if the lag is zero (assuming
> you
> > > don't write new data into the topic in parallel), terminate the
> > > application.
> > >
> > >
> > > -Matthias
> > >
> > > On 9/14/18 4:19 AM, Henning Røigaard-Petersen wrote:
> > > > Spin up a consumer, subscribe to EOF events, assign all partitions
> from
> > > the beginning, and keep polling until all partitions has reached EOF.
> > > > Though, if you have concurrent writers, new messages may be appended
> > > after you observe EOF on a partition, so you are never guaranteed to
> have
> > > read all messages at the time you choose to close the consumer.
> > > >
> > > > /Henning Røigaard-Petersen
> > > >
> > > > -----Original Message-----
> > > > From: David Espinosa <es...@gmail.com>
> > > > Sent: 14. september 2018 09:46
> > > > To: users@kafka.apache.org
> > > > Subject: Best way for reading all messages and close
> > > >
> > > > Hi all,
> > > >
> > > > Although the usage of Kafka is stream oriented, for a concrete use
> case
> > > I need to read all the messages existing in a topic and once all them
> has
> > > been read then closing the consumer.
> > > >
> > > > What's the best way or framework for doing this?
> > > >
> > > > Thanks in advance,
> > > > David,
> > > >
> > >
> > >
> >
>

Re: Best way for reading all messages and close

Posted by David Espinosa <es...@gmail.com>.
Thank you all for your responses!
I also asked this on the confluent slack channel (
https://confluentcommunity.slack.com) and I got this approach:

   1. Query the partitions' high watermark offset
   2. Set the consumer to consume from beginning
   3. Break out when you've reached the high offset

Still have some doubts regarding the implementation, but it seems a good
approach (I'm using a single partition so a single loop would be enough per
topic).
What do you think?

El sáb., 15 sept. 2018 a las 0:30, John Roesler (<jo...@confluent.io>)
escribió:

> Specifically, you can monitor the "records-lag-max" (
> https://docs.confluent.io/current/kafka/monitoring.html#fetch-metrics)
> metric. (or the more granular one per partition).
>
> Once this metric goes to 0, you know that you've caught up with the tail of
> the log.
>
> Hope this helps,
> -John
>
> On Fri, Sep 14, 2018 at 2:02 PM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
> > Using Kafka Streams this is a little tricky.
> >
> > The API itself has no built-in mechanism to do this. You would need to
> > monitor the lag of the application, and if the lag is zero (assuming you
> > don't write new data into the topic in parallel), terminate the
> > application.
> >
> >
> > -Matthias
> >
> > On 9/14/18 4:19 AM, Henning Røigaard-Petersen wrote:
> > > Spin up a consumer, subscribe to EOF events, assign all partitions from
> > the beginning, and keep polling until all partitions has reached EOF.
> > > Though, if you have concurrent writers, new messages may be appended
> > after you observe EOF on a partition, so you are never guaranteed to have
> > read all messages at the time you choose to close the consumer.
> > >
> > > /Henning Røigaard-Petersen
> > >
> > > -----Original Message-----
> > > From: David Espinosa <es...@gmail.com>
> > > Sent: 14. september 2018 09:46
> > > To: users@kafka.apache.org
> > > Subject: Best way for reading all messages and close
> > >
> > > Hi all,
> > >
> > > Although the usage of Kafka is stream oriented, for a concrete use case
> > I need to read all the messages existing in a topic and once all them has
> > been read then closing the consumer.
> > >
> > > What's the best way or framework for doing this?
> > >
> > > Thanks in advance,
> > > David,
> > >
> >
> >
>

Re: Best way for reading all messages and close

Posted by John Roesler <jo...@confluent.io>.
Specifically, you can monitor the "records-lag-max" (
https://docs.confluent.io/current/kafka/monitoring.html#fetch-metrics)
metric. (or the more granular one per partition).

Once this metric goes to 0, you know that you've caught up with the tail of
the log.

Hope this helps,
-John

On Fri, Sep 14, 2018 at 2:02 PM Matthias J. Sax <ma...@confluent.io>
wrote:

> Using Kafka Streams this is a little tricky.
>
> The API itself has no built-in mechanism to do this. You would need to
> monitor the lag of the application, and if the lag is zero (assuming you
> don't write new data into the topic in parallel), terminate the
> application.
>
>
> -Matthias
>
> On 9/14/18 4:19 AM, Henning Røigaard-Petersen wrote:
> > Spin up a consumer, subscribe to EOF events, assign all partitions from
> the beginning, and keep polling until all partitions has reached EOF.
> > Though, if you have concurrent writers, new messages may be appended
> after you observe EOF on a partition, so you are never guaranteed to have
> read all messages at the time you choose to close the consumer.
> >
> > /Henning Røigaard-Petersen
> >
> > -----Original Message-----
> > From: David Espinosa <es...@gmail.com>
> > Sent: 14. september 2018 09:46
> > To: users@kafka.apache.org
> > Subject: Best way for reading all messages and close
> >
> > Hi all,
> >
> > Although the usage of Kafka is stream oriented, for a concrete use case
> I need to read all the messages existing in a topic and once all them has
> been read then closing the consumer.
> >
> > What's the best way or framework for doing this?
> >
> > Thanks in advance,
> > David,
> >
>
>

Re: Best way for reading all messages and close

Posted by "Matthias J. Sax" <ma...@confluent.io>.
Using Kafka Streams this is a little tricky.

The API itself has no built-in mechanism to do this. You would need to
monitor the lag of the application, and if the lag is zero (assuming you
don't write new data into the topic in parallel), terminate the application.


-Matthias

On 9/14/18 4:19 AM, Henning Røigaard-Petersen wrote:
> Spin up a consumer, subscribe to EOF events, assign all partitions from the beginning, and keep polling until all partitions has reached EOF.
> Though, if you have concurrent writers, new messages may be appended after you observe EOF on a partition, so you are never guaranteed to have read all messages at the time you choose to close the consumer.
> 
> /Henning Røigaard-Petersen
> 
> -----Original Message-----
> From: David Espinosa <es...@gmail.com> 
> Sent: 14. september 2018 09:46
> To: users@kafka.apache.org
> Subject: Best way for reading all messages and close
> 
> Hi all,
> 
> Although the usage of Kafka is stream oriented, for a concrete use case I need to read all the messages existing in a topic and once all them has been read then closing the consumer.
> 
> What's the best way or framework for doing this?
> 
> Thanks in advance,
> David,
> 


RE: Best way for reading all messages and close

Posted by Henning Røigaard-Petersen <hr...@edlund.dk>.
Spin up a consumer, subscribe to EOF events, assign all partitions from the beginning, and keep polling until all partitions has reached EOF.
Though, if you have concurrent writers, new messages may be appended after you observe EOF on a partition, so you are never guaranteed to have read all messages at the time you choose to close the consumer.

/Henning Røigaard-Petersen

-----Original Message-----
From: David Espinosa <es...@gmail.com> 
Sent: 14. september 2018 09:46
To: users@kafka.apache.org
Subject: Best way for reading all messages and close

Hi all,

Although the usage of Kafka is stream oriented, for a concrete use case I need to read all the messages existing in a topic and once all them has been read then closing the consumer.

What's the best way or framework for doing this?

Thanks in advance,
David,