You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Amit K <am...@gmail.com> on 2016/11/06 09:30:11 UTC

Re-consume messages

Hi All,

I am using kafka 0.9.0.1 with high level java producer and consumer.
I need to handle a case wherein I need to re-consume the already consumed
(and processed) messages say for last 5 days (configurable).

Is there any way of achieving the same apart from identifying the offsets
for the partition (for given topic) and process the same?

For the records, I am not storing the offsets processed.

Thanks

Re: Re-consume messages

Posted by Amit K <am...@gmail.com>.

Thanks for your reply.

Will give it a shot.

On Sun, Nov 6, 2016 at 9:38 PM, Becket Qin <be...@gmail.com> wrote:

> Hi Amit,
>
> In Kafka 0.9, the closest approach would be use
> SimpleConsumer.getOffsetBefore() to search the offset by timestamp.  And
> then you can consume from the returned biggest offset. Notice that the
> search is at log segment level and the result may not be accurate if the
> partition has been moved. In the worst case, you may consume a lot more
> messages than you want to, but you should not miss any messages.
>
> Jiangjie (Becket) Qin
>
> On Sun, Nov 6, 2016 at 1:30 AM, Amit K <am...@gmail.com> wrote:
>
> > Hi All,
> >
> > I am using kafka 0.9.0.1 with high level java producer and consumer.
> > I need to handle a case wherein I need to re-consume the already consumed
> > (and processed) messages say for last 5 days (configurable).
> >
> > Is there any way of achieving the same apart from identifying the offsets
> > for the partition (for given topic) and process the same?
> >
> > For the records, I am not storing the offsets processed.
> >
> > Thanks
> >
>

Re: Re-consume messages

Posted by Becket Qin <be...@gmail.com>.

Hi Amit,

In Kafka 0.9, the closest approach would be use
SimpleConsumer.getOffsetBefore() to search the offset by timestamp.  And
then you can consume from the returned biggest offset. Notice that the
search is at log segment level and the result may not be accurate if the
partition has been moved. In the worst case, you may consume a lot more
messages than you want to, but you should not miss any messages.

Jiangjie (Becket) Qin

On Sun, Nov 6, 2016 at 1:30 AM, Amit K <am...@gmail.com> wrote:

> Hi All,
>
> I am using kafka 0.9.0.1 with high level java producer and consumer.
> I need to handle a case wherein I need to re-consume the already consumed
> (and processed) messages say for last 5 days (configurable).
>
> Is there any way of achieving the same apart from identifying the offsets
> for the partition (for given topic) and process the same?
>
> For the records, I am not storing the offsets processed.
>
> Thanks
>