You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yonghui Zhao <zh...@gmail.com> on 2015/11/19 10:15:36 UTC
offset consume in JAVA api
Hi,
I found in 0.9,
def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition,
OffsetAndMetadata], retryOnFailure: Boolean)
is added to ZookeeperConsumerConnector.
If I want to read data backwards or forwards, can I use this API to commit
new offset and then read data from new offset?
This api has same effect as Seek in KafkaConsumer?
In 0.8.1.1, there is no such api. If we don't
I think I can implement this feature by simply rewriting partition consumer
offset on zk, right?
Re: offset consume in JAVA api
Posted by Yonghui Zhao <zh...@gmail.com>.
Thanks Guozhang.
In our scenario, each one group only has 1 consumer, so it is safe to do
this.
2015-11-21 8:21 GMT+08:00 Guozhang Wang <wa...@gmail.com>:
> Yonghui,
>
> You can use ZookeeperConsumerConnector.commitOffsets() to commit arbitrary
> offsets, but be careful using it to seek forward / backward: you need to
> make sure everyone reads the committed offsets right after it is written
> (e.g. force a rebalance), and no one else override it beforehand.
>
> KafkaConsumer.seek() will only allow seeking on the assigned partitions,
> and requires no coordination between consumers, hence is lighter-weight.
>
>
> Guozhang
>
>
> On Thu, Nov 19, 2015 at 1:15 AM, Yonghui Zhao <zh...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I found in 0.9,
> >
> > def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition,
> > OffsetAndMetadata], retryOnFailure: Boolean)
> > is added to ZookeeperConsumerConnector.
> >
> >
> > If I want to read data backwards or forwards, can I use this API to
> commit
> > new offset and then read data from new offset?
> >
> >
> > This api has same effect as Seek in KafkaConsumer?
> >
> >
> > In 0.8.1.1, there is no such api. If we don't
> >
> > I think I can implement this feature by simply rewriting partition
> consumer
> > offset on zk, right?
> >
>
>
>
> --
> -- Guozhang
>
Re: offset consume in JAVA api
Posted by Guozhang Wang <wa...@gmail.com>.
Yonghui,
You can use ZookeeperConsumerConnector.commitOffsets() to commit arbitrary
offsets, but be careful using it to seek forward / backward: you need to
make sure everyone reads the committed offsets right after it is written
(e.g. force a rebalance), and no one else override it beforehand.
KafkaConsumer.seek() will only allow seeking on the assigned partitions,
and requires no coordination between consumers, hence is lighter-weight.
Guozhang
On Thu, Nov 19, 2015 at 1:15 AM, Yonghui Zhao <zh...@gmail.com> wrote:
> Hi,
>
> I found in 0.9,
>
> def commitOffsets(offsetsToCommit: java.util.Map[TopicAndPartition,
> OffsetAndMetadata], retryOnFailure: Boolean)
> is added to ZookeeperConsumerConnector.
>
>
> If I want to read data backwards or forwards, can I use this API to commit
> new offset and then read data from new offset?
>
>
> This api has same effect as Seek in KafkaConsumer?
>
>
> In 0.8.1.1, there is no such api. If we don't
>
> I think I can implement this feature by simply rewriting partition consumer
> offset on zk, right?
>
--
-- Guozhang