You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Helin Xiang <xk...@gmail.com> on 2014/12/08 11:59:13 UTC
In what condition does getoffsetbefore (latest) not return latest offset?
Hi,
We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
In one of our application, we want to get all partitions' latest offsets,
so we use getoffsetbefore java API (latest).
We believe at some time, 1 of the partition's latest offset we got is much
smaller than its real latest offset,(we saw in the application's log that
the partition's offset is much smaller than other partitions'). Since the
real data file of that partition was already deleted, we cannot prove our
guess, but we found some clue in the kafka's application log which helps us
to conclude that the partition's latest offset at that moment did have a
much larger number.
some additional useful information: the partition have 1 additional
replica(follower), and at that time, it was not synced with the leader
partition(far away behind the leader).
Does any one have the same issue? In what condition could lead to this
situation?
Thanks.
--
*Best Regards向河林*
Re: In what condition does getoffsetbefore (latest) not return latest offset?
Posted by Helin Xiang <xk...@gmail.com>.
Hi, Jun
The getOffsetBefore API only return the offset array, i can't find anywhere
to get the error code.
Thanks.
On Sat, Dec 13, 2014 at 1:02 AM, Jun Rao <ju...@confluent.io> wrote:
>
> Yes, for regular consumers, we always return the high watermark as the
> latest offset. Did you check the error code in the getOffsetBefore request?
>
> Thanks,
>
> Jun
>
> On Tue, Dec 9, 2014 at 12:31 AM, 向河林 <xi...@mediav.com> wrote:
>
> > Thanks Guozhang，
> >
> > I checked the server log, and I am 90% sure that no leader movement
> > happened at that moment.
> >
> > Could it be that the data was in the memory but yet not flushed to the
> > disk?
> > I read the doc in the wiki
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication,
> > and said
> >
> > The high watermark (HW) is the offset of the last committed message.
> > Each log is periodically synced to disks. Data before the flushed
> > offset is guaranteed to be persisted on disks. As we will see, the
> > flush offset can be before or after HW.
> >
> > since the https://issues.apache.org/jira/browse/KAFKA-501 said
> >
> > For regular consumers, getOffset should return highWatermark as the
> > latest offset.
> >
> > Thanks again.
> >
> >
> > On Tue, Dec 9, 2014 at 5:28 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Helin,
> > >
> > > Is there a leader movement just before the get latest offset call? If
> > your
> > > follower is not synced and it then becomes the leader due to some
> reason,
> > > it will not have the complete partition data.
> > >
> > > Guozhang
> > >
> > > On Mon, Dec 8, 2014 at 3:02 AM, Helin Xiang <xk...@gmail.com> wrote:
> > >
> > > > 1 additional information we found in the kafka’s application log
> since
> > > the
> > > > MAGIC time:
> > > >
> > > > 2014-12-04 09:59:36,726 [kafka-scheduler-2] INFO
> > > > kafka.cluster.Partition - Partition [a.s.3,26] on broker 5:
> Shrinking
> > > > ISR for partition [a.s.3,26] from 5,4 to 5
> > > > 2014-12-04 09:59:36,728 [kafka-scheduler-2] ERROR
> kafka.utils.ZkUtils$
> > > > - Conditional update of path
> > > > /brokers/topics/a.s.3/partitions/26/state with data
> > > >
> > {"controller_epoch":2,"leader":5,"version":1,"leader_epoch":4,"isr":[5]}
> > > > and expected version 675 failed due to
> > > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > > KeeperErrorCode = BadVersion for
> > > > /brokers/topics/a.s.3/partitions/26/state
> > > >
> > > >
> > > >
> > > > On Mon, Dec 8, 2014 at 6:59 PM, Helin Xiang <xk...@gmail.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
> > > > >
> > > > > In one of our application, we want to get all partitions' latest
> > > offsets,
> > > > > so we use getoffsetbefore java API (latest).
> > > > >
> > > > > We believe at some time, 1 of the partition's latest offset we got
> is
> > > > much
> > > > > smaller than its real latest offset,(we saw in the application's
> log
> > > that
> > > > > the partition's offset is much smaller than other partitions').
> Since
> > > the
> > > > > real data file of that partition was already deleted, we cannot
> prove
> > > our
> > > > > guess, but we found some clue in the kafka's application log which
> > > helps
> > > > us
> > > > > to conclude that the partition's latest offset at that moment did
> > have
> > > a
> > > > > much larger number.
> > > > >
> > > > > some additional useful information: the partition have 1 additional
> > > > > replica(follower), and at that time, it was not synced with the
> > leader
> > > > > partition(far away behind the leader).
> > > > >
> > > > > Does any one have the same issue? In what condition could lead to
> > this
> > > > > situation?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > --
> > > > >
> > > > >
> > > > > *Best Regards向河林*
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > >
> > > > *Best Regards向河林*
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> >
> > *向河林*
> >
> >
> > *MV AD **聚效广告* *上海* · 北京 · 广州 · 杭州
> > _______________________________________________________________
> >
> > 上海市闸北区天目中路585号新梅大厦4楼 200070
> >
> > MOB：18621368491
> >
> > TEL：021-52559088（分机：8133）
> >
> > EMAIL：xianghl@mediav.com <ya...@mediav.com>
> >
> > HTTP：www.mediav.com
> >
> >
> > -------------------------------CONFIDENTIAL
> ------------------------------
> > ------------
> >
> > 本邮件载有秘密信息，请您恪守保密义务，勿向第三人透露。谢谢合作。
> >
> > This email communication is confidential. Recipient(s) named above
> is(are)
> > obligated to maintain secrecy and is(are) not permitted to disclose the
> > contents of this communication to others. Thank you.
> >
>
--
*Best Regards向河林*
Re: In what condition does getoffsetbefore (latest) not return latest offset?
Posted by Jun Rao <ju...@confluent.io>.
Yes, for regular consumers, we always return the high watermark as the
latest offset. Did you check the error code in the getOffsetBefore request?
Thanks,
Jun
On Tue, Dec 9, 2014 at 12:31 AM, 向河林 <xi...@mediav.com> wrote:
> Thanks Guozhang，
>
> I checked the server log, and I am 90% sure that no leader movement
> happened at that moment.
>
> Could it be that the data was in the memory but yet not flushed to the
> disk?
> I read the doc in the wiki
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication,
> and said
>
> The high watermark (HW) is the offset of the last committed message.
> Each log is periodically synced to disks. Data before the flushed
> offset is guaranteed to be persisted on disks. As we will see, the
> flush offset can be before or after HW.
>
> since the https://issues.apache.org/jira/browse/KAFKA-501 said
>
> For regular consumers, getOffset should return highWatermark as the
> latest offset.
>
> Thanks again.
>
>
> On Tue, Dec 9, 2014 at 5:28 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Helin,
> >
> > Is there a leader movement just before the get latest offset call? If
> your
> > follower is not synced and it then becomes the leader due to some reason,
> > it will not have the complete partition data.
> >
> > Guozhang
> >
> > On Mon, Dec 8, 2014 at 3:02 AM, Helin Xiang <xk...@gmail.com> wrote:
> >
> > > 1 additional information we found in the kafka’s application log since
> > the
> > > MAGIC time:
> > >
> > > 2014-12-04 09:59:36,726 [kafka-scheduler-2] INFO
> > > kafka.cluster.Partition - Partition [a.s.3,26] on broker 5: Shrinking
> > > ISR for partition [a.s.3,26] from 5,4 to 5
> > > 2014-12-04 09:59:36,728 [kafka-scheduler-2] ERROR kafka.utils.ZkUtils$
> > > - Conditional update of path
> > > /brokers/topics/a.s.3/partitions/26/state with data
> > >
> {"controller_epoch":2,"leader":5,"version":1,"leader_epoch":4,"isr":[5]}
> > > and expected version 675 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > KeeperErrorCode = BadVersion for
> > > /brokers/topics/a.s.3/partitions/26/state
> > >
> > >
> > >
> > > On Mon, Dec 8, 2014 at 6:59 PM, Helin Xiang <xk...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
> > > >
> > > > In one of our application, we want to get all partitions' latest
> > offsets,
> > > > so we use getoffsetbefore java API (latest).
> > > >
> > > > We believe at some time, 1 of the partition's latest offset we got is
> > > much
> > > > smaller than its real latest offset,(we saw in the application's log
> > that
> > > > the partition's offset is much smaller than other partitions'). Since
> > the
> > > > real data file of that partition was already deleted, we cannot prove
> > our
> > > > guess, but we found some clue in the kafka's application log which
> > helps
> > > us
> > > > to conclude that the partition's latest offset at that moment did
> have
> > a
> > > > much larger number.
> > > >
> > > > some additional useful information: the partition have 1 additional
> > > > replica(follower), and at that time, it was not synced with the
> leader
> > > > partition(far away behind the leader).
> > > >
> > > > Does any one have the same issue? In what condition could lead to
> this
> > > > situation?
> > > >
> > > > Thanks.
> > > >
> > > > --
> > > >
> > > >
> > > > *Best Regards向河林*
> > > >
> > >
> > >
> > >
> > > --
> > >
> > >
> > > *Best Regards向河林*
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
> *向河林*
>
>
> *MV AD **聚效广告* *上海* · 北京 · 广州 · 杭州
> _______________________________________________________________
>
> 上海市闸北区天目中路585号新梅大厦4楼 200070
>
> MOB：18621368491
>
> TEL：021-52559088（分机：8133）
>
> EMAIL：xianghl@mediav.com <ya...@mediav.com>
>
> HTTP：www.mediav.com
>
>
> -------------------------------CONFIDENTIAL ------------------------------
> ------------
>
> 本邮件载有秘密信息，请您恪守保密义务，勿向第三人透露。谢谢合作。
>
> This email communication is confidential. Recipient(s) named above is(are)
> obligated to maintain secrecy and is(are) not permitted to disclose the
> contents of this communication to others. Thank you.
>
Re: In what condition does getoffsetbefore (latest) not return latest offset?
Posted by Guozhang Wang <wa...@gmail.com>.
The offset request does not consider HW, and hence should not be affected
if it is not advanced yet.
As Jun said, the getOffsetsBefore API just send an OffsetRequest to Kafka
and will get an OffsetResponse in return, but this may not be exposed to
the API though.
Guozhang
On Tue, Dec 9, 2014 at 12:31 AM, 向河林 <xi...@mediav.com> wrote:
>
> Thanks Guozhang，
>
> I checked the server log, and I am 90% sure that no leader movement
> happened at that moment.
>
> Could it be that the data was in the memory but yet not flushed to the
> disk?
> I read the doc in the wiki
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication,
> and said
>
> The high watermark (HW) is the offset of the last committed message.
> Each log is periodically synced to disks. Data before the flushed
> offset is guaranteed to be persisted on disks. As we will see, the
> flush offset can be before or after HW.
>
> since the https://issues.apache.org/jira/browse/KAFKA-501 said
>
> For regular consumers, getOffset should return highWatermark as the
> latest offset.
>
> Thanks again.
>
>
> On Tue, Dec 9, 2014 at 5:28 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Helin,
> >
> > Is there a leader movement just before the get latest offset call? If
> your
> > follower is not synced and it then becomes the leader due to some reason,
> > it will not have the complete partition data.
> >
> > Guozhang
> >
> > On Mon, Dec 8, 2014 at 3:02 AM, Helin Xiang <xk...@gmail.com> wrote:
> >
> > > 1 additional information we found in the kafka’s application log since
> > the
> > > MAGIC time:
> > >
> > > 2014-12-04 09:59:36,726 [kafka-scheduler-2] INFO
> > > kafka.cluster.Partition - Partition [a.s.3,26] on broker 5: Shrinking
> > > ISR for partition [a.s.3,26] from 5,4 to 5
> > > 2014-12-04 09:59:36,728 [kafka-scheduler-2] ERROR kafka.utils.ZkUtils$
> > > - Conditional update of path
> > > /brokers/topics/a.s.3/partitions/26/state with data
> > >
> {"controller_epoch":2,"leader":5,"version":1,"leader_epoch":4,"isr":[5]}
> > > and expected version 675 failed due to
> > > org.apache.zookeeper.KeeperException$BadVersionException:
> > > KeeperErrorCode = BadVersion for
> > > /brokers/topics/a.s.3/partitions/26/state
> > >
> > >
> > >
> > > On Mon, Dec 8, 2014 at 6:59 PM, Helin Xiang <xk...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
> > > >
> > > > In one of our application, we want to get all partitions' latest
> > offsets,
> > > > so we use getoffsetbefore java API (latest).
> > > >
> > > > We believe at some time, 1 of the partition's latest offset we got is
> > > much
> > > > smaller than its real latest offset,(we saw in the application's log
> > that
> > > > the partition's offset is much smaller than other partitions'). Since
> > the
> > > > real data file of that partition was already deleted, we cannot prove
> > our
> > > > guess, but we found some clue in the kafka's application log which
> > helps
> > > us
> > > > to conclude that the partition's latest offset at that moment did
> have
> > a
> > > > much larger number.
> > > >
> > > > some additional useful information: the partition have 1 additional
> > > > replica(follower), and at that time, it was not synced with the
> leader
> > > > partition(far away behind the leader).
> > > >
> > > > Does any one have the same issue? In what condition could lead to
> this
> > > > situation?
> > > >
> > > > Thanks.
> > > >
> > > > --
> > > >
> > > >
> > > > *Best Regards向河林*
> > > >
> > >
> > >
> > >
> > > --
> > >
> > >
> > > *Best Regards向河林*
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
>
> *向河林*
>
>
> *MV AD **聚效广告* *上海* · 北京 · 广州 · 杭州
> _______________________________________________________________
>
> 上海市闸北区天目中路585号新梅大厦4楼 200070
>
> MOB：18621368491
>
> TEL：021-52559088（分机：8133）
>
> EMAIL：xianghl@mediav.com <ya...@mediav.com>
>
> HTTP：www.mediav.com
>
>
> -------------------------------CONFIDENTIAL ------------------------------
> ------------
>
> 本邮件载有秘密信息，请您恪守保密义务，勿向第三人透露。谢谢合作。
>
> This email communication is confidential. Recipient(s) named above is(are)
> obligated to maintain secrecy and is(are) not permitted to disclose the
> contents of this communication to others. Thank you.
>
--
-- Guozhang
Re: In what condition does getoffsetbefore (latest) not return latest offset?
Posted by 向河林 <xi...@mediav.com>.
Thanks Guozhang，
I checked the server log, and I am 90% sure that no leader movement
happened at that moment.
Could it be that the data was in the memory but yet not flushed to the disk?
I read the doc in the wiki
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication,
and said
The high watermark (HW) is the offset of the last committed message.
Each log is periodically synced to disks. Data before the flushed
offset is guaranteed to be persisted on disks. As we will see, the
flush offset can be before or after HW.
since the https://issues.apache.org/jira/browse/KAFKA-501 said
For regular consumers, getOffset should return highWatermark as the
latest offset.
Thanks again.
On Tue, Dec 9, 2014 at 5:28 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Helin,
>
> Is there a leader movement just before the get latest offset call? If your
> follower is not synced and it then becomes the leader due to some reason,
> it will not have the complete partition data.
>
> Guozhang
>
> On Mon, Dec 8, 2014 at 3:02 AM, Helin Xiang <xk...@gmail.com> wrote:
>
> > 1 additional information we found in the kafka’s application log since
> the
> > MAGIC time:
> >
> > 2014-12-04 09:59:36,726 [kafka-scheduler-2] INFO
> > kafka.cluster.Partition - Partition [a.s.3,26] on broker 5: Shrinking
> > ISR for partition [a.s.3,26] from 5,4 to 5
> > 2014-12-04 09:59:36,728 [kafka-scheduler-2] ERROR kafka.utils.ZkUtils$
> > - Conditional update of path
> > /brokers/topics/a.s.3/partitions/26/state with data
> > {"controller_epoch":2,"leader":5,"version":1,"leader_epoch":4,"isr":[5]}
> > and expected version 675 failed due to
> > org.apache.zookeeper.KeeperException$BadVersionException:
> > KeeperErrorCode = BadVersion for
> > /brokers/topics/a.s.3/partitions/26/state
> >
> >
> >
> > On Mon, Dec 8, 2014 at 6:59 PM, Helin Xiang <xk...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
> > >
> > > In one of our application, we want to get all partitions' latest
> offsets,
> > > so we use getoffsetbefore java API (latest).
> > >
> > > We believe at some time, 1 of the partition's latest offset we got is
> > much
> > > smaller than its real latest offset,(we saw in the application's log
> that
> > > the partition's offset is much smaller than other partitions'). Since
> the
> > > real data file of that partition was already deleted, we cannot prove
> our
> > > guess, but we found some clue in the kafka's application log which
> helps
> > us
> > > to conclude that the partition's latest offset at that moment did have
> a
> > > much larger number.
> > >
> > > some additional useful information: the partition have 1 additional
> > > replica(follower), and at that time, it was not synced with the leader
> > > partition(far away behind the leader).
> > >
> > > Does any one have the same issue? In what condition could lead to this
> > > situation?
> > >
> > > Thanks.
> > >
> > > --
> > >
> > >
> > > *Best Regards向河林*
> > >
> >
> >
> >
> > --
> >
> >
> > *Best Regards向河林*
> >
>
>
>
> --
> -- Guozhang
>
--
*向河林*
*MV AD **聚效广告* *上海* · 北京 · 广州 · 杭州
_______________________________________________________________
上海市闸北区天目中路585号新梅大厦4楼 200070
MOB：18621368491
TEL：021-52559088（分机：8133）
EMAIL：xianghl@mediav.com <ya...@mediav.com>
HTTP：www.mediav.com
-------------------------------CONFIDENTIAL ------------------------------
------------
本邮件载有秘密信息，请您恪守保密义务，勿向第三人透露。谢谢合作。
This email communication is confidential. Recipient(s) named above is(are)
obligated to maintain secrecy and is(are) not permitted to disclose the
contents of this communication to others. Thank you.
Re: In what condition does getoffsetbefore (latest) not return latest offset?
Posted by Guozhang Wang <wa...@gmail.com>.
Helin,
Is there a leader movement just before the get latest offset call? If your
follower is not synced and it then becomes the leader due to some reason,
it will not have the complete partition data.
Guozhang
On Mon, Dec 8, 2014 at 3:02 AM, Helin Xiang <xk...@gmail.com> wrote:
> 1 additional information we found in the kafka’s application log since the
> MAGIC time:
>
> 2014-12-04 09:59:36,726 [kafka-scheduler-2] INFO
> kafka.cluster.Partition - Partition [a.s.3,26] on broker 5: Shrinking
> ISR for partition [a.s.3,26] from 5,4 to 5
> 2014-12-04 09:59:36,728 [kafka-scheduler-2] ERROR kafka.utils.ZkUtils$
> - Conditional update of path
> /brokers/topics/a.s.3/partitions/26/state with data
> {"controller_epoch":2,"leader":5,"version":1,"leader_epoch":4,"isr":[5]}
> and expected version 675 failed due to
> org.apache.zookeeper.KeeperException$BadVersionException:
> KeeperErrorCode = BadVersion for
> /brokers/topics/a.s.3/partitions/26/state
>
>
>
> On Mon, Dec 8, 2014 at 6:59 PM, Helin Xiang <xk...@gmail.com> wrote:
>
> > Hi,
> >
> > We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
> >
> > In one of our application, we want to get all partitions' latest offsets,
> > so we use getoffsetbefore java API (latest).
> >
> > We believe at some time, 1 of the partition's latest offset we got is
> much
> > smaller than its real latest offset,(we saw in the application's log that
> > the partition's offset is much smaller than other partitions'). Since the
> > real data file of that partition was already deleted, we cannot prove our
> > guess, but we found some clue in the kafka's application log which helps
> us
> > to conclude that the partition's latest offset at that moment did have a
> > much larger number.
> >
> > some additional useful information: the partition have 1 additional
> > replica(follower), and at that time, it was not synced with the leader
> > partition(far away behind the leader).
> >
> > Does any one have the same issue? In what condition could lead to this
> > situation?
> >
> > Thanks.
> >
> > --
> >
> >
> > *Best Regards向河林*
> >
>
>
>
> --
>
>
> *Best Regards向河林*
>
--
-- Guozhang
Re: In what condition does getoffsetbefore (latest) not return latest offset?
Posted by Helin Xiang <xk...@gmail.com>.
1 additional information we found in the kafka’s application log since the
MAGIC time:
2014-12-04 09:59:36,726 [kafka-scheduler-2] INFO
kafka.cluster.Partition - Partition [a.s.3,26] on broker 5: Shrinking
ISR for partition [a.s.3,26] from 5,4 to 5
2014-12-04 09:59:36,728 [kafka-scheduler-2] ERROR kafka.utils.ZkUtils$
- Conditional update of path
/brokers/topics/a.s.3/partitions/26/state with data
{"controller_epoch":2,"leader":5,"version":1,"leader_epoch":4,"isr":[5]}
and expected version 675 failed due to
org.apache.zookeeper.KeeperException$BadVersionException:
KeeperErrorCode = BadVersion for
/brokers/topics/a.s.3/partitions/26/state
On Mon, Dec 8, 2014 at 6:59 PM, Helin Xiang <xk...@gmail.com> wrote:
> Hi,
>
> We have currently upgraded our kafka cluster from 0.7.2 to 0.8.1.1.
>
> In one of our application, we want to get all partitions' latest offsets,
> so we use getoffsetbefore java API (latest).
>
> We believe at some time, 1 of the partition's latest offset we got is much
> smaller than its real latest offset,(we saw in the application's log that
> the partition's offset is much smaller than other partitions'). Since the
> real data file of that partition was already deleted, we cannot prove our
> guess, but we found some clue in the kafka's application log which helps us
> to conclude that the partition's latest offset at that moment did have a
> much larger number.
>
> some additional useful information: the partition have 1 additional
> replica(follower), and at that time, it was not synced with the leader
> partition(far away behind the leader).
>
> Does any one have the same issue? In what condition could lead to this
> situation?
>
> Thanks.
>
> --
>
>
> *Best Regards向河林*
>
--
*Best Regards向河林*