You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "kojie.fu" <ko...@gmail.com> on 2013/10/21 05:54:55 UTC
how can i use offset filter the msg
Hi all,
we have a scene like this:
A system produce the msg ,B system comsumer the msg.
for some reason the B system unwork,and then we need abandon the out of date msg.
but now in our msg there is no time stamp.
how can i use the offset to do this.
does kafka know when the msg produce?
how can i location the certain offset?
thanks
Re: Re: how can i use offset filter the msg
Posted by "kojie.fu" <ko...@gmail.com>.
Hi Guozhang.
thanks to your reply ,we manage this problem.
i will file it to the jira.
and now we compute the relation from the metris info.
kojie.fu
From: Guozhang Wang
Date: 2013-10-21 23:11
To: users@kafka.apache.org
Subject: Re: Re: how can i use offset filter the msg
Hi Kojie,
As Jun's FAQ indicates, today the only way you can do is to set/reset the
offset directly with Zookeeper. In addition, currently we do not have
correlations between offsets and timestamps, meaning that given a timestamp
you can not tell which message's offset is produced at around that time.
Can you file a jira about keeping track of the correlation between offsets
and timestamps?
Guozhang
On Mon, Oct 21, 2013 at 7:56 AM, Jun Rao <ju...@gmail.com> wrote:
> Just added an FAQ. Does that answer your question?
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowcanIrewindtheoffsetintheconsumer%3F
>
> Thanks,
>
> Jun
>
>
> On Sun, Oct 20, 2013 at 11:15 PM, kojie.fu <ko...@gmail.com> wrote:
>
> > hi guozhang,
> > i mean ,in a certain topic ，can i use the offset compute the msg's
> > time stamp(when the msg produced)？
> > in fact， i need an api look like:
> > setOffset(String topicId ,String groupId, long timeStamp){
> >
> > }
> > this method can be call by the comsumer ，and then the offset will be
> > reset.
> > after call this method,the comsumer only comsume the msg occur after the
> > timeStamp.
> >
> > i am very sorry .I do not specify whether I describe clearly.
> > From: Guozhang Wang
> > Date: 2013-10-21 12:49
> > To: users@kafka.apache.org; kojie.fu
> > Subject: Re: how can i use offset filter the msg
> > Hello,
> >
> > I do not fully understanding your question, did you mean that you want to
> > know when a message with certain offset is produced?
> >
> >
> > Guozhang
> >
> >
> >
> >
> > On Sun, Oct 20, 2013 at 8:54 PM, kojie.fu <ko...@gmail.com> wrote:
> >
> > Hi all,
> > we have a scene like this:
> > A system produce the msg ,B system comsumer the msg.
> > for some reason the B system unwork,and then we need abandon the out of
> > date msg.
> > but now in our msg there is no time stamp.
> > how can i use the offset to do this.
> >
> > does kafka know when the msg produce?
> >
> > how can i location the certain offset?
> >
> >
> >
> > thanks
> >
> >
> >
> > --
> >
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Re: how can i use offset filter the msg
Posted by "kojie.fu" <ko...@gmail.com>.
yes，thanks.that is what we wanted !
we now use the metrics info [LogEndOffset] compute the offset some seconds ago in each partition.
and then use importZKOffsets ,It works !
thanks very much!
kojie.fu
From: Jun Rao
Date: 2013-10-21 22:56
To: users@kafka.apache.org; kojie.fu
Subject: Re: Re: how can i use offset filter the msg
Just added an FAQ. Does that answer your question?
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowcanIrewindtheoffsetintheconsumer%3F
Thanks,
Jun
On Sun, Oct 20, 2013 at 11:15 PM, kojie.fu <ko...@gmail.com> wrote:
hi guozhang,
i mean ,in a certain topic ，can i use the offset compute the msg's time stamp(when the msg produced)？
in fact， i need an api look like:
setOffset(String topicId ,String groupId, long timeStamp){
}
this method can be call by the comsumer ，and then the offset will be reset.
after call this method,the comsumer only comsume the msg occur after the timeStamp.
i am very sorry .I do not specify whether I describe clearly.
From: Guozhang Wang
Date: 2013-10-21 12:49
To: users@kafka.apache.org; kojie.fu
Subject: Re: how can i use offset filter the msg
Hello,
I do not fully understanding your question, did you mean that you want to know when a message with certain offset is produced?
Guozhang
On Sun, Oct 20, 2013 at 8:54 PM, kojie.fu <ko...@gmail.com> wrote:
Hi all,
we have a scene like this:
A system produce the msg ,B system comsumer the msg.
for some reason the B system unwork,and then we need abandon the out of date msg.
but now in our msg there is no time stamp.
how can i use the offset to do this.
does kafka know when the msg produce?
how can i location the certain offset?
thanks
--
-- Guozhang
Re: Re: how can i use offset filter the msg
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Kojie,
As Jun's FAQ indicates, today the only way you can do is to set/reset the
offset directly with Zookeeper. In addition, currently we do not have
correlations between offsets and timestamps, meaning that given a timestamp
you can not tell which message's offset is produced at around that time.
Can you file a jira about keeping track of the correlation between offsets
and timestamps?
Guozhang
On Mon, Oct 21, 2013 at 7:56 AM, Jun Rao <ju...@gmail.com> wrote:
> Just added an FAQ. Does that answer your question?
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowcanIrewindtheoffsetintheconsumer%3F
>
> Thanks,
>
> Jun
>
>
> On Sun, Oct 20, 2013 at 11:15 PM, kojie.fu <ko...@gmail.com> wrote:
>
> > hi guozhang,
> > i mean ,in a certain topic ，can i use the offset compute the msg's
> > time stamp(when the msg produced)？
> > in fact， i need an api look like:
> > setOffset(String topicId ,String groupId, long timeStamp){
> >
> > }
> > this method can be call by the comsumer ，and then the offset will be
> > reset.
> > after call this method,the comsumer only comsume the msg occur after the
> > timeStamp.
> >
> > i am very sorry .I do not specify whether I describe clearly.
> > From: Guozhang Wang
> > Date: 2013-10-21 12:49
> > To: users@kafka.apache.org; kojie.fu
> > Subject: Re: how can i use offset filter the msg
> > Hello,
> >
> > I do not fully understanding your question, did you mean that you want to
> > know when a message with certain offset is produced?
> >
> >
> > Guozhang
> >
> >
> >
> >
> > On Sun, Oct 20, 2013 at 8:54 PM, kojie.fu <ko...@gmail.com> wrote:
> >
> > Hi all,
> > we have a scene like this:
> > A system produce the msg ,B system comsumer the msg.
> > for some reason the B system unwork,and then we need abandon the out of
> > date msg.
> > but now in our msg there is no time stamp.
> > how can i use the offset to do this.
> >
> > does kafka know when the msg produce?
> >
> > how can i location the certain offset?
> >
> >
> >
> > thanks
> >
> >
> >
> > --
> >
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Re: how can i use offset filter the msg
Posted by Jun Rao <ju...@gmail.com>.
Just added an FAQ. Does that answer your question?
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowcanIrewindtheoffsetintheconsumer%3F
Thanks,
Jun
On Sun, Oct 20, 2013 at 11:15 PM, kojie.fu <ko...@gmail.com> wrote:
> hi guozhang,
> i mean ,in a certain topic ，can i use the offset compute the msg's
> time stamp(when the msg produced)？
> in fact， i need an api look like:
> setOffset(String topicId ,String groupId, long timeStamp){
>
> }
> this method can be call by the comsumer ，and then the offset will be
> reset.
> after call this method,the comsumer only comsume the msg occur after the
> timeStamp.
>
> i am very sorry .I do not specify whether I describe clearly.
> From: Guozhang Wang
> Date: 2013-10-21 12:49
> To: users@kafka.apache.org; kojie.fu
> Subject: Re: how can i use offset filter the msg
> Hello,
>
> I do not fully understanding your question, did you mean that you want to
> know when a message with certain offset is produced?
>
>
> Guozhang
>
>
>
>
> On Sun, Oct 20, 2013 at 8:54 PM, kojie.fu <ko...@gmail.com> wrote:
>
> Hi all,
> we have a scene like this:
> A system produce the msg ,B system comsumer the msg.
> for some reason the B system unwork,and then we need abandon the out of
> date msg.
> but now in our msg there is no time stamp.
> how can i use the offset to do this.
>
> does kafka know when the msg produce?
>
> how can i location the certain offset?
>
>
>
> thanks
>
>
>
> --
>
> -- Guozhang
>
Re: Re: how can i use offset filter the msg
Posted by "kojie.fu" <ko...@gmail.com>.
hi guozhang,
i mean ,in a certain topic ，can i use the offset compute the msg's time stamp(when the msg produced)？
in fact， i need an api look like:
setOffset(String topicId ,String groupId, long timeStamp){
}
this method can be call by the comsumer ，and then the offset will be reset.
after call this method,the comsumer only comsume the msg occur after the timeStamp.
i am very sorry .I do not specify whether I describe clearly.
From: Guozhang Wang
Date: 2013-10-21 12:49
To: users@kafka.apache.org; kojie.fu
Subject: Re: how can i use offset filter the msg
Hello,
I do not fully understanding your question, did you mean that you want to know when a message with certain offset is produced?
Guozhang
On Sun, Oct 20, 2013 at 8:54 PM, kojie.fu <ko...@gmail.com> wrote:
Hi all,
we have a scene like this:
A system produce the msg ,B system comsumer the msg.
for some reason the B system unwork,and then we need abandon the out of date msg.
but now in our msg there is no time stamp.
how can i use the offset to do this.
does kafka know when the msg produce?
how can i location the certain offset?
thanks
--
-- Guozhang
回复kojie.fu的问题
Posted by linghongbo008 <li...@gmail.com>.
这个意识是这样的，我们有一个如下的场景：
一个A系统往kafka队列写消息，一个B系统负责从kafka消费消息
现可能由于某种原因B系统无法读取kafka的消息，此时A系统还一直在往队列写消息，所以当我们把B系统
修复起来后（比如说花了2天时间修复了），我们不想让B系统还从原来的offset读取数据，而是想从某个时间点开始读取数据，
不知道kafka有类似的 时间戳和offset对应的接口没？
From: Guozhang Wang
Date: 2013-10-21 12:49
To: users@kafka.apache.org; kojie.fu
Subject: Re: how can i use offset filter the msg
Hello,
I do not fully understanding your question, did you mean that you want to
know when a message with certain offset is produced?
Guozhang
On Sun, Oct 20, 2013 at 8:54 PM, kojie.fu <ko...@gmail.com> wrote:
> Hi all,
> we have a scene like this:
> A system produce the msg ,B system comsumer the msg.
> for some reason the B system unwork,and then we need abandon the out of
> date msg.
> but now in our msg there is no time stamp.
> how can i use the offset to do this.
>
> does kafka know when the msg produce?
>
> how can i location the certain offset?
>
>
>
> thanks
--
-- Guozhang
Re: how can i use offset filter the msg
Posted by Guozhang Wang <wa...@gmail.com>.
Hello,
I do not fully understanding your question, did you mean that you want to
know when a message with certain offset is produced?
Guozhang
On Sun, Oct 20, 2013 at 8:54 PM, kojie.fu <ko...@gmail.com> wrote:
> Hi all,
> we have a scene like this:
> A system produce the msg ,B system comsumer the msg.
> for some reason the B system unwork,and then we need abandon the out of
> date msg.
> but now in our msg there is no time stamp.
> how can i use the offset to do this.
>
> does kafka know when the msg produce?
>
> how can i location the certain offset?
>
>
>
> thanks
--
-- Guozhang