You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Calvin Lei <ck...@gmail.com> on 2014/01/07 17:31:41 UTC
PartitionOffsetRequestInfo only supports -1 and -2?
Does it support an long timestamp? Tried it and the resulting offset
response contains an empty long array.
--
Sent from Gmail Mobile
Re: PartitionOffsetRequestInfo only supports -1 and -2?
Posted by Joe Stein <jo...@stealth.ly>.
my bad, I was look at the fetch api instead of the offset api =8^/ ignore
my response about time being the wrong variable, its right for the offset
API
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI:)
thanks Jun!
On Tue, Jan 7, 2014 at 12:10 PM, Jun Rao <ju...@gmail.com> wrote:
> You can provide an arbitrary timestamp. However, the answer is not going to
> be exact. The broker only returns the offset of the first message in a log
> segment file whose last modified time is less than the provided timestamp.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jan 7, 2014 at 8:31 AM, Calvin Lei <ck...@gmail.com> wrote:
>
> > Does it support an long timestamp? Tried it and the resulting offset
> > response contains an empty long array.
> >
> >
> >
> >
> >
> > --
> > Sent from Gmail Mobile
> >
>
Re: PartitionOffsetRequestInfo only supports -1 and -2?
Posted by Calvin Lei <ck...@gmail.com>.
Jun,
Arbitrary timestamp does not work for me. It would return an empty long
array.
Cal
On Tuesday, January 7, 2014, Jun Rao wrote:
> You can provide an arbitrary timestamp. However, the answer is not going to
> be exact. The broker only returns the offset of the first message in a log
> segment file whose last modified time is less than the provided timestamp.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jan 7, 2014 at 8:31 AM, Calvin Lei <ckplei@gmail.com<javascript:;>>
> wrote:
>
> > Does it support an long timestamp? Tried it and the resulting offset
> > response contains an empty long array.
> >
> >
> >
> >
> >
> > --
> > Sent from Gmail Mobile
> >
>
--
Sent from Gmail Mobile
Re: PartitionOffsetRequestInfo only supports -1 and -2?
Posted by Jun Rao <ju...@gmail.com>.
You can provide an arbitrary timestamp. However, the answer is not going to
be exact. The broker only returns the offset of the first message in a log
segment file whose last modified time is less than the provided timestamp.
Thanks,
Jun
On Tue, Jan 7, 2014 at 8:31 AM, Calvin Lei <ck...@gmail.com> wrote:
> Does it support an long timestamp? Tried it and the resulting offset
> response contains an empty long array.
>
>
>
>
>
> --
> Sent from Gmail Mobile
>
Re: PartitionOffsetRequestInfo only supports -1 and -2?
Posted by Jun Rao <ju...@gmail.com>.
Hmm, it should be the same since the following is how
javaapi.OffsetResponse.offsets is implemented.
def offsets(topic: String, partition: Int) =
underlying.partitionErrorAndOffsets(TopicAndPartition(topic,
partition)).offsets.toArray
Thanks,
Jun
On Tue, Jan 7, 2014 at 5:44 PM, Calvin Lei <ck...@gmail.com> wrote:
> Jun,
> Regarding the arbitrary timestamp, I found that if I use the
> GetOffsetShell<
> https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/GetOffsetShell.scala
> >,
> it works. But if I use the Java API ( requestInfo.put(topicAndPartition,
> new
> PartitionOffsetRequestInfo(SOME_LONG_TIMESTAMP, 1));), it only returns an
> empty long[].
>
> I looked at the source of GetOffsetShell, the way the offset is
> retrieved is slightly different from using the java api. Could that be a
> bug?
> Scala API:
> val offsets = consumer.getOffsetsBefore(request).partitionErrorAndOffsets(
> topicAndPartition).offsets
>
> Java API:
> consumer.getOffsetsBefore(request).offsets(topic, partition);
>
>
> thanks,
> Cal
>
>
>
> On Tue, Jan 7, 2014 at 6:49 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > Joe,
> >
> > Yes, this is a bit confusing. We have 2 offset related requests. One is
> > OffsetFetchRequest and another is OffsetRequest. The former is used to
> get
> > the offset within a consumer group and the latter is to get the offset
> > before a particular timestamp. And there is of course the FetchRequest,
> > which fetches messages on a given offset.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 7, 2014 at 9:00 AM, Joe Stein <jo...@stealth.ly> wrote:
> >
> > > hmmm, that should be the "offset to fetch from", not sure why the
> > variable
> > > is called "time" =8^/ unless I am looking at something else from what
> > you
> > > are asking but this I think (another dev please confirm or explain why
> it
> > > is time).
> > >
> > > case class PartitionOffsetRequestInfo(time: Long, maxNumOffsets: Int)
> > >
> > > should be
> > >
> > > case class PartitionOffsetRequestInfo(offsetToFetchFrom: Long,
> > > maxNumOffsets: Int)
> > >
> > > which matches the Fetch API we have documented
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> > >
> > >
> > > /*******************************************
> > > Joe Stein
> > > Founder, Principal Consultant
> > > Big Data Open Source Security LLC
> > > http://www.stealth.ly
> > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > ********************************************/
> > >
> > >
> > > On Tue, Jan 7, 2014 at 11:31 AM, Calvin Lei <ck...@gmail.com> wrote:
> > >
> > > > Does it support an long timestamp? Tried it and the resulting offset
> > > > response contains an empty long array.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from Gmail Mobile
> > > >
> > >
> >
>
Re: PartitionOffsetRequestInfo only supports -1 and -2?
Posted by Calvin Lei <ck...@gmail.com>.
Jun,
Regarding the arbitrary timestamp, I found that if I use the
GetOffsetShell<https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/GetOffsetShell.scala>,
it works. But if I use the Java API ( requestInfo.put(topicAndPartition, new
PartitionOffsetRequestInfo(SOME_LONG_TIMESTAMP, 1));), it only returns an
empty long[].
I looked at the source of GetOffsetShell, the way the offset is
retrieved is slightly different from using the java api. Could that be a
bug?
Scala API:
val offsets = consumer.getOffsetsBefore(request).partitionErrorAndOffsets(
topicAndPartition).offsets
Java API:
consumer.getOffsetsBefore(request).offsets(topic, partition);
thanks,
Cal
On Tue, Jan 7, 2014 at 6:49 PM, Jun Rao <ju...@gmail.com> wrote:
> Joe,
>
> Yes, this is a bit confusing. We have 2 offset related requests. One is
> OffsetFetchRequest and another is OffsetRequest. The former is used to get
> the offset within a consumer group and the latter is to get the offset
> before a particular timestamp. And there is of course the FetchRequest,
> which fetches messages on a given offset.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jan 7, 2014 at 9:00 AM, Joe Stein <jo...@stealth.ly> wrote:
>
> > hmmm, that should be the "offset to fetch from", not sure why the
> variable
> > is called "time" =8^/ unless I am looking at something else from what
> you
> > are asking but this I think (another dev please confirm or explain why it
> > is time).
> >
> > case class PartitionOffsetRequestInfo(time: Long, maxNumOffsets: Int)
> >
> > should be
> >
> > case class PartitionOffsetRequestInfo(offsetToFetchFrom: Long,
> > maxNumOffsets: Int)
> >
> > which matches the Fetch API we have documented
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> >
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
> >
> > On Tue, Jan 7, 2014 at 11:31 AM, Calvin Lei <ck...@gmail.com> wrote:
> >
> > > Does it support an long timestamp? Tried it and the resulting offset
> > > response contains an empty long array.
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sent from Gmail Mobile
> > >
> >
>
Re: PartitionOffsetRequestInfo only supports -1 and -2?
Posted by Jun Rao <ju...@gmail.com>.
Joe,
Yes, this is a bit confusing. We have 2 offset related requests. One is
OffsetFetchRequest and another is OffsetRequest. The former is used to get
the offset within a consumer group and the latter is to get the offset
before a particular timestamp. And there is of course the FetchRequest,
which fetches messages on a given offset.
Thanks,
Jun
On Tue, Jan 7, 2014 at 9:00 AM, Joe Stein <jo...@stealth.ly> wrote:
> hmmm, that should be the "offset to fetch from", not sure why the variable
> is called "time" =8^/ unless I am looking at something else from what you
> are asking but this I think (another dev please confirm or explain why it
> is time).
>
> case class PartitionOffsetRequestInfo(time: Long, maxNumOffsets: Int)
>
> should be
>
> case class PartitionOffsetRequestInfo(offsetToFetchFrom: Long,
> maxNumOffsets: Int)
>
> which matches the Fetch API we have documented
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
>
> On Tue, Jan 7, 2014 at 11:31 AM, Calvin Lei <ck...@gmail.com> wrote:
>
> > Does it support an long timestamp? Tried it and the resulting offset
> > response contains an empty long array.
> >
> >
> >
> >
> >
> > --
> > Sent from Gmail Mobile
> >
>
Re: PartitionOffsetRequestInfo only supports -1 and -2?
Posted by Joe Stein <jo...@stealth.ly>.
hmmm, that should be the "offset to fetch from", not sure why the variable
is called "time" =8^/ unless I am looking at something else from what you
are asking but this I think (another dev please confirm or explain why it
is time).
case class PartitionOffsetRequestInfo(time: Long, maxNumOffsets: Int)
should be
case class PartitionOffsetRequestInfo(offsetToFetchFrom: Long,
maxNumOffsets: Int)
which matches the Fetch API we have documented
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Tue, Jan 7, 2014 at 11:31 AM, Calvin Lei <ck...@gmail.com> wrote:
> Does it support an long timestamp? Tried it and the resulting offset
> response contains an empty long array.
>
>
>
>
>
> --
> Sent from Gmail Mobile
>