You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sachin Mittal <sj...@gmail.com> on 2016/12/13 07:30:36 UTC
What does GetOffsetShell result represent
Hi,
I have some trouble interpreting the result of GetOffsetShell command.
Say if I run
bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell --broker-list
localhost:9092 --topic test-window-stream --time -2
test-window-stream:0:0
D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
test-window-stream --time -1
test-window-stream:0:724
bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell --broker-list
localhost:9092 --topic test-window-stream
test-window-stream:0:724
D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
test-window-stream --offset 2
test-window-stream:0:724,0
So what does 0:0 mean or for that mater 0:724
Also when I use -2 (earliest), why does it show 0:0, does this means
offset if at the start of the topic.
Also when I use -1 (latest), it shows 0:724, does it means none of the
records are processed.
However I was having a stream application running which did aggregate
atleast upto 720 records before I killed it. So why does it not show here?
Also it is not clear what --offset flag means and when I pass something
greater than 1 I get ,0 appended to the earlier result.
Thanks
Sachin
Re: What does GetOffsetShell result represent
Posted by Sachin Mittal <sj...@gmail.com>.
Hi,
Thanks for the lucid explanation.
So say if I run this
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
192.168.73.198:9092 --topic test-stream --time -1 --offsets 6
I get output like:
test-stream:0:81658712,81616045,81299569,80987584,80673168,80364832
This means these are the offset of latest message on these segment files.
So the current segment is holding (81658712 - 81616045) messages = 42667
message
Previous one had (81616045 - 81299569) = 316476 messages
Previous than that (81299569 - 80987584) = 311985 messages
and so on...
and when I run
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
192.168.73.198:9092 --topic test-stream --time -2
I get output like
test-advice-stream:0:69258918
This means that earliest message offset in the segment file is 69258918 and
messages before that have been deleted.
I guess my understating is correct.
Thanks
Sachin
On Sun, Dec 18, 2016 at 6:25 AM, Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> The tool writes output in the format:
>
> <topic>:<partition>:<comma separated list of valid offsets>
>
> So in the case of your example with --time -1 that returned
> test-window-stream:0:724, it is saying that test-window-stream has
> partition 0 with a valid log segment which has the first offset = 724. Note
> that --time -1 is a special code for "only give the latest segment". For
> the -2 option you saw 0:0 because the earliest segment is still currently
> the first segment created for that topic, i.e. the one starting with offset
> 0 (because your retention policy hasn't kicked in yet). In cases where it
> returned multiple comma separated values, those are multiple starting
> offsets for segments containing offsets before the timestamp you requested.
>
> Note that this is using the old simple consumer approach to querying for
> offsets. That implementation could only tell you about offsets based on
> entire log segments. If you're on a newer version of Kafka, you may be
> interested in the time-based offset query, which you can access in the new
> consumer API and gives you better granularity (
> http://kafka.apache.org/0101/
> javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#
> offsetsForTimes(java.util.Map)).
>
> -Ewen
>
> On Thu, Dec 15, 2016 at 3:52 AM, Sachin Mittal <sj...@gmail.com> wrote:
>
> > Folks any explanation for this. Or any link that can help me on that.
> >
> > On Tue, Dec 13, 2016 at 1:00 PM, Sachin Mittal <sj...@gmail.com>
> wrote:
> >
> > > Hi,
> > > I have some trouble interpreting the result of GetOffsetShell command.
> > >
> > > Say if I run
> > > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell
> --broker-list
> > > localhost:9092 --topic test-window-stream --time -2
> > > test-window-stream:0:0
> > >
> > > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> > > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> > > test-window-stream --time -1
> > > test-window-stream:0:724
> > >
> > > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell
> --broker-list
> > > localhost:9092 --topic test-window-stream
> > > test-window-stream:0:724
> > >
> > > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> > > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> > > test-window-stream --offset 2
> > > test-window-stream:0:724,0
> > >
> > > So what does 0:0 mean or for that mater 0:724
> > >
> > > Also when I use -2 (earliest), why does it show 0:0, does this means
> > > offset if at the start of the topic.
> > > Also when I use -1 (latest), it shows 0:724, does it means none of the
> > > records are processed.
> > > However I was having a stream application running which did aggregate
> > > atleast upto 720 records before I killed it. So why does it not show
> > here?
> > >
> > > Also it is not clear what --offset flag means and when I pass something
> > > greater than 1 I get ,0 appended to the earlier result.
> > >
> > > Thanks
> > > Sachin
> > >
> > >
> >
>
Re: What does GetOffsetShell result represent
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
The tool writes output in the format:
<topic>:<partition>:<comma separated list of valid offsets>
So in the case of your example with --time -1 that returned
test-window-stream:0:724, it is saying that test-window-stream has
partition 0 with a valid log segment which has the first offset = 724. Note
that --time -1 is a special code for "only give the latest segment". For
the -2 option you saw 0:0 because the earliest segment is still currently
the first segment created for that topic, i.e. the one starting with offset
0 (because your retention policy hasn't kicked in yet). In cases where it
returned multiple comma separated values, those are multiple starting
offsets for segments containing offsets before the timestamp you requested.
Note that this is using the old simple consumer approach to querying for
offsets. That implementation could only tell you about offsets based on
entire log segments. If you're on a newer version of Kafka, you may be
interested in the time-based offset query, which you can access in the new
consumer API and gives you better granularity (http://kafka.apache.org/0101/
javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#
offsetsForTimes(java.util.Map)).
-Ewen
On Thu, Dec 15, 2016 at 3:52 AM, Sachin Mittal <sj...@gmail.com> wrote:
> Folks any explanation for this. Or any link that can help me on that.
>
> On Tue, Dec 13, 2016 at 1:00 PM, Sachin Mittal <sj...@gmail.com> wrote:
>
> > Hi,
> > I have some trouble interpreting the result of GetOffsetShell command.
> >
> > Say if I run
> > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell --broker-list
> > localhost:9092 --topic test-window-stream --time -2
> > test-window-stream:0:0
> >
> > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> > test-window-stream --time -1
> > test-window-stream:0:724
> >
> > bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell --broker-list
> > localhost:9092 --topic test-window-stream
> > test-window-stream:0:724
> >
> > D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> > kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> > test-window-stream --offset 2
> > test-window-stream:0:724,0
> >
> > So what does 0:0 mean or for that mater 0:724
> >
> > Also when I use -2 (earliest), why does it show 0:0, does this means
> > offset if at the start of the topic.
> > Also when I use -1 (latest), it shows 0:724, does it means none of the
> > records are processed.
> > However I was having a stream application running which did aggregate
> > atleast upto 720 records before I killed it. So why does it not show
> here?
> >
> > Also it is not clear what --offset flag means and when I pass something
> > greater than 1 I get ,0 appended to the earlier result.
> >
> > Thanks
> > Sachin
> >
> >
>
Re: What does GetOffsetShell result represent
Posted by Sachin Mittal <sj...@gmail.com>.
Folks any explanation for this. Or any link that can help me on that.
On Tue, Dec 13, 2016 at 1:00 PM, Sachin Mittal <sj...@gmail.com> wrote:
> Hi,
> I have some trouble interpreting the result of GetOffsetShell command.
>
> Say if I run
> bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell --broker-list
> localhost:9092 --topic test-window-stream --time -2
> test-window-stream:0:0
>
> D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> test-window-stream --time -1
> test-window-stream:0:724
>
> bin\windows\kafka-run-class.bat kafka.tools.GetOffsetShell --broker-list
> localhost:9092 --topic test-window-stream
> test-window-stream:0:724
>
> D:\kafka_2.10-0.10.2.0-SNAPSHOT>bin\windows\kafka-run-class.bat
> kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic
> test-window-stream --offset 2
> test-window-stream:0:724,0
>
> So what does 0:0 mean or for that mater 0:724
>
> Also when I use -2 (earliest), why does it show 0:0, does this means
> offset if at the start of the topic.
> Also when I use -1 (latest), it shows 0:724, does it means none of the
> records are processed.
> However I was having a stream application running which did aggregate
> atleast upto 720 records before I killed it. So why does it not show here?
>
> Also it is not clear what --offset flag means and when I pass something
> greater than 1 I get ,0 appended to the earlier result.
>
> Thanks
> Sachin
>
>