You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Alexander Gallego <ga...@gmail.com> on 2012/09/26 03:11:25 UTC
Kafka SimpleConsumer.getOffsetsBefore Problem. Kafka 0.7
Hello,
I'm having problems with Kafka 0.7:
Here is a project I've put together to show the problem:
https://bitbucket.org/agallego/kafkaoffset (make sure to run twice:
once to generate data, and one to see the problem. )
I can't seem to get valid offsets from the queue using a
SimpleConsumer class.
(https://bitbucket.org/agallego/kafkaoffset/src/a9043505919dd8743e368e27467dd96ac0c9c0aa/src/main/java/test/kafka/SampleConsumer.java#cl-13)
I can read and write to the queue no problem given an offset.
Everything works just fine. However, I want to read from the queue the
last offset from 'now()' . That is to say, I want to pass in an
arbitrary time in the past and I want to get a list of correct
offsets. (off course given that there is data before that time)
** What am I seeing instead ? ***
I always get an array of 0's
quite literally long[] tmp = {0L, 0L, 0L....}
** What do I expect to see ***
An array of valid offsets as stated by the api.
Is this a bug ? What am I doing wrong. The whole code is about 20
lines of code. It assumes a running instance of kafka on localhost.
(off course you can change it)
Thanks in advance
Sincerely,
Alex
---*---
------*
* * *
Re: Kafka SimpleConsumer.getOffsetsBefore Problem. Kafka 0.7
Posted by Alexander Gallego <ga...@gmail.com>.
Zsolt: Thanks
Sincerely,
Alexander Gallego
---*---
------*
* * *
On Wed, Sep 26, 2012 at 1:44 PM, Zsolt Dollenstein <zs...@gmail.com> wrote:
> getOffsetsBefore works based on the mtime of the log segments
Re: Kafka SimpleConsumer.getOffsetsBefore Problem. Kafka 0.7
Posted by Zsolt Dollenstein <zs...@gmail.com>.
The answer you're looking for has already been said on this list, iirc.
getOffsetsBefore works based on the mtime of the log segments, which is why
the offsets you get are about 500 megs apart
On Sep 26, 2012 7:22 PM, "Alexander Gallego" <ga...@gmail.com>
wrote:
> This is a valuable follow up:
>
> After flushing about 5.5 G of data onto a test kafka node (localhost)
> SimpleConsumer.getOffsetsBefore() semi works. At least it doesn't
> return 0's anymore.
>
>
> I think the implementation is fine, what I think is missing is api
> documentation, perhaps I'll push a doc change. Here is what happens.
>
>
> 1. After you run the consumer for long enough you start getting some
> real offsets, the question becomes ** when ** and how (predictably)
> 2. Answer is easy.
>
> Here are the offsets for my test.
>
> 5368709220
> 4831838298
> 4294967376
> 3758096454
> 3221225532
> 2684354610
> 2147483688
> 1610612766
> 1073741844
> 536870922
> 0
>
>
> Here are the kafka logs.
>
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:51
> 00000000000000000000.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:53
> 00000000000536870922.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:53
> 00000000001073741844.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:54
> 00000000001610612766.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:55
> 00000000002147483688.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:55
> 00000000002684354610.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:56
> 00000000003221225532.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:56
> 00000000003758096454.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:57
> 00000000004294967376.kafka │
> -rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:58
> 00000000004831838298.kafka │
> -rw-r--r-- 1 agallego wheel 534062760 Sep 26 11:59
> 00000000005368709220.kafka
>
>
> To point it out.
>
> *** The offsets are the name of the files. ****
>
>
> In other words, the offsets don't get updated until a new file is
> generated.
>
>
>
> I'm not sure how to submit a push request for documentation for this
> finding. Perhaps a dev can comment if I missed the docs, certainly the
> api docs don't help much (they are not totally accurate)
>
>
> Thanks,
>
>
>
>
>
> Sincerely,
> Alexander Gallego
>
> ---*---
> ------*
> * * *
>
>
>
>
> On Wed, Sep 26, 2012 at 10:22 AM, Alexander Gallego
> <ga...@gmail.com> wrote:
> > Hello,
> >
> > I don't mean to spam the mailing list. I emailed yesterday, but I had
> > made the repo containing the examples 'private' by accident. It is now
> > open.
> >
> > Thanks again for reading.
> >
> >
> > Sincerely,
> > Alex
> >
> > ---*---
> > ------*
> > * * *
> >
> >
> >
> >
> > On Tue, Sep 25, 2012 at 9:11 PM, Alexander Gallego
> > <ga...@gmail.com> wrote:
> >> Hello,
> >>
> >> I'm having problems with Kafka 0.7:
> >>
> >> Here is a project I've put together to show the problem:
> >> https://bitbucket.org/agallego/kafkaoffset (make sure to run twice:
> >> once to generate data, and one to see the problem. )
> >>
> >> I can't seem to get valid offsets from the queue using a
> >> SimpleConsumer class.
> >> (
> https://bitbucket.org/agallego/kafkaoffset/src/a9043505919dd8743e368e27467dd96ac0c9c0aa/src/main/java/test/kafka/SampleConsumer.java#cl-13
> )
> >>
> >> I can read and write to the queue no problem given an offset.
> >> Everything works just fine. However, I want to read from the queue the
> >> last offset from 'now()' . That is to say, I want to pass in an
> >> arbitrary time in the past and I want to get a list of correct
> >> offsets. (off course given that there is data before that time)
> >>
> >>
> >> ** What am I seeing instead ? ***
> >>
> >> I always get an array of 0's
> >>
> >> quite literally long[] tmp = {0L, 0L, 0L....}
> >>
> >>
> >>
> >> ** What do I expect to see ***
> >>
> >> An array of valid offsets as stated by the api.
> >>
> >>
> >>
> >>
> >> Is this a bug ? What am I doing wrong. The whole code is about 20
> >> lines of code. It assumes a running instance of kafka on localhost.
> >> (off course you can change it)
> >>
> >>
> >> Thanks in advance
> >>
> >>
> >> Sincerely,
> >> Alex
> >>
> >> ---*---
> >> ------*
> >> * * *
>
Re: Kafka SimpleConsumer.getOffsetsBefore Problem. Kafka 0.7
Posted by Alexander Gallego <ga...@gmail.com>.
This is a valuable follow up:
After flushing about 5.5 G of data onto a test kafka node (localhost)
SimpleConsumer.getOffsetsBefore() semi works. At least it doesn't
return 0's anymore.
I think the implementation is fine, what I think is missing is api
documentation, perhaps I'll push a doc change. Here is what happens.
1. After you run the consumer for long enough you start getting some
real offsets, the question becomes ** when ** and how (predictably)
2. Answer is easy.
Here are the offsets for my test.
5368709220
4831838298
4294967376
3758096454
3221225532
2684354610
2147483688
1610612766
1073741844
536870922
0
Here are the kafka logs.
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:51
00000000000000000000.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:53
00000000000536870922.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:53
00000000001073741844.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:54
00000000001610612766.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:55
00000000002147483688.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:55
00000000002684354610.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:56
00000000003221225532.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:56
00000000003758096454.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:57
00000000004294967376.kafka │
-rw-r--r-- 1 agallego wheel 536870922 Sep 26 11:58
00000000004831838298.kafka │
-rw-r--r-- 1 agallego wheel 534062760 Sep 26 11:59
00000000005368709220.kafka
To point it out.
*** The offsets are the name of the files. ****
In other words, the offsets don't get updated until a new file is generated.
I'm not sure how to submit a push request for documentation for this
finding. Perhaps a dev can comment if I missed the docs, certainly the
api docs don't help much (they are not totally accurate)
Thanks,
Sincerely,
Alexander Gallego
---*---
------*
* * *
On Wed, Sep 26, 2012 at 10:22 AM, Alexander Gallego
<ga...@gmail.com> wrote:
> Hello,
>
> I don't mean to spam the mailing list. I emailed yesterday, but I had
> made the repo containing the examples 'private' by accident. It is now
> open.
>
> Thanks again for reading.
>
>
> Sincerely,
> Alex
>
> ---*---
> ------*
> * * *
>
>
>
>
> On Tue, Sep 25, 2012 at 9:11 PM, Alexander Gallego
> <ga...@gmail.com> wrote:
>> Hello,
>>
>> I'm having problems with Kafka 0.7:
>>
>> Here is a project I've put together to show the problem:
>> https://bitbucket.org/agallego/kafkaoffset (make sure to run twice:
>> once to generate data, and one to see the problem. )
>>
>> I can't seem to get valid offsets from the queue using a
>> SimpleConsumer class.
>> (https://bitbucket.org/agallego/kafkaoffset/src/a9043505919dd8743e368e27467dd96ac0c9c0aa/src/main/java/test/kafka/SampleConsumer.java#cl-13)
>>
>> I can read and write to the queue no problem given an offset.
>> Everything works just fine. However, I want to read from the queue the
>> last offset from 'now()' . That is to say, I want to pass in an
>> arbitrary time in the past and I want to get a list of correct
>> offsets. (off course given that there is data before that time)
>>
>>
>> ** What am I seeing instead ? ***
>>
>> I always get an array of 0's
>>
>> quite literally long[] tmp = {0L, 0L, 0L....}
>>
>>
>>
>> ** What do I expect to see ***
>>
>> An array of valid offsets as stated by the api.
>>
>>
>>
>>
>> Is this a bug ? What am I doing wrong. The whole code is about 20
>> lines of code. It assumes a running instance of kafka on localhost.
>> (off course you can change it)
>>
>>
>> Thanks in advance
>>
>>
>> Sincerely,
>> Alex
>>
>> ---*---
>> ------*
>> * * *
Re: Kafka SimpleConsumer.getOffsetsBefore Problem. Kafka 0.7
Posted by Alexander Gallego <ga...@gmail.com>.
Hello,
I don't mean to spam the mailing list. I emailed yesterday, but I had
made the repo containing the examples 'private' by accident. It is now
open.
Thanks again for reading.
Sincerely,
Alex
---*---
------*
* * *
On Tue, Sep 25, 2012 at 9:11 PM, Alexander Gallego
<ga...@gmail.com> wrote:
> Hello,
>
> I'm having problems with Kafka 0.7:
>
> Here is a project I've put together to show the problem:
> https://bitbucket.org/agallego/kafkaoffset (make sure to run twice:
> once to generate data, and one to see the problem. )
>
> I can't seem to get valid offsets from the queue using a
> SimpleConsumer class.
> (https://bitbucket.org/agallego/kafkaoffset/src/a9043505919dd8743e368e27467dd96ac0c9c0aa/src/main/java/test/kafka/SampleConsumer.java#cl-13)
>
> I can read and write to the queue no problem given an offset.
> Everything works just fine. However, I want to read from the queue the
> last offset from 'now()' . That is to say, I want to pass in an
> arbitrary time in the past and I want to get a list of correct
> offsets. (off course given that there is data before that time)
>
>
> ** What am I seeing instead ? ***
>
> I always get an array of 0's
>
> quite literally long[] tmp = {0L, 0L, 0L....}
>
>
>
> ** What do I expect to see ***
>
> An array of valid offsets as stated by the api.
>
>
>
>
> Is this a bug ? What am I doing wrong. The whole code is about 20
> lines of code. It assumes a running instance of kafka on localhost.
> (off course you can change it)
>
>
> Thanks in advance
>
>
> Sincerely,
> Alex
>
> ---*---
> ------*
> * * *