You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Tanguy tlrx <tl...@gmail.com> on 2014/07/17 14:22:05 UTC
Kafka 0.8.1.1 / Can't read latest message
Hi all,
I have a strange behavior with Kafka 0.8.1.1 where consumers do not
retrieve the latest messages.
I created a topic called "my-topic" with 8 partitions. My application
creates a SimpleConsumer instance for each topic-partition pair and
executes an OffsetRequest to retrieve the earliest and latest offsets for
each partition.
As an example, the partition 5 of the topic "my-topic" has an earliest
offset at 4 and a latest offset at 9.
Then, my application uses the same SimpleConsumer instance and executes a
FetchRequest to read messages (starting from earliest) for that partition.
Still with the example with partition 5, my application can read messages
with offset 4,5,6,7,8 and then no more messages can be read. The same
behavior exists for other partitions.
I'm not sure that the message with offset "9" really exists, but the Kafka
API returns a latest offset of 9 that let me thinking that this message
exists. I tested with a HighLevelConsumer and it works the same way,
reading messages 4...8 but not the last one.
I did not notice this behavior with the previous version of Kafka I used
(0.8.0-beta1).
Am i missing something? Anyone with the same problem?
Thanks
-- Tanguy
twitter @tlrx
https://github.com/tlrx
Re: Kafka 0.8.1.1 / Can't read latest message
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Tanguy,
Just updated the wording on this protocol wiki a bit:
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI
Hope it is now clearer.
Guozhang
On Thu, Jul 17, 2014 at 11:48 PM, Tanguy tlrx <tl...@gmail.com> wrote:
> Hi Jay, Hi Guozhang,
>
> Thanks for your response. I was suspecting something like that but did not
> found the documentation entry that explain that.
>
> Thanks again!
>
>
> 2014-07-17 23:41 GMT+02:00 Jay Kreps <ja...@gmail.com>:
>
> > This makes sense if you think about it. If you want to start "now" you
> > don't want the last message in the log, which could be seconds,
> > minutes, days, or weeks old, you actually want the next message that
> > comes in.
> >
> > -Jay
> >
> > On Thu, Jul 17, 2014 at 10:00 AM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > > Hello Tanguy,
> > >
> > > With "latest" the returned offset is actually the "logEndOffset", which
> > is
> > > the offset of the next message that will be appended to the log.
> > >
> > > Guozhang
> > >
> > >
> > > On Thu, Jul 17, 2014 at 5:26 AM, Tanguy tlrx <tl...@gmail.com>
> wrote:
> > >
> > >> Hi all,
> > >>
> > >> I have a strange behavior with Kafka 0.8.1.1 where consumers do not
> > >> retrieve the latest messages.
> > >>
> > >> I created a topic called "my-topic" with 8 partitions. My application
> > >> creates a SimpleConsumer instance for each topic-partition pair and
> > >> executes an OffsetRequest to retrieve the earliest and latest offsets
> > for
> > >> each partition.
> > >>
> > >> As an example, the partition 5 of the topic "my-topic" has an earliest
> > >> offset at 4 and a latest offset at 9.
> > >>
> > >> Then, my application uses the same SimpleConsumer instance and
> executes
> > a
> > >> FetchRequest to read messages (starting from earliest) for that
> > partition.
> > >>
> > >> Still with the example with partition 5, my application can read
> > messages
> > >> with offset 4,5,6,7,8 and then no more messages can be read. The same
> > >> behavior exists for other partitions.
> > >>
> > >> I'm not sure that the message with offset "9" really exists, but the
> > Kafka
> > >> API returns a latest offset of 9 that let me thinking that this
> message
> > >> exists. I tested with a HighLevelConsumer and it works the same way,
> > >> reading messages 4...8 but not the last one.
> > >>
> > >> I did not notice this behavior with the previous version of Kafka I
> used
> > >> (0.8.0-beta1).
> > >>
> > >> Am i missing something? Anyone with the same problem?
> > >>
> > >> Thanks
> > >>
> > >> -- Tanguy
> > >> twitter @tlrx
> > >> https://github.com/tlrx
> > >>
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> >
>
>
>
> --
> -- Tanguy
> twitter @tlrx
> https://github.com/tlrx
>
--
-- Guozhang
Re: Kafka 0.8.1.1 / Can't read latest message
Posted by Tanguy tlrx <tl...@gmail.com>.
Hi Jay, Hi Guozhang,
Thanks for your response. I was suspecting something like that but did not
found the documentation entry that explain that.
Thanks again!
2014-07-17 23:41 GMT+02:00 Jay Kreps <ja...@gmail.com>:
> This makes sense if you think about it. If you want to start "now" you
> don't want the last message in the log, which could be seconds,
> minutes, days, or weeks old, you actually want the next message that
> comes in.
>
> -Jay
>
> On Thu, Jul 17, 2014 at 10:00 AM, Guozhang Wang <wa...@gmail.com>
> wrote:
> > Hello Tanguy,
> >
> > With "latest" the returned offset is actually the "logEndOffset", which
> is
> > the offset of the next message that will be appended to the log.
> >
> > Guozhang
> >
> >
> > On Thu, Jul 17, 2014 at 5:26 AM, Tanguy tlrx <tl...@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> I have a strange behavior with Kafka 0.8.1.1 where consumers do not
> >> retrieve the latest messages.
> >>
> >> I created a topic called "my-topic" with 8 partitions. My application
> >> creates a SimpleConsumer instance for each topic-partition pair and
> >> executes an OffsetRequest to retrieve the earliest and latest offsets
> for
> >> each partition.
> >>
> >> As an example, the partition 5 of the topic "my-topic" has an earliest
> >> offset at 4 and a latest offset at 9.
> >>
> >> Then, my application uses the same SimpleConsumer instance and executes
> a
> >> FetchRequest to read messages (starting from earliest) for that
> partition.
> >>
> >> Still with the example with partition 5, my application can read
> messages
> >> with offset 4,5,6,7,8 and then no more messages can be read. The same
> >> behavior exists for other partitions.
> >>
> >> I'm not sure that the message with offset "9" really exists, but the
> Kafka
> >> API returns a latest offset of 9 that let me thinking that this message
> >> exists. I tested with a HighLevelConsumer and it works the same way,
> >> reading messages 4...8 but not the last one.
> >>
> >> I did not notice this behavior with the previous version of Kafka I used
> >> (0.8.0-beta1).
> >>
> >> Am i missing something? Anyone with the same problem?
> >>
> >> Thanks
> >>
> >> -- Tanguy
> >> twitter @tlrx
> >> https://github.com/tlrx
> >>
> >
> >
> >
> > --
> > -- Guozhang
>
--
-- Tanguy
twitter @tlrx
https://github.com/tlrx
Re: Kafka 0.8.1.1 / Can't read latest message
Posted by Jay Kreps <ja...@gmail.com>.
This makes sense if you think about it. If you want to start "now" you
don't want the last message in the log, which could be seconds,
minutes, days, or weeks old, you actually want the next message that
comes in.
-Jay
On Thu, Jul 17, 2014 at 10:00 AM, Guozhang Wang <wa...@gmail.com> wrote:
> Hello Tanguy,
>
> With "latest" the returned offset is actually the "logEndOffset", which is
> the offset of the next message that will be appended to the log.
>
> Guozhang
>
>
> On Thu, Jul 17, 2014 at 5:26 AM, Tanguy tlrx <tl...@gmail.com> wrote:
>
>> Hi all,
>>
>> I have a strange behavior with Kafka 0.8.1.1 where consumers do not
>> retrieve the latest messages.
>>
>> I created a topic called "my-topic" with 8 partitions. My application
>> creates a SimpleConsumer instance for each topic-partition pair and
>> executes an OffsetRequest to retrieve the earliest and latest offsets for
>> each partition.
>>
>> As an example, the partition 5 of the topic "my-topic" has an earliest
>> offset at 4 and a latest offset at 9.
>>
>> Then, my application uses the same SimpleConsumer instance and executes a
>> FetchRequest to read messages (starting from earliest) for that partition.
>>
>> Still with the example with partition 5, my application can read messages
>> with offset 4,5,6,7,8 and then no more messages can be read. The same
>> behavior exists for other partitions.
>>
>> I'm not sure that the message with offset "9" really exists, but the Kafka
>> API returns a latest offset of 9 that let me thinking that this message
>> exists. I tested with a HighLevelConsumer and it works the same way,
>> reading messages 4...8 but not the last one.
>>
>> I did not notice this behavior with the previous version of Kafka I used
>> (0.8.0-beta1).
>>
>> Am i missing something? Anyone with the same problem?
>>
>> Thanks
>>
>> -- Tanguy
>> twitter @tlrx
>> https://github.com/tlrx
>>
>
>
>
> --
> -- Guozhang
Re: Kafka 0.8.1.1 / Can't read latest message
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Tanguy,
With "latest" the returned offset is actually the "logEndOffset", which is
the offset of the next message that will be appended to the log.
Guozhang
On Thu, Jul 17, 2014 at 5:26 AM, Tanguy tlrx <tl...@gmail.com> wrote:
> Hi all,
>
> I have a strange behavior with Kafka 0.8.1.1 where consumers do not
> retrieve the latest messages.
>
> I created a topic called "my-topic" with 8 partitions. My application
> creates a SimpleConsumer instance for each topic-partition pair and
> executes an OffsetRequest to retrieve the earliest and latest offsets for
> each partition.
>
> As an example, the partition 5 of the topic "my-topic" has an earliest
> offset at 4 and a latest offset at 9.
>
> Then, my application uses the same SimpleConsumer instance and executes a
> FetchRequest to read messages (starting from earliest) for that partition.
>
> Still with the example with partition 5, my application can read messages
> with offset 4,5,6,7,8 and then no more messages can be read. The same
> behavior exists for other partitions.
>
> I'm not sure that the message with offset "9" really exists, but the Kafka
> API returns a latest offset of 9 that let me thinking that this message
> exists. I tested with a HighLevelConsumer and it works the same way,
> reading messages 4...8 but not the last one.
>
> I did not notice this behavior with the previous version of Kafka I used
> (0.8.0-beta1).
>
> Am i missing something? Anyone with the same problem?
>
> Thanks
>
> -- Tanguy
> twitter @tlrx
> https://github.com/tlrx
>
--
-- Guozhang
Fwd: Kafka 0.8.1.1 / Can't read latest message
Posted by Tanguy tlrx <tl...@gmail.com>.
Hi all,
I have a strange behavior with Kafka 0.8.1.1 where consumers do not
retrieve the latest messages.
I created a topic called "my-topic" with 8 partitions. My application
creates a SimpleConsumer instance for each topic-partition pair and
executes an OffsetRequest to retrieve the earliest and latest offsets for
each partition.
As an example, the partition 5 of the topic "my-topic" has an earliest
offset at 4 and a latest offset at 9.
Then, my application uses the same SimpleConsumer instance and executes a
FetchRequest to read messages (starting from earliest) for that partition.
Still with the example with partition 5, my application can read messages
with offset 4,5,6,7,8 and then no more messages can be read. The same
behavior exists for other partitions.
I'm not sure that the message with offset "9" really exists, but the Kafka
API returns a latest offset of 9 that let me thinking that this message
exists. I tested with a HighLevelConsumer and it works the same way,
reading messages 4...8 but not the last one.
I did not notice this behavior with the previous version of Kafka I used
(0.8.0-beta1).
Am i missing something? Anyone with the same problem?
Thanks
-- Tanguy
twitter @tlrx
https://github.com/tlrx