what response does broker send when there is not message on the partition

[Ganesh Nikam <ga...@gslab.com>]

Hi All,

I have the kafka client which is fetching data from particular 
partition. On the broker there are there are some messages (say 100) on 
this partition and producers has stop producing the messages.
Now my consumer consumers these 100 messages and the next fetch offset 
is set to 101.  when I send the fetch request with 101 offset, I did 
receive some response from broker, but I don't know what it is.
I am not able to parse it. Can you please tell me what response does 
broker sends, when there is no message on that partition ?

I want to block my consumers till there is no message on the broker. How 
can I do that ?

- Ganesh

Re: what response does broker send when there is not message on the partition

[Gwen Shapira <gs...@cloudera.com>]

So "offsetRequest" should give you what you need. Or the
SimpleConsumer getOffsetsBefore. You don't really need a specific
consumer for this - just a topic, partition and timestamp.

On Tue, Apr 21, 2015 at 12:12 AM, Alexey Borschenko
<ab...@elance-odesk.com> wrote:
> We trying to implement *offset rewind* procedure. For this we need to be
> able to get offset for topic and partition for specified timestamp.
>
> On Tue, Apr 21, 2015 at 4:36 AM, Gwen Shapira <gs...@cloudera.com> wrote:
>
>> I believe it doesn't take consumers into account at all. Just the
>> offset available on the partition. Why would you need it to?
>>
>> On Mon, Apr 20, 2015 at 3:46 AM, Alexey Borschenko
>> <ab...@elance-odesk.com> wrote:
>> >> You can also send a FetchOffsetRequest and check for the last
>> >> available offset (log end offset) - this way you won't have to send a
>> >> fetch request that is likely to fail.
>> >
>> > Does this takes in account specific consumer offsets stored in Zookeeper?
>> >
>> > On Fri, Apr 17, 2015 at 5:57 PM, Gwen Shapira <gs...@cloudera.com>
>> wrote:
>> >
>> >> You should receive OffsetOutOfRange code (1) for the partition.
>> >>
>> >> You can also send a FetchOffsetRequest and check for the last
>> >> available offset (log end offset) - this way you won't have to send a
>> >> fetch request that is likely to fail.
>> >>
>> >> Are you implementing your own consumer from scatch? or using one of
>> >> the existing consumers?
>> >> If you use the high level consumer API, its default behavior for
>> >> hasNext() is to block.
>> >>
>> >> Gwen
>> >>
>> >> On Thu, Apr 16, 2015 at 10:19 PM, Ganesh Nikam <ga...@gslab.com>
>> >> wrote:
>> >> > Hi All,
>> >> >
>> >> > I have the kafka client which is fetching data from particular
>> >> partition. On
>> >> > the broker there are there are some messages (say 100) on this
>> partition
>> >> and
>> >> > producers has stop producing the messages.
>> >> > Now my consumer consumers these 100 messages and the next fetch
>> offset is
>> >> > set to 101.  when I send the fetch request with 101 offset, I did
>> receive
>> >> > some response from broker, but I don't know what it is.
>> >> > I am not able to parse it. Can you please tell me what response does
>> >> broker
>> >> > sends, when there is no message on that partition ?
>> >> >
>> >> > I want to block my consumers till there is no message on the broker.
>> How
>> >> can
>> >> > I do that ?
>> >> >
>> >> > - Ganesh
>> >>
>>

Re: what response does broker send when there is not message on the partition

[Alexey Borschenko <ab...@elance-odesk.com>]

We trying to implement *offset rewind* procedure. For this we need to be
able to get offset for topic and partition for specified timestamp.

On Tue, Apr 21, 2015 at 4:36 AM, Gwen Shapira <gs...@cloudera.com> wrote:

> I believe it doesn't take consumers into account at all. Just the
> offset available on the partition. Why would you need it to?
>
> On Mon, Apr 20, 2015 at 3:46 AM, Alexey Borschenko
> <ab...@elance-odesk.com> wrote:
> >> You can also send a FetchOffsetRequest and check for the last
> >> available offset (log end offset) - this way you won't have to send a
> >> fetch request that is likely to fail.
> >
> > Does this takes in account specific consumer offsets stored in Zookeeper?
> >
> > On Fri, Apr 17, 2015 at 5:57 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
> >
> >> You should receive OffsetOutOfRange code (1) for the partition.
> >>
> >> You can also send a FetchOffsetRequest and check for the last
> >> available offset (log end offset) - this way you won't have to send a
> >> fetch request that is likely to fail.
> >>
> >> Are you implementing your own consumer from scatch? or using one of
> >> the existing consumers?
> >> If you use the high level consumer API, its default behavior for
> >> hasNext() is to block.
> >>
> >> Gwen
> >>
> >> On Thu, Apr 16, 2015 at 10:19 PM, Ganesh Nikam <ga...@gslab.com>
> >> wrote:
> >> > Hi All,
> >> >
> >> > I have the kafka client which is fetching data from particular
> >> partition. On
> >> > the broker there are there are some messages (say 100) on this
> partition
> >> and
> >> > producers has stop producing the messages.
> >> > Now my consumer consumers these 100 messages and the next fetch
> offset is
> >> > set to 101.  when I send the fetch request with 101 offset, I did
> receive
> >> > some response from broker, but I don't know what it is.
> >> > I am not able to parse it. Can you please tell me what response does
> >> broker
> >> > sends, when there is no message on that partition ?
> >> >
> >> > I want to block my consumers till there is no message on the broker.
> How
> >> can
> >> > I do that ?
> >> >
> >> > - Ganesh
> >>
>

Re: what response does broker send when there is not message on the partition

[Gwen Shapira <gs...@cloudera.com>]

I believe it doesn't take consumers into account at all. Just the
offset available on the partition. Why would you need it to?

On Mon, Apr 20, 2015 at 3:46 AM, Alexey Borschenko
<ab...@elance-odesk.com> wrote:
>> You can also send a FetchOffsetRequest and check for the last
>> available offset (log end offset) - this way you won't have to send a
>> fetch request that is likely to fail.
>
> Does this takes in account specific consumer offsets stored in Zookeeper?
>
> On Fri, Apr 17, 2015 at 5:57 PM, Gwen Shapira <gs...@cloudera.com> wrote:
>
>> You should receive OffsetOutOfRange code (1) for the partition.
>>
>> You can also send a FetchOffsetRequest and check for the last
>> available offset (log end offset) - this way you won't have to send a
>> fetch request that is likely to fail.
>>
>> Are you implementing your own consumer from scatch? or using one of
>> the existing consumers?
>> If you use the high level consumer API, its default behavior for
>> hasNext() is to block.
>>
>> Gwen
>>
>> On Thu, Apr 16, 2015 at 10:19 PM, Ganesh Nikam <ga...@gslab.com>
>> wrote:
>> > Hi All,
>> >
>> > I have the kafka client which is fetching data from particular
>> partition. On
>> > the broker there are there are some messages (say 100) on this partition
>> and
>> > producers has stop producing the messages.
>> > Now my consumer consumers these 100 messages and the next fetch offset is
>> > set to 101.  when I send the fetch request with 101 offset, I did receive
>> > some response from broker, but I don't know what it is.
>> > I am not able to parse it. Can you please tell me what response does
>> broker
>> > sends, when there is no message on that partition ?
>> >
>> > I want to block my consumers till there is no message on the broker. How
>> can
>> > I do that ?
>> >
>> > - Ganesh
>>

Re: what response does broker send when there is not message on the partition

[Alexey Borschenko <ab...@elance-odesk.com>]

> You can also send a FetchOffsetRequest and check for the last
> available offset (log end offset) - this way you won't have to send a
> fetch request that is likely to fail.

Does this takes in account specific consumer offsets stored in Zookeeper?

On Fri, Apr 17, 2015 at 5:57 PM, Gwen Shapira <gs...@cloudera.com> wrote:

> You should receive OffsetOutOfRange code (1) for the partition.
>
> You can also send a FetchOffsetRequest and check for the last
> available offset (log end offset) - this way you won't have to send a
> fetch request that is likely to fail.
>
> Are you implementing your own consumer from scatch? or using one of
> the existing consumers?
> If you use the high level consumer API, its default behavior for
> hasNext() is to block.
>
> Gwen
>
> On Thu, Apr 16, 2015 at 10:19 PM, Ganesh Nikam <ga...@gslab.com>
> wrote:
> > Hi All,
> >
> > I have the kafka client which is fetching data from particular
> partition. On
> > the broker there are there are some messages (say 100) on this partition
> and
> > producers has stop producing the messages.
> > Now my consumer consumers these 100 messages and the next fetch offset is
> > set to 101.  when I send the fetch request with 101 offset, I did receive
> > some response from broker, but I don't know what it is.
> > I am not able to parse it. Can you please tell me what response does
> broker
> > sends, when there is no message on that partition ?
> >
> > I want to block my consumers till there is no message on the broker. How
> can
> > I do that ?
> >
> > - Ganesh
>

Re: what response does broker send when there is not message on the partition

[Gwen Shapira <gs...@cloudera.com>]

You should receive OffsetOutOfRange code (1) for the partition.

You can also send a FetchOffsetRequest and check for the last
available offset (log end offset) - this way you won't have to send a
fetch request that is likely to fail.

Are you implementing your own consumer from scatch? or using one of
the existing consumers?
If you use the high level consumer API, its default behavior for
hasNext() is to block.

Gwen

On Thu, Apr 16, 2015 at 10:19 PM, Ganesh Nikam <ga...@gslab.com> wrote:
> Hi All,
>
> I have the kafka client which is fetching data from particular partition. On
> the broker there are there are some messages (say 100) on this partition and
> producers has stop producing the messages.
> Now my consumer consumers these 100 messages and the next fetch offset is
> set to 101.  when I send the fetch request with 101 offset, I did receive
> some response from broker, but I don't know what it is.
> I am not able to parse it. Can you please tell me what response does broker
> sends, when there is no message on that partition ?
>
> I want to block my consumers till there is no message on the broker. How can
> I do that ?
>
> - Ganesh