Using startingOffsets latest - no data from structured streaming kafka query

[Ruijing Li <li...@gmail.com>]

Hi all,

Apologies if this has been asked before, but I could not find the answer to
this question. We have a structured streaming job, but for some reason, if
we use startingOffsets = latest with foreachbatch mode, it doesn’t produce
any data.

Rather, in logs I see it repeats the message “ Fetcher [Consumer] Resetting
offset for partition to offset” over and over again..

However with startingOffsets=earliest, we don’t get this issue. I’m
wondering then how we can use startingOffsets=latest as I wish to start
from the latest offset available.
-- 
Cheers,
Ruijing Li

Re: Using startingOffsets latest - no data from structured streaming kafka query

[Ruijing Li <li...@gmail.com>]

For some reason, after restarting the app and trying again, latest now
works as expected. Not sure why it didn’t work before.

On Tue, Apr 21, 2020 at 1:46 PM Ruijing Li <li...@gmail.com> wrote:

> Yes, we did. But for some reason latest does not show them. The count is
> always 0.
>
> On Sun, Apr 19, 2020 at 3:42 PM Jungtaek Lim <ka...@gmail.com>
> wrote:
>
>> Did you provide more records to topic "after" you started the query?
>> That's the only one I can imagine based on such information.
>>
>> On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li <li...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> Apologies if this has been asked before, but I could not find the answer
>>> to this question. We have a structured streaming job, but for some reason,
>>> if we use startingOffsets = latest with foreachbatch mode, it doesn’t
>>> produce any data.
>>>
>>> Rather, in logs I see it repeats the message “ Fetcher [Consumer]
>>> Resetting offset for partition to offset” over and over again..
>>>
>>> However with startingOffsets=earliest, we don’t get this issue. I’m
>>> wondering then how we can use startingOffsets=latest as I wish to start
>>> from the latest offset available.
>>> --
>>> Cheers,
>>> Ruijing Li
>>>
>> --
> Cheers,
> Ruijing Li
>
-- 
Cheers,
Ruijing Li

Re: Using startingOffsets latest - no data from structured streaming kafka query

[Ruijing Li <li...@gmail.com>]

Yes, we did. But for some reason latest does not show them. The count is
always 0.

On Sun, Apr 19, 2020 at 3:42 PM Jungtaek Lim <ka...@gmail.com>
wrote:

> Did you provide more records to topic "after" you started the query?
> That's the only one I can imagine based on such information.
>
> On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li <li...@gmail.com> wrote:
>
>> Hi all,
>>
>> Apologies if this has been asked before, but I could not find the answer
>> to this question. We have a structured streaming job, but for some reason,
>> if we use startingOffsets = latest with foreachbatch mode, it doesn’t
>> produce any data.
>>
>> Rather, in logs I see it repeats the message “ Fetcher [Consumer]
>> Resetting offset for partition to offset” over and over again..
>>
>> However with startingOffsets=earliest, we don’t get this issue. I’m
>> wondering then how we can use startingOffsets=latest as I wish to start
>> from the latest offset available.
>> --
>> Cheers,
>> Ruijing Li
>>
> --
Cheers,
Ruijing Li

Re: Using startingOffsets latest - no data from structured streaming kafka query

[Jungtaek Lim <ka...@gmail.com>]

Did you provide more records to topic "after" you started the query? That's
the only one I can imagine based on such information.

On Fri, Apr 17, 2020 at 9:13 AM Ruijing Li <li...@gmail.com> wrote:

> Hi all,
>
> Apologies if this has been asked before, but I could not find the answer
> to this question. We have a structured streaming job, but for some reason,
> if we use startingOffsets = latest with foreachbatch mode, it doesn’t
> produce any data.
>
> Rather, in logs I see it repeats the message “ Fetcher [Consumer]
> Resetting offset for partition to offset” over and over again..
>
> However with startingOffsets=earliest, we don’t get this issue. I’m
> wondering then how we can use startingOffsets=latest as I wish to start
> from the latest offset available.
> --
> Cheers,
> Ruijing Li
>