kafka offset not working

[Benjamin Cuthbert <cu...@gmail.com>]

All

Our kafka consumer has the group.id property but when we receive messages
on the channel when not connect and then reconnect we don't get the message.

The kafka console consumer when running "-group testben" works perfectly
fine.

Is there some other place to setup flink to read from the latest offset?

Re: kafka offset not working

[Benjamin Cuthbert <cu...@gmail.com>]

So looks like checkpoints are not working, each time a job is stopped and
started it is a different checkpoint. Using savepoints manually seems to
work if you do a call to save the savepoint and then run it with -s
savepoint location. Why does checkpoints not work?

On Mon, Oct 7, 2019 at 10:25 PM Eduardo Winpenny Tejedor <
eduardo.winpenny@gmail.com> wrote:

> Flink only uses the specified starting point consumer-side setting the
> first time the application is deployed. After that Flink will store the
> Kafka offsets as part of the (source) operator state and when restarted
> from a checkpoint/savepoint it'll continue to consume from where it left
> off.
>
> Hope that makes sense,
> Eduardo
>
>
> On Mon, 7 Oct 2019, 16:52 Benjamin Cuthbert, <cu...@gmail.com>
> wrote:
>
>> All
>>
>> Our kafka consumer has the group.id property but when we receive
>> messages on the channel when not connect and then reconnect we don't get
>> the message.
>>
>> The kafka console consumer when running "-group testben" works perfectly
>> fine.
>>
>> Is there some other place to setup flink to read from the latest offset?
>>
>

Re: kafka offset not working

[Eduardo Winpenny Tejedor <ed...@gmail.com>]

Flink only uses the specified starting point consumer-side setting the
first time the application is deployed. After that Flink will store the
Kafka offsets as part of the (source) operator state and when restarted
from a checkpoint/savepoint it'll continue to consume from where it left
off.

Hope that makes sense,
Eduardo


On Mon, 7 Oct 2019, 16:52 Benjamin Cuthbert, <cu...@gmail.com> wrote:

> All
>
> Our kafka consumer has the group.id property but when we receive messages
> on the channel when not connect and then reconnect we don't get the message.
>
> The kafka console consumer when running "-group testben" works perfectly
> fine.
>
> Is there some other place to setup flink to read from the latest offset?
>