You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Gyula Fóra <gy...@gmail.com> on 2018/02/21 10:21:58 UTC
Strange Kafka consumer behaviour
Hi,
I have observed a weird behaviour when changing kafka topics when restoring
from a checkpoint.
It seems that the job started consuming both the topics from the state, and
the new topic that I assigned. This happened while changing from kafka 08
to kafka 10.
Is this expected?
Thanks,
Gyula
Re: Strange Kafka consumer behaviour
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Gyula,
Are you using Flink 1.4.x, and have partition discovery enabled?
If yes, then both the state of previously existing topics, as well as
partitions of the newly specified topics will be consumed.
Cheers,
Gordon
On Tue, Mar 20, 2018 at 6:01 AM, Ankit Chaudhary <my...@gmail.com> wrote:
> Did you changed the kafka group id?
>
> On Mon, Mar 19, 2018 at 9:54 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> I think it's because it's just taking the state for the existing topics
>> and starts reading from that.
>>
>> A workaround could be to change the uid of the operator to avoid reading
>> the old state.
>>
>> Aljoscha
>>
>> > On 21. Feb 2018, at 02:21, Gyula Fóra <gy...@gmail.com> wrote:
>> >
>> > Hi,
>> > I have observed a weird behaviour when changing kafka topics when
>> restoring from a checkpoint.
>> >
>> > It seems that the job started consuming both the topics from the state,
>> and the new topic that I assigned. This happened while changing from kafka
>> 08 to kafka 10.
>> >
>> > Is this expected?
>> >
>> > Thanks,
>> > Gyula
>>
>>
>
Re: Strange Kafka consumer behaviour
Posted by Ankit Chaudhary <my...@gmail.com>.
Did you changed the kafka group id?
On Mon, Mar 19, 2018 at 9:54 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> I think it's because it's just taking the state for the existing topics
> and starts reading from that.
>
> A workaround could be to change the uid of the operator to avoid reading
> the old state.
>
> Aljoscha
>
> > On 21. Feb 2018, at 02:21, Gyula Fóra <gy...@gmail.com> wrote:
> >
> > Hi,
> > I have observed a weird behaviour when changing kafka topics when
> restoring from a checkpoint.
> >
> > It seems that the job started consuming both the topics from the state,
> and the new topic that I assigned. This happened while changing from kafka
> 08 to kafka 10.
> >
> > Is this expected?
> >
> > Thanks,
> > Gyula
>
>
Re: Strange Kafka consumer behaviour
Posted by Aljoscha Krettek <al...@apache.org>.
I think it's because it's just taking the state for the existing topics and starts reading from that.
A workaround could be to change the uid of the operator to avoid reading the old state.
Aljoscha
> On 21. Feb 2018, at 02:21, Gyula Fóra <gy...@gmail.com> wrote:
>
> Hi,
> I have observed a weird behaviour when changing kafka topics when restoring from a checkpoint.
>
> It seems that the job started consuming both the topics from the state, and the new topic that I assigned. This happened while changing from kafka 08 to kafka 10.
>
> Is this expected?
>
> Thanks,
> Gyula