You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Milind Vaidya <ka...@gmail.com> on 2023/02/06 19:18:11 UTC
Incompatible KafkaProducer version
Hi
Getting the following exception while trying to upgrade Kafka Libraries
Flink version : 1.9.3
Old Kafka version : 2.2.1.8
New Kafka version : 2.4.1.4
Found a ticket : https://issues.apache.org/jira/browse/FLINK-9690
But that seems to be between 0.11 to 2.0.0.
Cause : Caused by: java.lang.NoSuchFieldException: nextSequence
Detail Exception :
java.lang.RuntimeException: Incompatible KafkaProducer version
at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:322)
at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:313)
at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.resumeTransaction(FlinkKafkaInternalProducer.java:203)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.recoverAndCommit(FlinkKafkaProducer.java:919)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.recoverAndCommit(FlinkKafkaProducer.java:98)
at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.recoverAndCommitInternal(TwoPhaseCommitSinkFunction.java:392)
at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:350)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initializeState(FlinkKafkaProducer.java:1037)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:901)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:415)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchFieldException: nextSequence
at java.lang.Class.getDeclaredField(Class.java:2070)
at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:318)
... 16 more
Re: Incompatible KafkaProducer version
Posted by Martijn Visser <ma...@apache.org>.
Hi Milind Vaidya,
I would recommend checking out the release notes for each version that
you're upgrading to and/or skipping.
Best regards,
Martijn
On Mon, Feb 6, 2023 at 10:46 PM Milind Vaidya <ka...@gmail.com> wrote:
> Thanks for your suggestion Martjin.
>
> I am in the process of upgrading but this is kind of a blocker for coming
> to a sane state before proceeding.
>
> Is there any generic guide for version upgrading ?
>
>
>
>
> On Mon, Feb 6, 2023 at 11:38 AM Martijn Visser <ma...@apache.org>
> wrote:
>
>> Hi Milind Vaidya,
>>
>> I would highly recommend upgrading your Flink cluster and
>> applications. Flink 1.9 was released in August 2019 and is no longer
>> supported by the community. Newer Kafka versions are supported on
>> newer Flink versions.
>>
>> Best regards,
>>
>> Martijn
>>
>> Op ma 6 feb. 2023 om 20:19 schreef Milind Vaidya <ka...@gmail.com>:
>> >
>> > Hi
>> >
>> > Getting the following exception while trying to upgrade Kafka Libraries
>> >
>> > Flink version : 1.9.3
>> > Old Kafka version : 2.2.1.8
>> > New Kafka version : 2.4.1.4
>> >
>> > Found a ticket : https://issues.apache.org/jira/browse/FLINK-9690
>> >
>> > But that seems to be between 0.11 to 2.0.0.
>> >
>> > Cause : Caused by: java.lang.NoSuchFieldException: nextSequence
>> >
>> > Detail Exception :
>> >
>> > java.lang.RuntimeException: Incompatible KafkaProducer version
>> >
>> > at
>> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:322)
>> >
>> > at
>> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:313)
>> >
>> > at
>> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.resumeTransaction(FlinkKafkaInternalProducer.java:203)
>> >
>> > at
>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.recoverAndCommit(FlinkKafkaProducer.java:919)
>> >
>> > at
>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.recoverAndCommit(FlinkKafkaProducer.java:98)
>> >
>> > at
>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.recoverAndCommitInternal(TwoPhaseCommitSinkFunction.java:392)
>> >
>> > at
>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:350)
>> >
>> > at
>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initializeState(FlinkKafkaProducer.java:1037)
>> >
>> > at
>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>> >
>> > at
>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>> >
>> > at
>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>> >
>> > at
>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
>> >
>> > at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:901)
>> >
>> > at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:415)
>> >
>> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
>> >
>> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
>> >
>> > at java.lang.Thread.run(Thread.java:748)
>> >
>> > Caused by: java.lang.NoSuchFieldException: nextSequence
>> >
>> > at java.lang.Class.getDeclaredField(Class.java:2070)
>> >
>> > at
>> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:318)
>> >
>> > ... 16 more
>>
>
Re: Incompatible KafkaProducer version
Posted by Milind Vaidya <ka...@gmail.com>.
Thanks for your suggestion Martjin.
I am in the process of upgrading but this is kind of a blocker for coming
to a sane state before proceeding.
Is there any generic guide for version upgrading ?
On Mon, Feb 6, 2023 at 11:38 AM Martijn Visser <ma...@apache.org>
wrote:
> Hi Milind Vaidya,
>
> I would highly recommend upgrading your Flink cluster and
> applications. Flink 1.9 was released in August 2019 and is no longer
> supported by the community. Newer Kafka versions are supported on
> newer Flink versions.
>
> Best regards,
>
> Martijn
>
> Op ma 6 feb. 2023 om 20:19 schreef Milind Vaidya <ka...@gmail.com>:
> >
> > Hi
> >
> > Getting the following exception while trying to upgrade Kafka Libraries
> >
> > Flink version : 1.9.3
> > Old Kafka version : 2.2.1.8
> > New Kafka version : 2.4.1.4
> >
> > Found a ticket : https://issues.apache.org/jira/browse/FLINK-9690
> >
> > But that seems to be between 0.11 to 2.0.0.
> >
> > Cause : Caused by: java.lang.NoSuchFieldException: nextSequence
> >
> > Detail Exception :
> >
> > java.lang.RuntimeException: Incompatible KafkaProducer version
> >
> > at
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:322)
> >
> > at
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:313)
> >
> > at
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.resumeTransaction(FlinkKafkaInternalProducer.java:203)
> >
> > at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.recoverAndCommit(FlinkKafkaProducer.java:919)
> >
> > at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.recoverAndCommit(FlinkKafkaProducer.java:98)
> >
> > at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.recoverAndCommitInternal(TwoPhaseCommitSinkFunction.java:392)
> >
> > at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:350)
> >
> > at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initializeState(FlinkKafkaProducer.java:1037)
> >
> > at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> >
> > at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> >
> > at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> >
> > at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
> >
> > at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:901)
> >
> > at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:415)
> >
> > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
> >
> > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
> >
> > at java.lang.Thread.run(Thread.java:748)
> >
> > Caused by: java.lang.NoSuchFieldException: nextSequence
> >
> > at java.lang.Class.getDeclaredField(Class.java:2070)
> >
> > at
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:318)
> >
> > ... 16 more
>
Re: Incompatible KafkaProducer version
Posted by Martijn Visser <ma...@apache.org>.
Hi Milind Vaidya,
I would highly recommend upgrading your Flink cluster and
applications. Flink 1.9 was released in August 2019 and is no longer
supported by the community. Newer Kafka versions are supported on
newer Flink versions.
Best regards,
Martijn
Op ma 6 feb. 2023 om 20:19 schreef Milind Vaidya <ka...@gmail.com>:
>
> Hi
>
> Getting the following exception while trying to upgrade Kafka Libraries
>
> Flink version : 1.9.3
> Old Kafka version : 2.2.1.8
> New Kafka version : 2.4.1.4
>
> Found a ticket : https://issues.apache.org/jira/browse/FLINK-9690
>
> But that seems to be between 0.11 to 2.0.0.
>
> Cause : Caused by: java.lang.NoSuchFieldException: nextSequence
>
> Detail Exception :
>
> java.lang.RuntimeException: Incompatible KafkaProducer version
>
> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:322)
>
> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:313)
>
> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.resumeTransaction(FlinkKafkaInternalProducer.java:203)
>
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.recoverAndCommit(FlinkKafkaProducer.java:919)
>
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.recoverAndCommit(FlinkKafkaProducer.java:98)
>
> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.recoverAndCommitInternal(TwoPhaseCommitSinkFunction.java:392)
>
> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:350)
>
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.initializeState(FlinkKafkaProducer.java:1037)
>
> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>
> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>
> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281)
>
> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:901)
>
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:415)
>
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
>
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.NoSuchFieldException: nextSequence
>
> at java.lang.Class.getDeclaredField(Class.java:2070)
>
> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaInternalProducer.getValue(FlinkKafkaInternalProducer.java:318)
>
> ... 16 more