You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Praneeth Ramesh <sr...@gmail.com> on 2022/04/09 01:21:38 UTC
Flink Kafka Issue with EXACTLY_ONCE semantics
Hi All
I have a job which reads from kafka and applies some transactions and
writes the data back to kafka topics.
When I use the exactly once semantics I see that the kafka producer is not
initialized and the operator task is hanging in INITIALIZING state and
fails eventually.
It works fine when I change the mode to AT_LEAST_ONCE. I see that in this
mode the transactionId is null in the producer config and
enable.idempotence is false.
But when I enable EXACTLY_ONCE I see that the producer configs having
transactionId which I set and enable.idempotence to true. From trace logs
what I can see is the transaction manager is initializing and
reinitializing the transactionId again and again and does not make any
progress.
I tried to analyze the thread dump and see that there are threads blocked
at
at
org.apache.kafka.clients.producer.internals.TransactionalRequestResult.await(TransactionalRequestResult.java:68)
at
org.apache.kafka.clients.producer.KafkaProducer.initTransactions(KafkaProducer.java:632)
at
org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.initTransactionId(FlinkKafkaInternalProducer.java:156)
I made sure that there are no ACLs problems with brokers. Any idea what
could be the problem?
Attaching the TM logs where the application is reinitializing the
transactions.
Thanks for any help or pointers.
--
Regards
Praneeth Ramesh
Re: Flink Kafka Issue with EXACTLY_ONCE semantics
Posted by Frank Dekervel <fr...@kapernikov.com>.
Hello Praneeth,
that looks correct then. In that case, maybe somebody else can chime in.
Are you aware of this post on SO:
https://stackoverflow.com/questions/45047876/apache-kafka-exactly-once-implementation-not-sending-messages
Frank
On Sat, Apr 9, 2022 at 7:42 PM Praneeth Ramesh <sr...@gmail.com>
wrote:
> Hi Frank
>
> Thanks for the response.
> I see that the min.isr value is 2 and replication factor is 4 in my case.
> Do you see any issues with these values..?
>
> Thank you in advance
>
>
> On Fri, Apr 8, 2022 at 11:28 PM Frank Dekervel <fr...@kapernikov.com>
> wrote:
>
>> Hello,
>>
>> Check if your topic replication factor is not below min.isr setting of
>> Kafka. I had the same problem and that was it for me.
>>
>> Frank
>>
>> Op za 9 apr. 2022 04:01 schreef Praneeth Ramesh <sr...@gmail.com>:
>>
>>> Hi All
>>>
>>> I have a job which reads from kafka and applies some transactions and
>>> writes the data back to kafka topics.
>>> When I use the exactly once semantics I see that the kafka producer is
>>> not initialized and the operator task is hanging in INITIALIZING state and
>>> fails eventually.
>>>
>>> It works fine when I change the mode to AT_LEAST_ONCE. I see that in
>>> this mode the transactionId is null in the producer config and
>>> enable.idempotence is false.
>>>
>>> But when I enable EXACTLY_ONCE I see that the producer configs having
>>> transactionId which I set and enable.idempotence to true. From trace logs
>>> what I can see is the transaction manager is initializing and
>>> reinitializing the transactionId again and again and does not make any
>>> progress.
>>>
>>> I tried to analyze the thread dump and see that there are threads
>>> blocked at
>>> at
>>> org.apache.kafka.clients.producer.internals.TransactionalRequestResult.await(TransactionalRequestResult.java:68)
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.initTransactions(KafkaProducer.java:632)
>>> at
>>> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.initTransactionId(FlinkKafkaInternalProducer.java:156)
>>>
>>>
>>> I made sure that there are no ACLs problems with brokers. Any idea what
>>> could be the problem?
>>>
>>> Attaching the TM logs where the application is reinitializing the
>>> transactions.
>>>
>>>
>>> Thanks for any help or pointers.
>>>
>>>
>>>
>>>
>>> --
>>> Regards
>>> Praneeth Ramesh
>>>
>>
>
> --
> Regards
> Praneeth Ramesh
>
--
[image: Kapernikov] <https://kapernikov.com/>
Frank Dekervel
+32 473 94 34 21 <+32473943421>
www.kapernikov.com <https://kapernikov.com/>
[image: Blog] <https://www.kapernikov.com/emailsigs/blog/redirector.php>
Re: Flink Kafka Issue with EXACTLY_ONCE semantics
Posted by Praneeth Ramesh <sr...@gmail.com>.
Hi Frank
Thanks for the response.
I see that the min.isr value is 2 and replication factor is 4 in my case.
Do you see any issues with these values..?
Thank you in advance
On Fri, Apr 8, 2022 at 11:28 PM Frank Dekervel <fr...@kapernikov.com> wrote:
> Hello,
>
> Check if your topic replication factor is not below min.isr setting of
> Kafka. I had the same problem and that was it for me.
>
> Frank
>
> Op za 9 apr. 2022 04:01 schreef Praneeth Ramesh <sr...@gmail.com>:
>
>> Hi All
>>
>> I have a job which reads from kafka and applies some transactions and
>> writes the data back to kafka topics.
>> When I use the exactly once semantics I see that the kafka producer is
>> not initialized and the operator task is hanging in INITIALIZING state and
>> fails eventually.
>>
>> It works fine when I change the mode to AT_LEAST_ONCE. I see that in this
>> mode the transactionId is null in the producer config and
>> enable.idempotence is false.
>>
>> But when I enable EXACTLY_ONCE I see that the producer configs having
>> transactionId which I set and enable.idempotence to true. From trace logs
>> what I can see is the transaction manager is initializing and
>> reinitializing the transactionId again and again and does not make any
>> progress.
>>
>> I tried to analyze the thread dump and see that there are threads blocked
>> at
>> at
>> org.apache.kafka.clients.producer.internals.TransactionalRequestResult.await(TransactionalRequestResult.java:68)
>> at
>> org.apache.kafka.clients.producer.KafkaProducer.initTransactions(KafkaProducer.java:632)
>> at
>> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.initTransactionId(FlinkKafkaInternalProducer.java:156)
>>
>>
>> I made sure that there are no ACLs problems with brokers. Any idea what
>> could be the problem?
>>
>> Attaching the TM logs where the application is reinitializing the
>> transactions.
>>
>>
>> Thanks for any help or pointers.
>>
>>
>>
>>
>> --
>> Regards
>> Praneeth Ramesh
>>
>
--
Regards
Praneeth Ramesh
Re: Flink Kafka Issue with EXACTLY_ONCE semantics
Posted by Frank Dekervel <fr...@kapernikov.com>.
Hello,
Check if your topic replication factor is not below min.isr setting of
Kafka. I had the same problem and that was it for me.
Frank
Op za 9 apr. 2022 04:01 schreef Praneeth Ramesh <sr...@gmail.com>:
> Hi All
>
> I have a job which reads from kafka and applies some transactions and
> writes the data back to kafka topics.
> When I use the exactly once semantics I see that the kafka producer is not
> initialized and the operator task is hanging in INITIALIZING state and
> fails eventually.
>
> It works fine when I change the mode to AT_LEAST_ONCE. I see that in this
> mode the transactionId is null in the producer config and
> enable.idempotence is false.
>
> But when I enable EXACTLY_ONCE I see that the producer configs having
> transactionId which I set and enable.idempotence to true. From trace logs
> what I can see is the transaction manager is initializing and
> reinitializing the transactionId again and again and does not make any
> progress.
>
> I tried to analyze the thread dump and see that there are threads blocked
> at
> at
> org.apache.kafka.clients.producer.internals.TransactionalRequestResult.await(TransactionalRequestResult.java:68)
> at
> org.apache.kafka.clients.producer.KafkaProducer.initTransactions(KafkaProducer.java:632)
> at
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducer.initTransactionId(FlinkKafkaInternalProducer.java:156)
>
>
> I made sure that there are no ACLs problems with brokers. Any idea what
> could be the problem?
>
> Attaching the TM logs where the application is reinitializing the
> transactions.
>
>
> Thanks for any help or pointers.
>
>
>
>
> --
> Regards
> Praneeth Ramesh
>