[jira] [Commented] (CAMEL-16914) camel-kafka: possible corruption of idempotency messages when using KafkaIdempotentRepository

["Otavio Rodolfo Piske (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408731#comment-17408731 ] 

Otavio Rodolfo Piske commented on CAMEL-16914:
----------------------------------------------

This ALWAYS happen in a block of 128 records: it works OK for 128 records, then skip 128, then works OK for the subsequent 128 and so on.

 

After a a preliminary observation, I think this is being caused because the code is assuming the key is always String: https://github.com/apache/camel/blob/camel-3.11.x/components/camel-kafka/src/main/java/org/apache/camel/processor/idempotent/kafka/KafkaIdempotentRepository.java#L456

 

This seems to be messing up with the add/confirms/remove logic:

 
{code:java}
[1] 2021-09-02 10:42:51,230 DEBUG [org.apa.cam.pro.ide.kaf.KafkaIdempotentRepository$TopicPoller] (Camel (camel-1) thread #0 - KafkaIdempotentRepository) Adding to cache messageId:q 
[1] 2021-09-02 10:42:52,229 DEBUG [org.apa.cam.pro.ide.kaf.KafkaIdempotentRepository$TopicPoller] (Camel (camel-1) thread #0 - KafkaIdempotentRepository) Adding to cache messageId:r{code}

> camel-kafka: possible corruption of idempotency messages when using KafkaIdempotentRepository 
> ----------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-16914
>                 URL: https://issues.apache.org/jira/browse/CAMEL-16914
>             Project: Camel
>          Issue Type: Task
>          Components: camel-kafka
>    Affects Versions: 3.11.1
>            Reporter: Otavio Rodolfo Piske
>            Priority: Major
>         Attachments: FakeDataProducer.java, MessageLossReproducer.java
>
>
> [^FakeDataProducer.java][^MessageLossReproducer.java]I noticed that when using the KafkaIdempotentRepository, sometimes it seems that it is losing or corrupting the ID information, causing loss of messages.
>  
> When reading the topic used for publishing the idempotency data with kafkacat I get:
>  
> {code:java}
> ERROR: Failed to format message in fake-data-inserts [1] at offset 39: expected key end-of-input, but 2 bytes remaining : terminating{code}
>  
> As a result, this causes the loss of messages because the consumers cannot go further than the corrupted record.
>  
> I found this while using Camel K and Strimzi (on a setup with 3 Kafkas using 1 topic with 3 partitions).
>  
> I couldn't test w/ 3.12 yet, but I suspect it would also have the same problem.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)