Kafka does not contain message though the producer callback did not contain any exception

[Konrad Baczynski <ko...@empirica.io.INVALID>]

Hello,
I have an issue. I hope you are able to resolve it.
I attached my test code.*
*

*Environment:*
3 node clustered kafka (2.12-2.1.1)
zookeeper (3.4)*
*

*Kafka topic description:*

Topic:test_topic PartitionCount:5 ReplicationFactor:2 
Configs:min.insync.replicas=2

Topic: test_topic Partition: 0 Leader: 3 Replicas: 3,1 Isr: 3,1
Topic: test_topic Partition: 1 Leader: 1 Replicas: 1,2 Isr: 2,1
Topic: test_topic Partition: 2 Leader: 2 Replicas: 2,3 Isr: 3,2
Topic: test_topic Partition: 3 Leader: 3 Replicas: 3,2 Isr: 3,2
Topic: test_topic Partition: 4 Leader: 1 Replicas: 1,3 Isr: 3,1

*Used kafka client in tests:*
org.apache.servicemix.bundles/org.apache.servicemix.bundles.kafka-clients/2.4.0_1*
*

*Test description:*
The test starts a consumer and a producer in separate threads. The 
producer produces a message every SLEEP_INTERVAL_MS. If kafka is 
available, it receives the message and the consumer is able to consume 
it. When kafka is not available, and you receive TimeoutException, you 
persist the message to an in-memory backup message store. The main 
thread tries to restore the messages every 
PERSIST_BACKUP_CHECK_INTERVAL_MS. When kafka is available again, you 
successfully restore messages but kafka doesn’t contain some of them 
(the consumer is not able to consume them).*
*

*Issue reproduction:*
1. All the kafka nodes work.

2. Start the/shouldConsumeAllProducedMessages() /test.
3. After consuming some messages (before consuming all) stop all the 
kafka nodes.
4. Wait until org.apache.kafka.common.errors.TimeoutException appears.
5. Start kafka nodes.*
*

*Result:*
Many messages couldn’t be consumed because kafka doesn’t contain them, 
though the producer didn’t have any exception for them in the callback.

*Question:
*In my opinion either kafka should contain the message and the consumer 
should be able to take it or the callback should contain an exception. 
Could you confirm? Could you explain the situation I have?
**

Best regards
Konrad Bączyński

-- 
Konrad Bączyński
Software developer
konrad.baczynski@empirica.io

Empirica Sp. z o.o.
ul. Curie-Skłodowskiej 12
50-381 Wrocław

www.empirica.pl
  
NIP 8982201370, REGON 021955151

Zarejestrowana w Sądzie Rejonowym dla Wrocławia-Fabrycznej, VI Wydział Gospodarczy Krajowego Rejestru Sądowego pod numerem KRS 0000914199. Kapitał zakładowy: 650.000,00 zł opłacony w całości.