You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Guozhang Wang (JIRA)" <ji...@apache.org> on 2018/12/21 00:09:00 UTC
[jira] [Commented] (KAFKA-7716) Unprocessed messages when Broker
fails
[ https://issues.apache.org/jira/browse/KAFKA-7716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726341#comment-16726341 ]
Guozhang Wang commented on KAFKA-7716:
--------------------------------------
[~Finbarr] Thanks for the PR, I've read through the SO thread as well. The first thing I'd ask is also about the replication factor: if your internal topics are not configured with replication factor >= 3 then when a broker fails it is still possible that you'll lose your acked messages, even with EOS turned on.
> Unprocessed messages when Broker fails
> --------------------------------------
>
> Key: KAFKA-7716
> URL: https://issues.apache.org/jira/browse/KAFKA-7716
> Project: Kafka
> Issue Type: Bug
> Components: core, streams
> Affects Versions: 1.0.0, 2.0.1
> Reporter: Finbarr Naughton
> Priority: Major
>
> This occurs when running on Kubernetes on bare metal.
> A Streams application with a single topology listening to two input topics A and B. A is read as a GlobalKTable, B as a KStream. The topology joins the stream to the GKTable and writes an updated message to topic A. The application is configured to use exactly_once processing.
> There are three worker nodes. Kafka brokers are deployed as a statefulset on the three nodes using the helm chart from here -[https://github.com/helm/charts/tree/master/incubator/kafka]
> The application has three instances spread across the three nodes.
> During a test, topic A is pre-populated with 50k messages over 5 minutes. Then 50k messages with the same key-set are sent to topic B over 5 minutes. The expected behaviour is that Topic A will contain 50k updated messages afterwards. While all brokers are available this is the case, even when one of the application pods is deleted.
> When a broker fails, however, a few expected updated messages fail to appear on topic A despite their existence on topic B.
>
> More complete description here - [https://stackoverflow.com/questions/53557247/some-events-unprocessed-by-kafka-streams-application-on-kubernetes-on-bare-metal]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)