You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2017/09/07 15:26:01 UTC
[jira] [Commented] (KAFKA-3039) Temporary loss of leader resulted
in log being completely truncated
[ https://issues.apache.org/jira/browse/KAFKA-3039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157087#comment-16157087 ]
Jun Rao commented on KAFKA-3039:
--------------------------------
Hmm, interesting. Not sure why this is happening. Currently, when a broker is restarted, we expect the very first LeaderAndIsrRequest to include all partitions on this broker. Partitions not included in that request will later be initialized with a 0 HW. Could you check if the partition is included in the state-change log of that broker? You should see sth like the following.
TRACE Broker 1 received LeaderAndIsr request PartitionState(controllerEpoch=1, leader=1, leaderEpoch=0, isr=1,2, zkVersion=0, replicas=1,2, isNew=true) correlation id 2 from controller 0 epoch 1 for partition [test,0] (state.change.logger)
> Temporary loss of leader resulted in log being completely truncated
> -------------------------------------------------------------------
>
> Key: KAFKA-3039
> URL: https://issues.apache.org/jira/browse/KAFKA-3039
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.9.0.0
> Environment: Debian 3.2.54-2 x86_64 GNU/Linux
> Reporter: Imran Patel
> Priority: Critical
> Labels: reliability
>
> We had an event recently where the temporarily loss of a leader for a partition (during a manual restart), resulted in the leader coming back with no high watermark state and truncating its log to zero. Logs (attached below) indicate that it did have the data but not the commit state. How is this possible?
> Leader (broker 3)
> [2015-12-18 21:19:44,666] INFO Completed load of log messages-14 with log end offset 14175963374 (kafka.log.Log)
> [2015-12-18 21:19:45,170] INFO Partition [messages,14] on broker 3: No checkpointed highwatermark is found for partition [messages,14] (kafka.cluster.Partition)
> [2015-12-18 21:19:45,238] INFO Truncating log messages-14 to offset 0. (kafka.log.Log)
> [2015-12-18 21:20:34,066] INFO Partition [messages,14] on broker 3: Expanding ISR for partition [messages,14] from 3 to 3,10 (kafka.cluster.Partition)
> Replica (broker 10)
> [2015-12-18 21:19:19,525] INFO Partition [messages,14] on broker 10: Shrinking ISR for partition [messages,14] from 3,10,4 to 10,4 (kafka.cluster.Partition)
> [2015-12-18 21:20:34,049] ERROR [ReplicaFetcherThread-0-3], Current offset 14175984203 for partition [messages,14] out of range; reset offset to 35977 (kafka.server.ReplicaFetcherThread)
> [2015-12-18 21:20:34,033] WARN [ReplicaFetcherThread-0-3], Replica 10 for partition [messages,14] reset its fetch offset from 14175984203 to current leader 3's latest offset 35977 (kafka.server.ReplicaFetcherThread)
> Some relevant config parameters:
> offsets.topic.replication.factor = 3
> offsets.commit.required.acks = -1
> replica.high.watermark.checkpoint.interval.ms = 5000
> unclean.leader.election.enable = false
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)