You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Andy Coates (JIRA)" <ji...@apache.org> on 2016/07/01 09:36:11 UTC
[jira] [Issue Comment Deleted] (KAFKA-3919) Broker faills to start
after ungraceful shutdown due to non-monotonically incrementing offsets in
logs
[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andy Coates updated KAFKA-3919:
-------------------------------
Comment: was deleted
(was: Hi [~junrao], thanks for taking the time to look at this.
Note: I've incorporated some of what I say below into the problem description above, so that it doesn't get lost in the comments.
First off, we have unclean leadership elections disable. (We did later enable them to help get around some other issues we were having, but this was several hours after this issue manifested).
We did look through data logs that were causing the brokers to not start. What we found before the incident was a monotonically increasing offset, where each compressed batch normally contained one or two records. Then the is a batch that contains many records, whose first records has an offset below the previous batch and whose last record has an offset above the previous batch. Following on from this there continues a period of large batches, with monotonically increasing offsets, and then the log returns to batches with one or two records.
Our working assumption here is that the period before the offset dip is pre-outage normal operation. The period of larger batches is from just after the outage, where producers have a back log to processes when the partition becomes available, and then things return to normal batch sizes again once the back log clears.
We did also look through the Kafka's application logs to try and piece together the series of events leading up to this:
Here’s what I know happened, with regards to one partition that has issues, from the logs:
Prior to outage:
Replicas for the partition are brokers 2011, 2012, 2024, with 2024 being the preferred leader.
Producers using acks=1, compression=gzip
Brokers configured with unclean.elections=false, zk.session-timeout=36s
Post outage:
2011 comes up first, (also as the Controller), recovers unflushed log set 1239444214, completes load with offset 1239740602, and becomes leader of the partition.
2012 comes up next, recovers its log, recovers unflushed log set 1239444214, truncates to offset 1239742830, (thats 2,228 records ahead of the recovered offset of the current leader), and starts following.
2024 comes up quickly after 2012. recovers unflushed log set 1239444214, truncates to offset 1239742250, (thats 1,648 records ahead of the recovered offset of the current leader), and starts following.
The Controller adds 2024 to the replica set just before 2024 halts due to another partition having an offset greater than the leader.
The Controller adds 2012 to the replica set just before 2012 halts due to another partition having an offset greater than the leader.
When 2012 is next restarted, it fails to fully start as its complaining of invalid offsets in the log.
Our working hypothesis here is that the partition becomes writeable again, possibly as brokers 2012 & 2024 get added to the ISR set before halting, and maybe don’t remove themselves when they halt? - hence remain in the ISR set for 36 seconds. Mean while our producers are happily sending large compressed batches, as they have a backlog, to broker 2011, which is accepting them, (as there are enough replicas in the ISR set), and appending them to its log - moving its offset beyond brokers 2012 and 2024.
Log entries:
(Interleaved log entries from the three brokers - the broker id is in the [id] brackets)
Just as the power was going out I see this in the broker that was the controller:
2016-04-11 12:01:42 - [2026] - "[Partition state machine on Controller 2026]: Invoking state change to OnlinePartition for partitions [mt_xp_its_music_main_itsevent,20]”
2016-04-11 12:01:56 - [2026] - "[Replica state machine on controller 2026]: Invoking state change to OfflineReplica for replicas
[Topic=mt_xp_its_music_main_itsevent,Partition=20,Replica=2024]
2016-04-11 12:01:56 - [2026] - "[Controller 2026]: Cannot remove replica 2024 from ISR of partition [mt_xp_its_music_main_itsevent,20] since it is not in the ISR. Leader = 2011 ; ISR = List(2011, 2012)”
2016-04-11 12:01:56 - [2026] - "[Channel manager on controller 2026]: Not sending request {controller_id=2026,controller_epoch=111,delete_partitions=0,partitions=[{topic=mt_xp_its_music_main_itsevent,partition=20}]} to broker 2024, since it is offline.”
2016-04-11 12:04:46 - [2026] - [Replica state machine on controller 2026]: Invoking state change to OnlineReplica for replicas
[Topic=mt_xp_its_music_main_itsevent,Partition=20,Replica=2024]
2016-04-11 12:04:58 - [2026] - "[Controller 2026]: Starting preferred replica leader election for partitions [mt_xp_its_music_main_itsevent,20]”
2016-04-11 12:04:58 - [2026] - "[Partition state machine on Controller 2026]: Invoking state change to OnlinePartition for partitions [mt_xp_its_music_main_itsevent,20]”
2016-04-11 12:04:58 - [2026] - "[PreferredReplicaPartitionLeaderSelector]: Current leader 2011 for partition [mt_xp_its_music_main_itsevent,20] is not the preferred replica. Trigerring preferred replica leader election”
2016-04-11 12:04:58 - [2026] - "[Controller 2026]: Partition [mt_xp_its_music_main_itsevent,20] completed preferred replica leader election. New leader is 2024"
Then following on from the outage I see this:
2016-06-02 13:00:49 - [2011] - starts back up
2016-06-02 13:02:10 - [2011] - "Found a corrupted index file, /data3/kafka/mt_xp_its_music_main_itsevent-20/00000000001239444214.index, deleting and rebuilding index..."
2016-06-02 13:02:13 - [2011] - "Recovering unflushed segment 1239444214 in log mt_xp_its_music_main_itsevent-20.”
2016-06-02 13:02:15 - [2011] - "Completed load of log mt_xp_its_music_main_itsevent-20 with log end offset 1239740602”
2016-06-02 13:03:27 - [2011] -[Replica state machine on controller 2011]: Started replica state machine with initial state ->
[Topic=mt_xp_its_music_main_itsevent,Partition=20,Replica=2011] -> ReplicaDeletionIneligible, …
2016-06-02 13:03:27,560 - [2011] - "Controller 2011 epoch 114 initiated state change for partition [mt_xp_its_music_main_itsevent,20] from OfflinePartition to OnlinePartition failed" kafka.common.NoReplicaOnlineException: No broker in ISR for partition [mt_xp_its_music_main_itsevent,20] is alive. Live brokers are: [Set()], ISR brokers are: [2012,2024,2011]
2016-06-02 13:03:32 - [2011] - [Partition state machine on Controller 2011]: Started partition state machine with initial state ->
[mt_xp_its_music_main_itsevent,20] -> OfflinePartition, ...
2016-06-02 13:03:32 - [2011] - [Replica state machine on controller 2011]: Invoking state change to OnlineReplica for replicas …
[Topic=mt_xp_its_music_main_itsevent,Partition=20,Replica=2011], …
2016-06-02 13:03:32 - [2011] - "[OfflinePartitionLeaderSelector]: Selected new leader and ISR {"leader":2011,"leader_epoch":12,"isr":[2011]} for offline partition [mt_xp_its_music_main_itsevent,20]”
2016-06-02 13:03:32 - [2011] - "Broker 2011 received LeaderAndIsrRequest with correlation id 1 from controller 2011 epoch 113 for partition [mt_xp_its_music_main_itsevent,20] but cannot become follower since the new leader 2024 is unavailable."
2016-06-02 13:03:36 - [2011] - "[ReplicaFetcherManager on broker 2011] Removed fetcher for partitions…
[mt_xp_its_music_main_itsevent,20], …
2016/06/02 13:05:37 - [2012] - starts back up.
2016-06-02 13:05:39 - [2024] - starts back up.
2016-06-02 13:06:29 - [2012] - "Recovering unflushed segment 1239444214 in log mt_xp_its_music_main_itsevent-20.”
2016-06-02 13:06:31 - [2012] - "Completed load of log mt_xp_its_music_main_itsevent-20 with log end offset 1239742830”
2016-06-02 13:06:50- [2024] - "Found a corrupted index file, /data2/kafka/mt_xp_its_music_main_itsevent-20/00000000001239444214.index, deleting and rebuilding index…"
2016-06-02 13:06:54 - [2024] - Recovering unflushed segment 1239444214 in log mt_xp_its_music_main_itsevent-20."
2016-06-02 13:06:56 - [2024] - Completed load of log mt_xp_its_music_main_itsevent-20 with log end offset 1239742250"
2016-06-02 13:07:59 - [2011] - [Replica state machine on controller 2011]: Invoking state change to OnlineReplica for replicas
[Topic=mt_xp_its_music_main_itsevent,Partition=20,Replica=2024], …
2016-06-02 13:08:00 - [2024] - "Truncating log mt_xp_its_music_main_itsevent-20 to offset 1239742250.”
2016-06-02 13:08:00 - [2024] - "[ReplicaFetcherManager on broker 2024] Removed fetcher for partitions [… [mt_xp_its_music_main_itsevent,20]”
2016-06-02 13:08:00 - [2024] - "[ReplicaFetcherManager on broker 2024] Added fetcher for partitions List([... [[mt_xp_its_music_main_itsevent,20], initOffset 1239742250 to broker BrokerEndPoint(2011,st13p00it-bdkafka122.rock.apple.com,9092)] … “
2016-06-02 13:08:00 - [2024] - "[ReplicaFetcherThread-0-2017], Halting because log truncation is not allowed for topic mt_xp_its_main_itsevent, Current leader 2017's latest offset 7128021741 is less than replica 2024's latest offset 7128915601"
2016-06-02 13:08:12 - [2011] - [Replica state machine on controller 2011]: Invoking state change to OfflineReplica for replicas
...[Topic=mt_xp_its_music_main_itsevent,Partition=20,Replica=2024],…"
2016-06-02 13:08:13 - [2011] - [Controller 2011]: Cannot remove replica 2024 from ISR of partition [mt_xp_its_music_main_itsevent,20] since it is not in the ISR. Leader = 2011 ; ISR = List(2011)”
2016-06-02 13:08:13 - [2011] - [Channel manager on controller 2011]: Not sending request {controller_id=2011,controller_epoch=114,partition_states=[...,{topic=mt_xp_its_music_main_itsevent,partition=20,controller_epoch=114,leader=2011,leader_epoch=12,isr=[2011],zk_version=25,replicas=[2024,2011,2012]},...} to broker 2012, since it is offline.”
2016-06-02 13:08:13 - [2011] - "Broker 2011 ignoring LeaderAndIsr request from controller 2011 with correlation id 30 epoch 114 for partition [mt_xp_its_music_main_itsevent,20] since its associated leader epoch 12 is old. Current leader epoch is 12”
2016-06-02 13:08:13 - [2011] - "[Channel manager on controller 2011]: Not sending request {controller_id=2011,controller_epoch=114,delete_partitions=0,partitions=[{topic=mt_xp_its_music_main_itsevent,partition=20}]} to broker 2024, since it is offline.”
2016-06-02 13:08:15 - [2011] - "[Replica state machine on controller 2011]: Invoking state change to OnlineReplica for replicas …,[Topic=mt_xp_its_music_main_itsevent,Partition=20,Replica=2012],...
2016-06-02 13:08:16 - [2012] - [ReplicaFetcherManager on broker 2012] Removed fetcher for partitions:
[mt_xp_its_music_main_itsevent,20], …
2016-06-02 13:08:16 - [2012] - "Truncating log mt_xp_its_music_main_itsevent-20 to offset 1239742830."
2016-06-02 13:08:16 - [2012] - [ReplicaFetcherManager on broker 2012] Added fetcher for partitions List(
..[[mt_xp_its_music_main_itsevent,20], initOffset 1239742830 to broker BrokerEndPoint(2011,st13p00it-bdkafka122.rock.apple.com,9092)]), ...
2016-06-02 13:08:17 - [2012] - [ReplicaFetcherThread-0-2011], Halting because log truncation is not allowed for topic OneDirection_output, Current leader 2011's latest offset 44631162 is less than replica 2012's latest offset 44738961
2016-06-02 13:08:27 - [2011] - "[Replica state machine on controller 2011]: Invoking state change to OfflineReplica for replicas ...,[Topic=mt_xp_its_music_main_itsevent,Partition=20,Replica=2012],…"
2016-06-02 13:08:27 - [2011] - "[Controller 2011]: Cannot remove replica 2012 from ISR of partition [mt_xp_its_music_main_itsevent,20] since it is not in the ISR. Leader = 2011 ; ISR = List(2011)”
2016-06-02 13:08:28 - [2011] -"[Channel manager on controller 2011]: Not sending request {controller_id=2011,controller_epoch=114,partition_states=[..,{topic=mt_xp_its_music_main_itsevent,partition=20,controller_epoch=114,leader=2011,leader_epoch=12,isr=[2011],zk_version=25,replicas=[2024,2011,2012]},...]} to broker 2024, since it is offline."
2016-06-02 13:08:28 - [2011] - "Broker 2011 ignoring LeaderAndIsr request from controller 2011 with correlation id 42 epoch 114 for partition [mt_xp_its_music_main_itsevent,20] since its associated leader epoch 12 is old. Current leader epoch is 12”
2016-06-02 13:08:28 - [2011] - "[Channel manager on controller 2011]: Not sending request {controller_id=2011,controller_epoch=114,delete_partitions=0,partitions=[{topic=mt_xp_its_music_main_itsevent,partition=20}]} to broker 2012, since it is offline."
2016-06-02 13:10:07 - [2012] - Starts back up
2016-06-02 13:10:48 - [2012] - "Found a corrupted index file, /data3/kafka/OneDirection_output-14/00000000000044555803.index, deleting and rebuilding index…"
2016-06-02 13:10:53 - [2012] - "There was an error in one of the threads during logs loading: kafka.common.InvalidOffsetException: Attempt to append an offset (1239742691) to position 35728 no larger than the last offset appended (1239742822) to /data3/kafka/mt_xp_its_music_main_itsevent-20/00000000001239444214.index.”
Broker config:
timestamp="2016-06-02 13:05:39,440" thread="main" level=INFO logger=kafka.server.KafkaConfig message="KafkaConfig values:
advertised.host.name = null
metric.reporters = [com.apple.its.bigdata.kafka.metrics.HubbleMetricsReporter]
quota.producer.default = 9223372036854775807
offsets.topic.num.partitions = 50
log.flush.interval.messages = 9223372036854775807
auto.create.topics.enable = true
controller.socket.timeout.ms = 30000
log.flush.interval.ms = null
principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
replica.socket.receive.buffer.bytes = 65536
min.insync.replicas = 2
replica.fetch.wait.max.ms = 500
num.recovery.threads.per.data.dir = 2
ssl.keystore.type = JKS
default.replication.factor = 3
ssl.truststore.password = null
log.preallocate = false
sasl.kerberos.principal.to.local.rules = [DEFAULT]
fetch.purgatory.purge.interval.requests = 1000
ssl.endpoint.identification.algorithm = null
replica.socket.timeout.ms = 30000
message.max.bytes = 10485760
num.io.threads = 16
offsets.commit.required.acks = -1
log.flush.offset.checkpoint.interval.ms = 60000
delete.topic.enable = true
quota.window.size.seconds = 1
ssl.truststore.type = JKS
offsets.commit.timeout.ms = 5000
quota.window.num = 11
zookeeper.connect = st11p00it-volacct029:2181,st11p00it-volacct030:2181,st11p00it-volacct054:2181/kafka-three
authorizer.class.name =
num.replica.fetchers = 1
log.retention.ms = null
log.roll.jitter.hours = 0
log.cleaner.enable = true
offsets.load.buffer.size = 5242880
log.cleaner.delete.retention.ms = 86400000
ssl.client.auth = none
controlled.shutdown.max.retries = 3
queued.max.requests = 500
offsets.topic.replication.factor = 3
log.cleaner.threads = 1
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
socket.request.max.bytes = 104857600
ssl.trustmanager.algorithm = PKIX
zookeeper.session.timeout.ms = 6000
log.retention.bytes = -1
sasl.kerberos.min.time.before.relogin = 60000
zookeeper.set.acl = false
connections.max.idle.ms = 600000
offsets.retention.minutes = 1440
replica.fetch.backoff.ms = 1000
inter.broker.protocol.version = 0.9.0.0
log.retention.hours = 720
num.partitions = 150
broker.id.generation.enable = true
listeners = null
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
log.roll.ms = null
log.flush.scheduler.interval.ms = 9223372036854775807
ssl.cipher.suites = null
log.index.size.max.bytes = 10485760
ssl.keymanager.algorithm = SunX509
security.inter.broker.protocol = PLAINTEXT
replica.fetch.max.bytes = 10485760
advertised.port = null
log.cleaner.dedupe.buffer.size = 134217728
replica.high.watermark.checkpoint.interval.ms = 5000
log.cleaner.io.buffer.size = 524288
sasl.kerberos.ticket.renew.window.factor = 0.8
zookeeper.connection.timeout.ms = 1000000
controlled.shutdown.retry.backoff.ms = 5000
log.roll.hours = 168
log.cleanup.policy = delete
host.name =
log.roll.jitter.ms = null
max.connections.per.ip = 2147483647
offsets.topic.segment.bytes = 104857600
background.threads = 10
quota.consumer.default = 9223372036854775807
request.timeout.ms = 30000
log.index.interval.bytes = 4096
log.dir = /tmp/kafka-logs
log.segment.bytes = 1073741824
log.cleaner.backoff.ms = 15000
offset.metadata.max.bytes = 4096
ssl.truststore.location = null
group.max.session.timeout.ms = 600000
ssl.keystore.password = null
zookeeper.sync.time.ms = 2000
port = 9092
log.retention.minutes = null
log.segment.delete.delay.ms = 60000
log.dirs = /data1/kafka,/data2/kafka,/data3/kafka
controlled.shutdown.enable = true
compression.type = producer
max.connections.per.ip.overrides =
sasl.kerberos.kinit.cmd = /usr/bin/kinit
log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
auto.leader.rebalance.enable = true
leader.imbalance.check.interval.seconds = 300
log.cleaner.min.cleanable.ratio = 0.5
replica.lag.time.max.ms = 10000
num.network.threads = 16
ssl.key.password = null
reserved.broker.max.id = 8000
metrics.num.samples = 2
socket.send.buffer.bytes = 1048576
ssl.protocol = TLS
socket.receive.buffer.bytes = 1048576
ssl.keystore.location = null
replica.fetch.min.bytes = 1
unclean.leader.election.enable = false
group.min.session.timeout.ms = 6000
log.cleaner.io.buffer.load.factor = 0.9
offsets.retention.check.interval.ms = 600000
producer.purgatory.purge.interval.requests = 1000
metrics.sample.window.ms = 60000
broker.id = 2024
offsets.topic.compression.codec = 0
log.retention.check.interval.ms = 300000
advertised.listeners = null
leader.imbalance.per.broker.percentage = 10)
> Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs
> ------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-3919
> URL: https://issues.apache.org/jira/browse/KAFKA-3919
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.9.0.1
> Reporter: Andy Coates
>
> Hi All,
> I encountered an issue with Kafka following a power outage that saw a proportion of our cluster disappear. When the power came back on several brokers halted on start up with the error:
> {noformat}
> Fatal error during KafkaServerStartable startup. Prepare to shutdown”
> kafka.common.InvalidOffsetException: Attempt to append an offset (1239742691) to position 35728 no larger than the last offset appended (1239742822) to /data3/kafka/mt_xp_its_music_main_itsevent-20/00000000001239444214.index.
> at kafka.log.OffsetIndex$$anonfun$append$1.apply$mcV$sp(OffsetIndex.scala:207)
> at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:197)
> at kafka.log.OffsetIndex$$anonfun$append$1.apply(OffsetIndex.scala:197)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at kafka.log.OffsetIndex.append(OffsetIndex.scala:197)
> at kafka.log.LogSegment.recover(LogSegment.scala:188)
> at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:188)
> at kafka.log.Log$$anonfun$loadSegments$4.apply(Log.scala:160)
> at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
> at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> at kafka.log.Log.loadSegments(Log.scala:160)
> at kafka.log.Log.<init>(Log.scala:90)
> at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:150)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:60)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The only way to recover the brokers was to delete the log files that contained non monotonically incrementing offsets.
> I’ve spent some time digging through the logs and I feel I may have worked out the sequence of events leading to this issue, (though this is based on some assumptions I've made about the way Kafka is working, which may be wrong).
> Given:
> * A topic that is produced to using acks = 1
> * A topic that is produced to using gzip compression
> * A topic that has min.isr set to less than the number of replicas, (i.e. min.isr=2, #replicas=3)
> * Following ISRs are lagging behind the leader by some small number of messages, (which is normal with acks=1)
> * brokers are configured with fairly large zk session timeout e.g. 30s.
> * brokers are configured so that unclean leader elections are disabled.
> Then:
> When something like a power outage take out all three replicas, its possible to get into a state such that the indexes won’t rebuild on a restart and a broker fails to start. This can happen when:
> * Enough brokers, but not the pre-outage leader, come on-line for the partition to be writeable
> * Producers produce enough records to the partition that the head offset is now greater than the pre-outage leader head offset.
> * The pre-outage leader comes back online.
> At this point the logs on the pre-outage leader have diverged from the other replicas. It has some messages that are not in the other replicas, and the other replicas have some records not in the pre-outage leader's log - at the same offsets.
> I’m assuming that because the current leader has at higher offset than the pre-outage leader, the pre-outage leader just starts following the leader and requesting the records it thinks its missing.
> I’m also assuming that because the producers were using gzip, so each record is actual a compressed message set, that iwhen the pre-outage leader requests records from the leader, the offset it requests could just happened to be in the middle of a compressed batch, but the leader returns the full batch. When the pre-outage leader appends this batch to its own log it thinks all is OK. But what has happened is that the offsets in the log are no longer monotonically incrementing. Instead they actually dip by the number of records in the compressed batch that were before the requested offset. If and when this broker restarts this dip may be at the 4K boundary the indexer checks. If it is, the broker won’t start.
> Several of our brokers were unlucky enough to hit that 4K boundary, causing a protracted outage. We’ve written a little utility that shows several more brokers have a dip outside of the 4K boundary.
> There are some assumptions in there, which I’ve not got around to confirming / denying. (A quick attempt to recreate this failed and I've not found the time to invest more).
> Of course I'd really appreciate the community / experts stepping in and commenting on whether my assumptions are right or wrong, or if there is another explanation to the problem.
> But assuming I’m mostly right, then the fact the broker won’t start is obviously a bug, and one I’d like to fix. A Kafka broker should not corrupt its own log during normal operation to the point that it can’t restart!
> A secondary issue is if we think the divergent logs are acceptable? This may be deemed acceptable given the producers have chosen availability over consistency when they produced with acks = 1? Though personally, the system having diverging replicas of an immutable commit log just doesn't sit right.
> I see us having a few options here:
> * Have the replicas detect the divergence of their logs e.g. a follower compares the checksum of its last record with the same offset on the leader. The follower can then workout that its log has diverged from the leader. At which point it could either halt, stop replicating that partition or search backwards to find the point of divergence, truncate and recover. (possibly saving the truncated part somewhere). This would be a protocol change for Kafka. This solution trades availability, (you’ve got less ISRs during the extended re-sync process), for consistency.
> * Leave the logs as they are and have the indexing of offsets in the log on start up handle such a situation gracefully. This leaves logs in a divergent state between replicas, (meaning replays would yield different messages if the leader was up to down), but gives better availability, (no time spent not being an ISR while it repairs any divergence).
> * Support multiple options and allow it be tuned, ideally by topic.
> * Something else...
> I’m happy/keen to contribute here. But I’d like to first discuss which option should be investigated.
> Andy
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)