You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "chaos (Jira)" <ji...@apache.org> on 2021/08/30 10:12:00 UTC
[jira] [Updated] (KAFKA-13251) ISR shrink to an error broker
[ https://issues.apache.org/jira/browse/KAFKA-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chaos updated KAFKA-13251:
--------------------------
Summary: ISR shrink to an error broker (was: 2.4.0)
> ISR shrink to an error broker
> -----------------------------
>
> Key: KAFKA-13251
> URL: https://issues.apache.org/jira/browse/KAFKA-13251
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.4.0
> Environment: linux 4.1.0
> Reporter: chaos
> Priority: Major
>
> Disk error occurred in broker(=42),and then Shrinking ISR to itself.
> so why Shrinking ISR to an error broker?
> i.e. why not "Shrinking ISR from 55,42 to 55" but "Shrinking ISR from 55,42 to 42".
> note:
> other partition(110) shrink correctly.
>
> kafka logs:
> broker42:
> [2021-08-26 20:20:55,640] ERROR [ReplicaManager broker=42] Error processing fetch with max size 1048576 from consumer on partition topic_xx-123: (fetchOffset=11061228956, logStartOffset=-1, maxBytes=1048576, currentLeaderEpoch=Optional.empty) (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.CorruptRecordException: Found record size 0 smaller than minimum record overhead (14) in file /data4/kafka-logs/topic_xx-123/00000000011060934646.log.
> [2021-08-26 20:20:55,640] ERROR Error while appending records to topic_xx-123 in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> [2021-08-26 20:20:55,645] ERROR Error while deleting segments for topic_xx-123 in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> java.nio.file.FileSystemException: /data4/kafka-logs/topic_xx-123/00000000011040402299.log -> /data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only file system
> Suppressed: java.nio.file.FileSystemException: /data4/kafka-logs/topic_xx-123/00000000011040402299.log -> /data4/kafka-logs/topic_xx-123/00000000011040402299.log.deleted: Read-only file system
> [2021-08-26 20:20:55,644] ERROR Error while appending records to topic_xx-123 in dir /data4/kafka-logs (kafka.server.LogDirFailureChannel)
> [2021-08-26 20:20:55,652] INFO [Partition topic_xx-123 broker=42] Shrinking ISR from 55,42 to 42. Leader: (highWatermark: 11061228956, endOffset: 11061228965). Out of sync replicas: (brokerId: 55, endOffset: 11061228956). (kafka.cluster.Partition)
>
> broker55:
> [2021-08-26 20:20:32,456] WARN [ReplicaFetcher replicaId=55, leaderId=42, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=55, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=830774713, epoch=1562806014), rackId=) (kafka.server.ReplicaFetcherThread)
> [2021-08-26 20:20:43,503] INFO [Partition topic_xxx-110 broker=55] Shrinking ISR from 55,42 to 55. Leader: (highWatermark: 11061384367, endOffset: 11061388788). Out of sync replicas: (brokerId: 42, endOffset: 11061384367). (kafka.cluster.Partition)
>
> disk error on broker42 is:
> Aug 26 20:20:55 kernel: sd 0:2:5:0: [sdf] tag#33 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
--
This message was sent by Atlassian Jira
(v8.3.4#803005)