You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Shivangi Singh (Jira)" <ji...@apache.org> on 2020/01/06 17:31:00 UTC
[jira] [Commented] (KAFKA-9367) CRC failure
[ https://issues.apache.org/jira/browse/KAFKA-9367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009032#comment-17009032 ]
Shivangi Singh commented on KAFKA-9367:
---------------------------------------
We are also witnessing following exceptions
[2020-01-06 21:42:12,375] ERROR [ReplicaFetcher replicaId=1009, leaderId=1006, fetcherId=0] Found invalid messages during fetch for partition _topic_name_ offset 909070901 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.CorruptRecordException: Invalid magic found in record: 117
[2020-01-06 21:42:13,417] ERROR [ReplicaFetcher replicaId=1009, leaderId=1006, fetcherId=0] Found invalid messages during fetch for partition payments_payment_data_flow_instances-26 offset 909070901 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.record.InvalidRecordException: Record is corrupt (stored crc = 3006481486, computed crc = 2816728803)
> CRC failure
> ------------
>
> Key: KAFKA-9367
> URL: https://issues.apache.org/jira/browse/KAFKA-9367
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 2.0.0
> Reporter: Shivangi Singh
> Priority: Major
>
> We have a 14 node kafka(2.0.0) cluster
> In our case
> *Leader* : *Broker Id* : 1003 *Ip*: 10.84.198.238
> *Replica* : *Broker Id* : 1014 *Ip*: 10.22.2.74
> A request was sent from replica -> leader to which leader(10.84.198.238) had the following exception
> var/log/kafka/server.log.2019-12-26-00:[2019-12-26 00:13:04,386] ERROR Closing socket for 10.84.198.238:6667-10.22.2.74:53118-121025 because of error (kafka.network.Processor)
> /var/log/kafka/server.log.2019-12-26-00-org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: FETCH, apiVersion: 8, connectionId: 10.84.198.238:6667-10.22.2.74:53118-121025, listenerName: ListenerName(PLAINTEXT), principal: User:ANONYMOUS
> /var/log/kafka/server.log.2019-12-26-00-Caused by: org.apache.kafka.common.protocol.types.SchemaException: *Error reading field 'forgotten_topics_data':* Error reading array of size 23668, only 69 bytes available
> /var/log/kafka/server.log.2019-12-26-00- at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:77)
> /var/log/kafka/server.log.2019-12-26-00- at org.apache.kafka.common.protocol.ApiKeys.parseRequest(ApiKeys.java:290)
> /var/log/kafka/server.log.2019-12-26-00- at org.apache.kafka.common.requests.RequestContext.parseRequest(RequestContext.java:63)
> ------------------------------------------------------------------------
> In response to this, replica (10.22.2.74) had the following log in it
>
> [2019-12-26 00:13:04,390] WARN [ReplicaFetcher replicaId=1014, leaderId=1003, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId=1014, maxWait=500, minBytes=1, maxBytes=10485760, fetchData={_topic_name_=(offset=50344687, logStartOffset=24957467, maxBytes=1048576)}, isolationLevel=READ_UNCOMMITTED, toForget=, metadata=(sessionId=1747349875, epoch=183382033)) (kafka.server.ReplicaFetcherThread)
> java.io.IOException: Connection to 1003 was disconnected before the response was read
> at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:97)
> at kafka.server.ReplicaFetcherBlockingSend.sendRequest(ReplicaFetcherBlockingSend.scala:96)
> at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:240)
> at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:43)
> at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:149)
> at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:114)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
> ------------------------------------------------------------------------
> Post this broker 1003 had the following exception
> /var/log/kafka/server.log.2019-12-26-00:[2019-12-26 00:16:37,828] ERROR [ReplicaFetcher replicaId=1003, leaderId=1014, fetcherId=0] Found invalid messages during etch for partition _topic_name_ offset 91200983 (kafka.server.ReplicaFetcherThread)
> /var/log/kafka/server.log.2019-12-26-00-*org.apache.kafka.common.record.InvalidRecordException: Record is corrupt (stored crc = 1460037823, computed crc = 114378201)*
> /var/log/kafka/server.log.2019-12-26-00:[2019-12-26 00:16:40,690] ERROR Closing socket for 10.84.198.238:6667-10.22.2.74:49850-740543 because of error (kafka.network.Processor)
> /var/log/kafka/server.log.2019-12-26-00-org.apache.kafka.common.errors.InvalidRequestException: Error getting request for apiKey: FETCH, apiVersion: 8, connectionId: 10.84.198.238:6667-10.22.2.74:49850-740543, listenerName: ListenerName(PLAINTEXT), principal: User:ANONYMOUS
> Could you help us with the above issue?
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)