[jira] [Commented] (KAFKA-9092) Kafka Streams Upgrade Magic v0 does not support record headers

["Matthias J. Sax (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16958557#comment-16958557 ] 

Matthias J. Sax commented on KAFKA-9092:
----------------------------------------

It seem your topics are using older message format. To use headers, you need to upgrade the topic message format to 0.11 (or newer).

The older Kafka Streams application worked, because it did not support headers.

Try to upgrade the message format.

> Kafka Streams Upgrade Magic v0 does not support record headers
> --------------------------------------------------------------
>
>                 Key: KAFKA-9092
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9092
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 1.0.0, 2.0.0
>         Environment: kafka 1.0.0
> kafka streams lib 2.0.0
>            Reporter: wxmimperio
>            Priority: Major
>
> My kafka cluster version: 1.0.0 and run a streams app to topic v1.
>  
> Then I upgrade kafka streams lib to 2.0.0 and want to use some new apis.
> Reference Upgrade doc: [http://kafka.apache.org/20/documentation/streams/upgrade-guide]
>  
> {code:java}
> <dependency>
>  <groupId>org.apache.kafka</groupId>
>  <artifactId>kafka-streams</artifactId>
>  <version>2.0.0</version>
> </dependency>{code}
>  
> {code:java}
> // streams config
> settings.put(StreamsConfig.UPGRADE_FROM_CONFIG, "1.0");{code}
>  
> Then stop streams app and rebuild a new jar to start.
> It’s no problem just starting to run. After a few hours kafka broker logs error and I delete topic recreate a few hours kafka broker get same error：
> {code:java}
> [2019-10-24 10:15:45,780] ERROR [KafkaApi-109] Error when handling request {replica_id=-1,max_wait_time=100,min_bytes=1,topics=[{topic=xxx,partitions=[{partition=4,fetch_offset=3165633,max_bytes=1048576}]}]} (kafka.server.KafkaApis)[2019-10-24 10:15:45,780] ERROR [KafkaApi-109] Error when handling request {replica_id=-1,max_wait_time=100,min_bytes=1,topics=[{topic=xxx,partitions=[{partition=4,fetch_offset=3165633,max_bytes=1048576}]}]} (kafka.server.KafkaApis)java.lang.IllegalArgumentException: Magic v0 does not support record headers at org.apache.kafka.common.record.MemoryRecordsBuilder.appendWithOffset(MemoryRecordsBuilder.java:403) at org.apache.kafka.common.record.MemoryRecordsBuilder.append(MemoryRecordsBuilder.java:586) at org.apache.kafka.common.record.AbstractRecords.convertRecordBatch(AbstractRecords.java:134) at org.apache.kafka.common.record.AbstractRecords.downConvert(AbstractRecords.java:109) at org.apache.kafka.common.record.FileRecords.downConvert(FileRecords.java:253) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$convertedPartitionData$1$1$$anonfun$apply$4.apply(KafkaApis.scala:520) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$convertedPartitionData$1$1$$anonfun$apply$4.apply(KafkaApis.scala:518) at scala.Option.map(Option.scala:146) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$convertedPartitionData$1$1.apply(KafkaApis.scala:518) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$convertedPartitionData$1$1.apply(KafkaApis.scala:508) at scala.Option.flatMap(Option.scala:171) at kafka.server.KafkaApis.kafka$server$KafkaApis$$convertedPartitionData$1(KafkaApis.scala:508) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$createResponse$2$1.apply(KafkaApis.scala:556) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$createResponse$2$1.apply(KafkaApis.scala:555) at scala.collection.Iterator$class.foreach(Iterator.scala:891) at scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at kafka.server.KafkaApis.kafka$server$KafkaApis$$createResponse$2(KafkaApis.scala:555) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$fetchResponseCallback$1$1.apply(KafkaApis.scala:569) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$fetchResponseCallback$1$1.apply(KafkaApis.scala:569) at kafka.server.KafkaApis$$anonfun$sendResponseMaybeThrottle$1.apply$mcVI$sp(KafkaApis.scala:2034) at kafka.server.ClientRequestQuotaManager.maybeRecordAndThrottle(ClientRequestQuotaManager.scala:52) at kafka.server.KafkaApis.sendResponseMaybeThrottle(KafkaApis.scala:2033) at kafka.server.KafkaApis.kafka$server$KafkaApis$$fetchResponseCallback$1(KafkaApis.scala:569) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$processResponseCallback$1$1.apply$mcVI$sp(KafkaApis.scala:588) at kafka.server.ClientQuotaManager.maybeRecordAndThrottle(ClientQuotaManager.scala:175) at kafka.server.KafkaApis.kafka$server$KafkaApis$$processResponseCallback$1(KafkaApis.scala:587) at kafka.server.KafkaApis$$anonfun$handleFetchRequest$3.apply(KafkaApis.scala:604) at kafka.server.KafkaApis$$anonfun$handleFetchRequest$3.apply(KafkaApis.scala:604) at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:820) at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:596) at kafka.server.KafkaApis.handle(KafkaApis.scala:100) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:65) at java.lang.Thread.run(Thread.java:744){code}
>  
> I found this issue:https://issues.apache.org/jira/browse/KAFKA-6739?src=confmacro
> It is a bug for kafka 1.0.0?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)