You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Upendra Yadav <up...@gmail.com> on 2019/11/27 17:03:16 UTC
Producer slowness and Exception on broker restart -
kafka.common.KafkaException: Unknown group metadata version 1
Hi Team,
In my production setup, I’m getting below exception(for 4-5
__consumer_offsets partitions) while restarting kafka broker(s).
Before restart, my producers(sync) are very slow. for some messages its
taking 5 seconds to acknowledge.
But after restart, all messages wishing 2 milliseconds.
After sometime once again same producer slowness.
Before restart was able to produce 7k messages, but after restart able to
push 1lakh messages in 10 minutes.
Kafka broker and client version- 10.0.0.1
Using latest producer java client for this version.
But consumers are old scala api.
Exception:
[2019-11-27 05:54:43,724] INFO [Group Metadata Manager on Broker 3]:
Loading offsets and group metadata from [__consumer_offsets,34]
(kafka.coordinator.GroupMetadataManager)
[2019-11-27 05:54:43,735] ERROR [Group Metadata Manager on Broker 3]: Error
in loading offsets from [__consumer_offsets,34]
(kafka.coordinator.GroupMetadataManager)
kafka.common.KafkaException: Unknown group metadata version 1
at
kafka.coordinator.GroupMetadataManager$.schemaForGroup(GroupMetadataManager.scala:764)
at
kafka.coordinator.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:934)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:412)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:383)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at kafka.message.MessageSet.foreach(MessageSet.scala:71)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply$mcV$sp(GroupMetadataManager.scala:383)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
at
kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
at
kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1(GroupMetadataManager.scala:374)
at
kafka.coordinator.GroupMetadataManager$$anonfun$loadGroupsForPartition$1.apply$mcV$sp(GroupMetadataManager.scala:353)
at
kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Re: Producer slowness and Exception on broker restart -
kafka.common.KafkaException: Unknown group metadata version 1
Posted by Upendra Yadav <up...@gmail.com>.
Hi,
We also did version upgrade and then degraded to old due to some issue.
But that we done ~1 year before.
Do I need to follow this to solve this issue?
http://mail-archives.apache.org/mod_mbox/kafka-dev/201907.mbox/%3CJIRA.13246826.1563975667000.40318.1563975720170@Atlassian.JIRA%3E
In this he follow below steps to evicted the broken records with
temporarily setting the segment time to a very low value and deactivation
of compaction.
/opt/kafka/bin/kafka-topics.sh --alter --config segment.ms=900000
--topic __consumer_offsets
--zookeeper localhost:2181
/opt/kafka/bin/kafka-topics.sh --alter --config cleanup.policy=delete
--topic __consumer_offsets
--zookeeper localhost:2181
< wait for the cleaner to clean up >
/opt/kafka/bin/kafka-topics.sh --alter --config segment.ms=604800000
--topic __consumer_offsets
--zookeeper localhost:2181
/opt/kafka/bin/kafka-topics.sh --alter --config cleanup.policy=compact
--topic __consumer_offsets
--zookeeper localhost:2181{code}
On Wed, Nov 27, 2019 at 10:33 PM Upendra Yadav <up...@gmail.com>
wrote:
> Hi Team,
>
> In my production setup, I’m getting below exception(for 4-5
> __consumer_offsets partitions) while restarting kafka broker(s).
> Before restart, my producers(sync) are very slow. for some messages its
> taking 5 seconds to acknowledge.
> But after restart, all messages wishing 2 milliseconds.
> After sometime once again same producer slowness.
> Before restart was able to produce 7k messages, but after restart able to
> push 1lakh messages in 10 minutes.
>
> Kafka broker and client version- 10.0.0.1
> Using latest producer java client for this version.
> But consumers are old scala api.
>
> Exception:
>
> [2019-11-27 05:54:43,724] INFO [Group Metadata Manager on Broker 3]:
> Loading offsets and group metadata from [__consumer_offsets,34]
> (kafka.coordinator.GroupMetadataManager)
> [2019-11-27 05:54:43,735] ERROR [Group Metadata Manager on Broker 3]:
> Error in loading offsets from [__consumer_offsets,34]
> (kafka.coordinator.GroupMetadataManager)
> kafka.common.KafkaException: Unknown group metadata version 1
> at
> kafka.coordinator.GroupMetadataManager$.schemaForGroup(GroupMetadataManager.scala:764)
> at
> kafka.coordinator.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:934)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:412)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1$$anonfun$apply$mcV$sp$1.apply(GroupMetadataManager.scala:383)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
> at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at kafka.message.MessageSet.foreach(MessageSet.scala:71)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply$mcV$sp(GroupMetadataManager.scala:383)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1$1.apply(GroupMetadataManager.scala:374)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:231)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:239)
> at
> kafka.coordinator.GroupMetadataManager.kafka$coordinator$GroupMetadataManager$$loadGroupsAndOffsets$1(GroupMetadataManager.scala:374)
> at
> kafka.coordinator.GroupMetadataManager$$anonfun$loadGroupsForPartition$1.apply$mcV$sp(GroupMetadataManager.scala:353)
> at
> kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:110)
> at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>