You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Mickael Maison (JIRA)" <ji...@apache.org> on 2017/07/13 10:34:00 UTC
[jira] [Commented] (KAFKA-3239) Timing issue in controller metrics
on topic delete
[ https://issues.apache.org/jira/browse/KAFKA-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085523#comment-16085523 ]
Mickael Maison commented on KAFKA-3239:
---------------------------------------
We are still seeing this with 0.10.2.1:
{quote}
[2017-04-19 14:56:26,251] ERROR Error printing regular metrics: (com.airbnb.metrics.StatsDReporter)
java.util.NoSuchElementException: key not found: [TOPIC_NAME_REDACTED,0]
at scala.collection.MapLike$class.default(MapLike.scala:228)
at scala.collection.AbstractMap.default(Map.scala:59)
at scala.collection.mutable.HashMap.apply(HashMap.scala:65)
at kafka.controller.KafkaController$$anon$3$$anonfun$value$2$$anonfun$apply$mcI$sp$2.apply(KafkaController.scala:210)
at kafka.controller.KafkaController$$anon$3$$anonfun$value$2$$anonfun$apply$mcI$sp$2.apply(KafkaController.scala:208)
at scala.collection.TraversableOnce$$anonfun$count$1.apply(TraversableOnce.scala:114)
at scala.collection.TraversableOnce$$anonfun$count$1.apply(TraversableOnce.scala:113)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at scala.collection.TraversableOnce$class.count(TraversableOnce.scala:113)
at scala.collection.AbstractTraversable.count(Traversable.scala:104)
at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply$mcI$sp(KafkaController.scala:208)
at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply(KafkaController.scala:205)
at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply(KafkaController.scala:205)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
at kafka.controller.KafkaController$$anon$3.value(KafkaController.scala:204)
at kafka.controller.KafkaController$$anon$3.value(KafkaController.scala:202)
at com.airbnb.metrics.StatsDReporter.processGauge(StatsDReporter.java:163)
at com.airbnb.metrics.StatsDReporter.processGauge(StatsDReporter.java:37)
at com.yammer.metrics.core.Gauge.processWith(Gauge.java:28)
at com.airbnb.metrics.StatsDReporter.sendAMetric(StatsDReporter.java:131)
at com.airbnb.metrics.StatsDReporter.sendAllKafkaMetrics(StatsDReporter.java:119)
at com.airbnb.metrics.StatsDReporter.run(StatsDReporter.java:85)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:483)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:316)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:190)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:627)
at java.lang.Thread.run(Thread.java:809)
{quote}
The topic was stuck in "Marked for deletion" state. Such topics don't have leaders, so it's probably the issue is probably with:
{quote}
controllerContext.partitionLeadershipInfo(topicPartition)
{quote}
We're trying out a fix at the moment, if that works I'll send a PR.
> Timing issue in controller metrics on topic delete
> --------------------------------------------------
>
> Key: KAFKA-3239
> URL: https://issues.apache.org/jira/browse/KAFKA-3239
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Affects Versions: 0.9.0.0
> Reporter: Rajini Sivaram
> Assignee: Rajini Sivaram
>
> Noticed this exception in our logs:
> {quote}
> java.util.NoSuchElementException: key not found: [sometopic,0]
> at scala.collection.MapLike$class.default(MapLike.scala:228)
> at scala.collection.AbstractMap.default(Map.scala:59)
> at scala.collection.mutable.HashMap.apply(HashMap.scala:65)
> at kafka.controller.KafkaController$$anon$3$$anonfun$value$2$$anonfun$apply$mcI$sp$2.apply(KafkaController.scala:209)
> at kafka.controller.KafkaController$$anon$3$$anonfun$value$2$$anonfun$apply$mcI$sp$2.apply(KafkaController.scala:208)
> at scala.collection.TraversableOnce$$anonfun$count$1.apply(TraversableOnce.scala:114)
> at scala.collection.TraversableOnce$$anonfun$count$1.apply(TraversableOnce.scala:113)
> at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
> at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
> at scala.collection.TraversableOnce$class.count(TraversableOnce.scala:113)
> at scala.collection.AbstractTraversable.count(Traversable.scala:104)
> at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply$mcI$sp(KafkaController.scala:208)
> at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply(KafkaController.scala:205)
> at kafka.controller.KafkaController$$anon$3$$anonfun$value$2.apply(KafkaController.scala:205)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:262)
> at kafka.controller.KafkaController$$anon$3.value(KafkaController.scala:204)
> at kafka.controller.KafkaController$$anon$3.value(KafkaController.scala:202)
> at com.airbnb.metrics.StatsDReporter.processGauge(StatsDReporter.java:163)
> at com.airbnb.metrics.StatsDReporter.processGauge(StatsDReporter.java:37)
> at com.yammer.metrics.core.Gauge.processWith(Gauge.java:28)
> at com.airbnb.metrics.StatsDReporter.sendAMetric(StatsDReporter.java:131)
> at com.airbnb.metrics.StatsDReporter.sendAllKafkaMetrics(StatsDReporter.java:119)
> at com.airbnb.metrics.StatsDReporter.run(StatsDReporter.java:85)
> {quote}
> The exception indicates that the topic was in {{controllerContext.partitionReplicaAssignment}} but not in {{controllerContext.partitionLeadershipInfo}}. This can occur during {{KafkaController.removeTopic()}} since it is not synchronized.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)