[jira] [Comment Edited] (KAFKA-5766) Very high CPU-load of consumer when broker is down

["Gary Russell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16234738#comment-16234738 ] 

Gary Russell edited comment on KAFKA-5766 at 11/1/17 9:26 PM:
--------------------------------------------------------------

Yes, you can reduce the CPU load for a down broker by adding {{reconnect.backoff.ms}} but the {{poll()}} never exits so there's no way for the app consumer code to know that the broker is down.

It would be useful if there was an option to throw an exception in this case and/or add some kind of {{Listener}} strategy so the app can discover that the broker is down.

{{poll()}} is blocked indefinitely in {{ensureCoordinatorReady(0, Long.MAX_VALUE);}} until woken, but we seem to have no way to know that a wakeup is needed.


was (Author: grussell):
Yes, you can reduce the CPU load for a down broker by adding {{reconnect.backoff.ms}} but the {{poll()}} never exits so there's no way for the app consumer code to know that the broker is down.

It would be useful if there was an option to throw an exception in this case and/or add some kind of {{Listener}} strategy so the app can discover that the broker is down.

> Very high CPU-load of consumer when broker is down
> --------------------------------------------------
>
>                 Key: KAFKA-5766
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5766
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>            Reporter: Sebastian Bernauer
>            Priority: Major
>
> Hi,
> i have a single broker instance at localhost.
> I set up a Consumer with the following code:
> {code:java}
>         Map<String, Object> configs = new HashMap<>();
>         configs.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
>         configs.put(ConsumerConfig.GROUP_ID_CONFIG, "gh399");
>         configs.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
>         configs.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
>         KafkaConsumer<String, String> consumer = new KafkaConsumer<>(configs);
>         consumer.assign(Collections.singletonList(new TopicPartition("foo", 0)));
>         while (true) {
>             ConsumerRecords<String, String> records = consumer.poll(1000);
>             System.out.println(records.count());
>         }
> {code}
> This works all fine, until i shut down the broker.
> If i do so, it causes a 100% CPU-load of my application.
> After starting the broker again the usage decreases back to a normal level. 
> It would be very nice if you could help me!
> Thanks,
> Sebastian
> Spring-Kafka: 2.0.0.M3
> Kafka: 0.10.2.0
> JDK: 1.8.0_121



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)