You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Edoardo Comar (JIRA)" <ji...@apache.org> on 2016/05/23 14:03:12 UTC
[jira] [Issue Comment Deleted] (KAFKA-3727) Consumer.poll() stuck
in loop on non-existent topic manually assigned
[ https://issues.apache.org/jira/browse/KAFKA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edoardo Comar updated KAFKA-3727:
---------------------------------
Comment: was deleted
(was: To explain better, this is a stack trace when the consumer is stuck in the loop :
{quote}
Fetcher<K,V>.listOffset(TopicPartition, long) line: 320
Fetcher<K,V>.resetOffset(TopicPartition) line: 294
Fetcher<K,V>.updateFetchPositions(Set<TopicPartition>) line: 170
KafkaConsumer<K,V>.updateFetchPositions(Set<TopicPartition>) line: 1408
KafkaConsumer<K,V>.pollOnce(long) line: 982
KafkaConsumer<K,V>.poll(long) line: 937
{quote}
that is, the ConsumerNetworkClient will keep invoking {{ awaitMetadataUpdate(); }}
{quote}
private long listOffset(TopicPartition partition, long timestamp) {
while (true) {
RequestFuture<Long> future = sendListOffsetRequest(partition, timestamp);
client.poll(future);
if (future.succeeded())
return future.value();
if (!future.isRetriable())
throw future.exception();
if (future.exception() instanceof InvalidMetadataException)
client.awaitMetadataUpdate();
else
time.sleep(retryBackoffMs);
}
}
{quote}
Don't you think the consumer should give up at some point ?)
> Consumer.poll() stuck in loop on non-existent topic manually assigned
> ---------------------------------------------------------------------
>
> Key: KAFKA-3727
> URL: https://issues.apache.org/jira/browse/KAFKA-3727
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Reporter: Edoardo Comar
>
> The behavior of a consumer on poll() for a non-existing topic is surprisingly different/inconsistent
> between a consumer that subscribed to the topic and one that had the topic-partition manually assigned.
> The "subscribed" consumer will return an empty collection
> The "assigned" consumer will *loop forever* - this feels a bug to me.
> sample snippet to reproduce:
> {quote}
> KafkaConsumer<String, String> assignKc = new KafkaConsumer<>(props1);
> KafkaConsumer<String, String> subsKc = new KafkaConsumer<>(props2);
> List<TopicPartition> tps = new ArrayList<>();
> tps.add(new TopicPartition("topic-not-exists", 0));
> assignKc.assign(tps);
> subsKc.subscribe(Arrays.asList("topic-not-exists"));
> System.out.println("********* subscribe k consumer ");
> ConsumerRecords<String, String> crs2 = subsKc.poll(1000L);
> print("subscribeKc", crs2); // returns empty
> System.out.println("********* assign k consumer ");
> ConsumerRecords<String, String> crs1 = assignKc.poll(1000L);
> // will loop forever !
> print("assignKc", crs1);
> {quote}
> the logs for the "assigned" consumer show:
> [2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = []) (org.apache.kafka.clients.Metadata)
> [2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for fetching offset, wait for metadata refresh (org.apache.kafka.clients.consumer.internals.Fetcher)
> [2016-05-18 17:33:10,010] DEBUG Sending metadata request {topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
> [2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)