You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Ramkumar (JIRA)" <ji...@apache.org> on 2017/11/17 16:00:00 UTC
[jira] [Updated] (KAFKA-6227) Kafka 0.11.01 New consumer - multiple
consumers under same group not working as expected
[ https://issues.apache.org/jira/browse/KAFKA-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ramkumar updated KAFKA-6227:
----------------------------
Description:
In Kafka 0.8 High level consumers, the consumer.id under group.id differentiates the consumers connection and manage the rebalancing the partitions in zookeeper. Our Service uses this logic and keeps the Kafka stream connection in a cache (Concurrent Hashmap). so that consecutive http client connection doesn’t have to make a stream connection, but takes from cache and read off the messages. This also helps multiple consumers under same group.id can simulatenously make connection to kafka and read off the message (load balancing).
In Kafka 0.11.0.1, the New consumer API the design has changed. The consumer.id properties are no more supported and the connections with zookeeper are managed by Kafka itself. When 2 consumers instances under the same group attempts to make a connection simulatenously , one connection waits on consumer.poll() method until the other one drops the connection. http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html That is at any point of only one active consumer instance is able to poll the messages from the topic. This slightly would change the behavior of our service that we have to restrict only one consumer connection for a group for a topic. That is we couldn’t hold the connection in cache if multiple consumer under same group needs to use the Kafka.
I couldn’t find any properties that aids to make multiple consumer connections on the same group
The manual partition assignment may be a work around but this is way complex to handle that in a service. This is complex because the service needs to track the consumer connections and assign the partitions of the topic and do the rebalancing (what Kafka 0.8 high level consumer does originally). Unfortunately we cannot use legacy api's n Kafka 0.11 since documentation refers about performance degradation in using old apis in new versions.
Was there a solution devised how the highlevel consumer of kafka 0.8 can be migrated with out any change to the behavior from the users perspective
was:
In Kafka 0.8 High level consumers, the consumer.id under group.id differentiates the consumers connection and manage the rebalancing the partitions. Our Service uses this logic and keeps the Kafka stream connection in a cache (Concurrent Hashmap). so that consecutive http client connection doesn’t have to make a stream connection, but takes from cache and read off the messages. This also helps multiple consumers under same group.id can simulatenously make connection to kafka and read off the message (load balancing).
In Kafka 0.11.0.1, the New consumer API the design has changed. The consumer.id properties are no more supported and the connections with zookeeper are managed by Kafka itself. When 2 consumers instances under the same group attempts to make a connection simulatenously , one connection waits on consumer.poll() method until the other one drops the connection. http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html That is at any point of only one active consumer instance is able to poll the messages from the topic. This slightly would change the behavior of our service that we have to restrict only one consumer connection for a group for a topic. That is we couldn’t hold the connection in cache if multiple consumer under same group needs to use the Kafka.
I couldn’t find any properties that aids to make multiple consumer connections on the same group
The manual partition assignment may be a work around but this is way complex to handle that in a service. This is complex because the service needs to track the consumer connections and assign the partitions of the topic and do the rebalancing (what Kafka 0.8 high level consumer does originally). Unfortunately we cannot use legacy api's n Kafka 0.11 since documentation refers about performance degradation in using old apis in new versions.
Was there a solution devised how the highlevel consumer of kafka 0.8 can be migrated with out any change to the behavior from the users perspective
> Kafka 0.11.01 New consumer - multiple consumers under same group not working as expected
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-6227
> URL: https://issues.apache.org/jira/browse/KAFKA-6227
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.11.0.1
> Reporter: Ramkumar
>
> In Kafka 0.8 High level consumers, the consumer.id under group.id differentiates the consumers connection and manage the rebalancing the partitions in zookeeper. Our Service uses this logic and keeps the Kafka stream connection in a cache (Concurrent Hashmap). so that consecutive http client connection doesn’t have to make a stream connection, but takes from cache and read off the messages. This also helps multiple consumers under same group.id can simulatenously make connection to kafka and read off the message (load balancing).
> In Kafka 0.11.0.1, the New consumer API the design has changed. The consumer.id properties are no more supported and the connections with zookeeper are managed by Kafka itself. When 2 consumers instances under the same group attempts to make a connection simulatenously , one connection waits on consumer.poll() method until the other one drops the connection. http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html That is at any point of only one active consumer instance is able to poll the messages from the topic. This slightly would change the behavior of our service that we have to restrict only one consumer connection for a group for a topic. That is we couldn’t hold the connection in cache if multiple consumer under same group needs to use the Kafka.
> I couldn’t find any properties that aids to make multiple consumer connections on the same group
> The manual partition assignment may be a work around but this is way complex to handle that in a service. This is complex because the service needs to track the consumer connections and assign the partitions of the topic and do the rebalancing (what Kafka 0.8 high level consumer does originally). Unfortunately we cannot use legacy api's n Kafka 0.11 since documentation refers about performance degradation in using old apis in new versions.
> Was there a solution devised how the highlevel consumer of kafka 0.8 can be migrated with out any change to the behavior from the users perspective
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)