You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Vyacheslav Koptilin (Jira)" <ji...@apache.org> on 2023/03/28 12:18:00 UTC
[jira] [Updated] (IGNITE-19102) CDC: Use manual assignment for consumer in KafkatoIgniteMetadataUpdater
[ https://issues.apache.org/jira/browse/IGNITE-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vyacheslav Koptilin updated IGNITE-19102:
-----------------------------------------
Summary: CDC: Use manual assignment for consumer in KafkatoIgniteMetadataUpdater (was: CDC: Use manual assignment for consumer in KafkatoIgniteMetadtaUpdater)
> CDC: Use manual assignment for consumer in KafkatoIgniteMetadataUpdater
> -----------------------------------------------------------------------
>
> Key: IGNITE-19102
> URL: https://issues.apache.org/jira/browse/IGNITE-19102
> Project: Ignite
> Issue Type: Task
> Components: extensions
> Reporter: Ilya Shishkov
> Priority: Major
> Labels: IEP-59, ise
>
> Currently, {{KafkaToIgniteMetadataUpdater}} uses {{KafkaConsumer#subscribe}}, which lead to utilization of the consumer group management [1, "Consumer Groups and Topic Subscriptions"]. Kafka group management has some possible side-effects, which can be reduced be manual assignment:
> # {{KafkaConsumer}} leaves consumer group after {{max.poll.interval.ms}} exceeds [2], because {{KafkaToIgniteMetadataUpdater}} polls data only on demand, not in continuous manner.
> # Delays during consumer group rebalancing.
> # Metadata topic should have single partition. Multiple consumers from the same group can not subscribe on the same partition.
> *Important:* {{KafkaConsumer}} in {{KafkaToIgniteMetadataUpdater}} must be subscribed / assigned to all partitions.
> Links:
> # https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> # https://kafka.apache.org/27/documentation.html#consumerconfigs_max.poll.interval.ms
--
This message was sent by Atlassian Jira
(v8.20.10#820010)