[jira] [Assigned] (CAMEL-22811) camel-kafka: Support Kafka 4.0 KIP-848 consumer rebalance protocol (group.protocol=consumer)

["Andrea Cosentino (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-22811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrea Cosentino reassigned CAMEL-22811:
----------------------------------------

    Assignee: Andrea Cosentino

> camel-kafka: Support Kafka 4.0 KIP-848 consumer rebalance protocol (group.protocol=consumer)
> --------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-22811
>                 URL: https://issues.apache.org/jira/browse/CAMEL-22811
>             Project: Camel
>          Issue Type: New Feature
>          Components: camel-kafka
>            Reporter: SRIKANTH PRATHIPATI
>            Assignee: Andrea Cosentino
>            Priority: Minor
>             Fix For: 4.x
>
>
> h2. Background
> Apache Kafka 4.0 introduces the “Next Generation of the Consumer Rebalance Protocol” (KIP-848), which is GA and designed to be fully incremental (no global sync barrier), improving rebalance times and scalability.​
> With the new protocol, some behavior and configuration moves to the broker: heartbeat interval, session timeout, and the assignment strategy are controlled server-side for {{Consumer}} groups.​
> h2. Problem statement (Camel impact)
> KIP-848 expects the broker to manage {{{}partitionAssignor{}}}, {{{}heartbeatIntervalMs{}}}, and {{{}sessionTimeoutMs{}}}, while the current Camel Kafka component “defaulted these if these are not supplied.”​
> Kafka’s own documentation for the new protocol states that {{{}heartbeat.interval.ms{}}}, {{{}session.timeout.ms{}}}, and {{partition.assignment.strategy}} are “no longer usable” when {{group.protocol=consumer}} is enabled.
> Please refer to the related thread - [#camel > Kafka | Next Generation of the Consumer Rebalance Protocol|https://camel.zulipchat.com/#narrow/channel/257298-camel/topic/Kafka.20.7C.20Next.20Generation.20of.20the.20Consumer.20Rebalance.20Protocol/with/541949206]
> h2. Requested change (what to implement)
>  * Add/confirm first-class support for enabling the new protocol via consumer property {{group.protocol=consumer}} (pass-through supported), and ensure Camel does not inject defaults that conflict with the new protocol.​​
>  * When {{{}group.protocol=consumer{}}}, avoid setting (or warn/error if user sets) classic-only properties such as {{{}heartbeat.interval.ms{}}}, {{{}session.timeout.ms{}}}, and {{partition.assignment.strategy}} (and Camel-level equivalents like {{{}heartbeatIntervalMs{}}}, {{{}sessionTimeoutMs{}}}, {{partitionAssignor}} defaults).​​
>  * Optionally expose/document {{group.remote.assignor}} for selecting a server-side assignor (override broker default) when using the consumer protocol.
> h2. Acceptance criteria (suggested)
>  * With Kafka client 4.0+ and {{{}group.protocol=consumer{}}}, Camel consumer starts successfully without requiring {{session.timeout.ms}} / {{heartbeat.interval.ms}} / {{partition.assignment.strategy}} and without Camel adding defaults for them.​​
>  * Add an integration test matrix (at least one) that runs Camel Kafka consumer with {{group.protocol=consumer}} against a Kafka 4.0+ broker.​
>  * Documentation update: clearly describe “Classic vs Consumer” protocol selection, and call out unsupported configs under the consumer protocol.​
> [https://www.confluent.io/blog/kip-848-consumer-rebalance-protocol/]
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)