[jira] [Comment Edited] (FLINK-32893) Make client.id configurable from KafkaSource

["Rich (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-32893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870914#comment-17870914 ] 

Rich edited comment on FLINK-32893 at 8/5/24 3:47 AM:
------------------------------------------------------

I would vote for supporting a static `client.id` for the Kafka source.

1. This aligns with the Kafka client and gives developers the option to use either `client.id` or `client.id.prefix` with trade off [1]. Currently, when `client.id` is passed without `client.id.prefix`, it is confusing and unexpected that the client ID is [randomly generated|https://github.com/apache/flink-connector-kafka/blob/86f796a01cba0d7b3adeb95f413e412c30e466f1/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L484-L490].

propose behaviors as follows, what do you think?
||client.id||client.id.prefix||current||proposal||
|null|null|random|random|
|my-client-id|null|random|my-client-id|
|null|my-prefix|my-prefix-\{subTaskId}|my-prefix-\{subTaskId}|
|my-client-id|my-prefix|my-prefix-\{subTaskId}|my-prefix-\{subTaskId}|

   
2. I agree with the principle (i.e. {{{}user_id{}}}) of controlling the quota. However, Kafka allows [setting quotas by client_id, user_id, or both|https://kafka.apache.org/documentation/#quotas]. There are also use cases for mono applications which share the same {{{}user_id{}}}. In order to control the quota for different components in this mono application, {{client.id}} is required.

 

[1] When using a static `client.id` for Kafka clients, developers must acknowledge and accept the risk of incomplete metrics from Kafka clients. Registering multiple KafkaConsumers with the same `client.id` can result in a `javax.management.InstanceAlreadyExistsException` as a warning.
https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/kafka/#kafka-consumer-metrics


was (Author: JIRAUSER300785):
I would vote for supporting a static `client.id` for the Kafka source.

1. This aligns with the Kafka client and gives developers the option to use either `client.id` or `client.id.prefix`. Currently, when `client.id` is passed without `client.id.prefix`, it is confusing and unexpected that the client ID is [randomly generated|https://github.com/apache/flink-connector-kafka/blob/86f796a01cba0d7b3adeb95f413e412c30e466f1/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L484-L490].

propose behaviors as follows, what do you think?


||client.id||client.id.prefix||current||proposal||
|null|null|random|random|
|my-client-id|null|random|my-client-id|
|null|my-prefix|my-prefix-\{subTaskId}|my-prefix-\{subTaskId}|
|my-client-id|my-prefix|my-prefix-\{subTaskId}|my-prefix-\{subTaskId}|


   
2. I agree with the principle (i.e. {{{}user_id{}}}) of controlling the quota. However, Kafka allows [setting quotas by client_id, user_id, or both|https://kafka.apache.org/documentation/#quotas]. There are also use cases for mono applications which share the same {{{}user_id{}}}. In order to control the quota for different components in this mono application, {{client.id}} is required.

 

> Make client.id configurable from KafkaSource
> --------------------------------------------
>
>                 Key: FLINK-32893
>                 URL: https://issues.apache.org/jira/browse/FLINK-32893
>             Project: Flink
>          Issue Type: Improvement
>    Affects Versions: kafka-3.0.0
>            Reporter: Mason Chen
>            Priority: Major
>              Labels: pull-request-available
>
> The client id is not strictly configurable from the KafkaSource because it appends a configurable prefix and subtask information to avoid the mbean conflict exception messages that are internal to the Kafka client.
>  
> However, various users reported that they need use this client.id for Kafka quotas and they need to have control over the client.id to enforce quotas properly.
>  
> Affects Kafka connector 3.1 and below.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)