[jira] [Comment Edited] (FLINK-39054) DynamicKafkaSourceEnumStateSerializer should preserve full per-cluster properties instead of only bootstrap.servers

["Bowen Li (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-39054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18059129#comment-18059129 ] 

Bowen Li edited comment on FLINK-39054 at 2/17/26 6:32 PM:
-----------------------------------------------------------

on further research, i think persisting all configs or not should be a decision the job makes, instead of forcing on all jobs

ideally there should be a configurable option to let jobs decide to persist only bare minimal vs all configs

thus i'm going to hold on pr for it


was (Author: bli):
on further research, i think persisting all configs or not should be a decision the job makes, instead of forcing on all jobs

ideally there should be a configurable option to let jobs decide to persist only bare minimal vs all configs

> DynamicKafkaSourceEnumStateSerializer should preserve full per-cluster properties instead of only bootstrap.servers
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-39054
>                 URL: https://issues.apache.org/jira/browse/FLINK-39054
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka
>    Affects Versions: kafka-4.0.1
>            Reporter: Bowen Li
>            Assignee: Bowen Li
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: kafka-4.1.0
>
>
> Problem:
> DynamicKafkaSourceEnumStateSerializer persists only `bootstrap.servers` in V1, dropping other cluster properties on restore. We added V2 in FLINK-38876 while maintaining the same configs as V1
> serialization: [https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L146]
> deserialization: [https://github.com/apache/flink-connector-kafka/blob/cdbd635171fc4322ba7182eb93e920472d6d9d91/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/dynamic/source/enumerator/DynamicKafkaSourceEnumStateSerializer.java#L179]
> Essentially, V1 state causes loss of configs like auth/network after checkpoint restore and avoids incorrect defaults.
>  
> Solution:
> Since DynamicKafkaSourceEnumStateSerializer V2 is not released with a new flink-connector-kafka version yet as of 2/9/2026, we can update V2 of DynamicKafkaSourceEnumStateSerializer to checkpoint full per-cluster Properties, and still restores V1 states.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)