[jira] [Created] (FLINK-39940) DynamicKafkaSource should refresh restored cluster properties from KafkaMetadataService

["Bowen Li (Jira)" <ji...@apache.org>]

Bowen Li created FLINK-39940:
--------------------------------

             Summary: DynamicKafkaSource should refresh restored cluster properties from KafkaMetadataService
                 Key: FLINK-39940
                 URL: https://issues.apache.org/jira/browse/FLINK-39940
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kafka
            Reporter: Bowen Li
            Assignee: Bowen Li


{{DynamicKafkaSource}} restores {{KafkaStream}} metadata from the checkpointed enumerator state when a job is resumed from a checkpoint or savepoint. Before this change, restored sub-enumerators were rebuilt from the checkpointed cluster properties without first refreshing them from {{{}KafkaMetadataService{}}}. That means non-bootstrap Kafka client properties such as authentication or security settings can be stale at restore time, and the job can fail before the next periodic metadata refresh has a chance to reconcile the latest metadata.

The restore path should use the latest metadata-service properties immediately, while still preserving the checkpointed {{bootstrap.servers}} for restored clusters so the enumerator reconnects to the same restored cluster identity. Restore-time metadata lookup failures should also fail fast instead of being deferred until a later discovery cycle.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)