[jira] [Commented] (FLINK-27872) Allow KafkaBuilder to set arbitrary subscribers

["Salva (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17545390#comment-17545390 ] 

Salva commented on FLINK-27872:
-------------------------------

Based on a quick experiment, it seems that using a custom subscriber requires to create your own fork for these two:
- `KafkaSourceBuilder`

-`KafkaSource`

The rest of the connector seems reusable, but that is still far from ideal (although better than what I originally thought).

> Allow KafkaBuilder to set arbitrary subscribers
> -----------------------------------------------
>
>                 Key: FLINK-27872
>                 URL: https://issues.apache.org/jira/browse/FLINK-27872
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / Kafka
>            Reporter: Salva
>            Priority: Major
>
> Currently, [KafkaBuilder|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java] has two setters:
>  * [setTopics|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L157]
>  * [setTopicPattern|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L168]
> which under the hood instantiate the corresponding (concrete) subscribers. This covers the most common needs, I agree, but it might fall short in some cases. Why not add a more generic setter:
>  * {{setSubscriber (???)}}
> Otherwise, how can users read from kafka in combination with custom subscribing logic? Without looking much into it, it seems that they basically cannot, at least without having to replicate some parts of the connector, which seems rather inconvenient.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)