You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Salva (Jira)" <ji...@apache.org> on 2022/06/01 16:15:00 UTC
[jira] [Updated] (FLINK-27872) Allow KafkaBuilder to set arbitrary subscribers
[ https://issues.apache.org/jira/browse/FLINK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Salva updated FLINK-27872:
--------------------------
Description:
The [KafkaBuilder|[https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java]) has the following methods:
* [setTopics|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L157]
* [setTopicsPattern|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L168]
which under the hood instantiate the corresponding subscribers. Those two subscribers should cover most of the cases, I agree, but in some cases they might fall short. Why not adding a more generic setter:
* `setKafkaSubscriber`
Otherwise, how can users read from kafka in combination with custom subscribing logic? Without looking much into it, it seems that they basically cannot, at least without having to replicate some parts of the connector, which seems rather inconvenient.
was:
The [KafkaBuilder](https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java) has the following methods:
* [setTopics|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L157]
* [setTopicsPattern|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L168]
which under the hood instantiate the corresponding subscribers. Those two subscribers should cover most of the cases, I agree, but in some cases they might fall short. Why not adding a more generic setter:
* `setKafkaSubscriber`
Otherwise, how can users read from kafka in combination with custom subscribing logic? Without looking much into it, it seems that they basically cannot, at least without having to replicate some parts of the connector, which seems rather inconvenient.
> Allow KafkaBuilder to set arbitrary subscribers
> -----------------------------------------------
>
> Key: FLINK-27872
> URL: https://issues.apache.org/jira/browse/FLINK-27872
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Kafka
> Reporter: Salva
> Priority: Major
>
> The [KafkaBuilder|[https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java]) has the following methods:
> * [setTopics|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L157]
> * [setTopicsPattern|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L168]
> which under the hood instantiate the corresponding subscribers. Those two subscribers should cover most of the cases, I agree, but in some cases they might fall short. Why not adding a more generic setter:
> * `setKafkaSubscriber`
> Otherwise, how can users read from kafka in combination with custom subscribing logic? Without looking much into it, it seems that they basically cannot, at least without having to replicate some parts of the connector, which seems rather inconvenient.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)