You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Blake Miller (JIRA)" <ji...@apache.org> on 2018/02/28 19:13:00 UTC
[jira] [Commented] (KAFKA-4936) Allow dynamic routing of output
records
[ https://issues.apache.org/jira/browse/KAFKA-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380868#comment-16380868 ]
Blake Miller commented on KAFKA-4936:
-------------------------------------
IMO, such a feature does not need to support automatic topic creation in order to be useful. As [~mjsax] pointed out, it adds significant complexity. I might consider picking this up. I see from the Confluent Google Group that Damian Guy suggested a workaround:
"If you don't know the set of topics then you would need to use a custom Processor and you would also need to create an instance of the KafkaProducer. Keeping in mind that in order to guarantee at-least-once each producer.send would need to be synchronous."
[https://groups.google.com/forum/#!topic/confluent-platform/wnXLKw1-XQk]
I suppose a proper implementation in the KStreams API would do something analogous under the hood. Does that sound reasonable?
Supporting exactly-once here sounds plausible but the details are a little beyond me at present. It seems like the feature might be useful without that, as well, since currently the only options seem to be a custom Processor + Producer, or just fall back to using Producer & Consumer directly instead of Kafka Streams, neither of which would support exactly-once.
> Allow dynamic routing of output records
> ---------------------------------------
>
> Key: KAFKA-4936
> URL: https://issues.apache.org/jira/browse/KAFKA-4936
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Major
> Labels: needs-kip
>
> Currently, all used output topics must be know beforehand, and thus, it's not possible to send output records to topic in a dynamic fashion.
> There have been couple of request for this feature and we should consider adding it. There are many open questions however, with regard to topic creation and configuration (replication factor, number of partitions) etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)